• 22
• FTP Fetching Files from Everywhere
• Basic Operation
• Transfer Mode
• Netiquette
• Essential Commands
• Common Problems
• Finding Sites
• Mac/PC Usage
• Tools
• FTP by Mail
• Advanced Use

22

FTP Fetching Files from Everywhere

File Transfer Protocol (FTP) is one of the most venerable methods of transferring files from one computer to another. The earliest specification for FTP dates from April 1971. It is still the workhorse of the Internet and will remain so for many years. Two crucial aspects of FTP make it particularly useful: its capability to transfer files between computers of completely dissimilar types and its provision for public file sharing.

This chapter elaborates on the procedures for using FTP, beginning with some simple examples. The chapter also deals with finding sources: How do you find that super-slick software you heard was available through FTP? You also learn about Mac and PC use of FTP, including some tools for those platforms. Also described are additional tools, ranging from X Window interfaces to FTP using e-mail. The "Advanced Use" section, later in this chapter, explores the uses of automated login and wildcards.

Basic Operation

In essence, there are two ways of using FTP. You can use it to transfer files from machine to machine or to access public files that exist on anonymous FTP servers. This latter use is the most common, because there are innumerable anonymous servers providing everything from aardvark fonts to computer operating systems to ZOO files.

Publicly accessible FTP servers are called anonymous servers because of the procedure used to connect to them. Traditionally, the account logged on to a server was anonymous; currently, it is common to use the FTP account. Similarly, the traditional password was guest, but most current systems request the use of your e-mail address as the password. For example, if you want to connect to a machine called amber.fubar.com, you enter the following:

ftp amber.fubar.com

If your system can connect to that machine, you see something similar to the following:

Connected to amber.fubar.com.

220 amber ftp server (SunOS 4.1) ready.

Name (amber:kdb): ftp

331 Guest login ok, send ident as password.

Password:

230 Guest login ok, access restrictions apply.

ftp>

The first line confirms the connection to the desired machine. The line beginning with 220 indicates that the FTP server process is ready (in this case, running under SunOS 4.1). Next, the system prompts for a Name matching your current username on the FTP machine. If you had an account with that name, you could press Return. In this example, ftp is entered as the username, which is followed by the request to send ident (e-mail address) as password. The password is not echoed back, so you cannot see any typing mistakes you make. However, in most cases, you enter only your username followed by @ (most FTP servers can determine your e-mail address anyway).

The final response (the line beginning with 230) indicates that you are logged in and can execute all available commands. To get a list of those commands, enter a ?. To get information on a given command, enter ? followed by the command, as in this example:

ftp> ? dir

dir list contents of remote directory

The dir command is the equivalent of the UNIX ls -l (or the DOS DIR command). Note that the directory being listed is referred to as the remote directory. The server's side is always considered the remote; the machine on which you originate the FTP process is known as local. The concept of local versus remote is important because it underlies many of the FTP commands and procedures. For example, when you want to change directories on the remote machine, use the cd command. However, to change directories on the local machine, use the lcd (local cd) command.

Another concept important in FTP is the client-server model. When you engage FTP, you are actually running the FTP client on your machine. The client intercepts commands and interprets them, passing on the appropriate requests to the server. One of the reasons it is important to keep this in mind is that not all servers have the same commands available. Thus, the remotehelp command is useful for listing the commands supported by the remote server.

Continuing with the example session, the next step is to obtain a directory listing to see what files and directories are available. As noted, you obtain this listing with the dir command:

ftp> dir

200 PORT command successful.

150 ASCII data connection for /bin/ls (193.131.225.1,3247) (0 bytes). total 5

drwxr-xr-x 2 root system 512 Nov 29 1994 bin

drwxr-xr-x 2 root system 512 Sep 8 1994 dev

drwxr-xr-x 2 root system 512 Nov 29 1994 etc

drwxrwxr-x 24 ftp system 1024 Dec 19 00:18 pub

drwxr-xr-x 3 root system 512 Nov 29 1994 usr

226 ASCII Transfer complete.

436 bytes received in 0.047 seconds (9.1 Kbytes/s)

The dir command produced a listing typical of many servers. The bin, dev, etc, and usr directories are of little interest because they contain only system files to enable FTP to operate normally.

As is usually the case, the pub directory is the one you want. If you don't see a pub directory, chances are the server has put you there automatically. To change to the pub directory, use the cd command:

ftp> cd pub

250 CWD command successful.

ftp> dir

200 PORT command successful.

150 ASCII data connection for /bin/ls (193.131.225.1,3248) (0 bytes).

total 18

-rw-rw-r— 1 ftp daemon 15248 Jan 14 19:23 LS-lR.Z

-rw-rw-r— 1 ftp daemon 5323 Dec 9 9:18 README

drwxrwxr-x 2 ftp staff 512 Feb 17 1995 axel

-rw-rw-rw- 1 ftp daemon 71258 Jan 6 1995 donen.ps

drwxr-xr-x 9 ftp daemon 512 Sep 1 1994 mailers

-rw-rw-rw- 1 ftp daemon 3443 Dec 19 00:18 netrc

-rw-rw-r— 1 ftp daemon 319616 Oct 27 18:21 superbridge.sea

-rwxr-xr-x 1 ftp daemon 19272 Oct 21 1994 quest.hqx

-rw-r-r— 1 ftp daemon 28425 Nov 9 1994 vxw

226 ASCII Transfer complete.

3092 bytes received in 0.55 seconds (5.5 Kbytes/s)

The format you receive back from the dir command depends on the system to which you're connected. In this example, the system is a UNIX machine running SunOS, so you see the equivalent of an ls -lga command (see the discussion of file listings in Appendix C of this book).

Assume that you want the README file: download it by typing get README. A file named README is placed in the directory you were in when you started the FTP program. If you want to use another name for the file to avoid overwriting an existing README file, specify the name as follows:

get README amber.readme

This get command results in the README file being copied to your local host with the name amber.readme.

Transfer Mode

The default transfer mode is ASCII, meaning that the transfer is treated as text (IBM mainframers take heart—ASCII should really be called text because it gets translated to EBCDIC automatically in the mainframe world). This means that the file is readable on the local host because it is translated according to the local requirement. However, the translation is beneficial only with text files; it can make other files unusable.

Binary files, such as executable programs, require the binary transfer mode. This mode is also referred to as image because it copies an exact image of the original. Of course, even though the copied program is an exact duplicate, it does not change the fact that a binary program can run only on the machine for which it is compiled. In other words, a PC binary runs only on a PC, regardless of where it is transferred by FTP. However, you can transfer that PC binary using a machine of a completely different type, transfer the file to a PC, and the file will run.

The question, then, is how to determine which transfer mode to use for a given file? One method uses file extensions, or suffixes, to discern the file type. (Table 22.1 lists some common file types and their transfer modes.) Unfortunately, only some files are flagged by a telling extension. Mostly, using file extensions is a question of educated guesswork. If a file is not marked by one of the extensions listed in Table 22.1 and is not executable (that is, a dir listing of the file shows no x—see Appendix C), chances are it can be transferred in ASCII mode. On the other hand, if you know that the remote and local hosts are identical, you can use binary mode for both text and data files. Because you have a 50/50 chance of being right, transfer the file in binary mode and then check it on the local machine. NonASCII machines are becoming less common, so the translation of ASCII mode often is not required.

Table 22.1. File types, extensions, and transfer mode.

File Type

Extension

Transfer Mode

To switch to binary mode, type binary. Following is an example of downloading the ls-lR.Z file from the server:

ftp> binary

200 Type set to I.

ftp> get LS-lR.Z

200 PORT command successful.

150 Binary data connection for LS-lR.Z (193.131.225.1,3924) (15248 bytes).

226 Binary Transfer complete.

local: LS-lR.Z remote: LS-lR.Z

15248 bytes received in 0.0044 seconds (3.4e+03 Kbytes/s)

ftp>

The message, Type set to I, shows the server's response: the transfer type has been set to Image (a synonym for binary mode). To return to ASCII mode, type ascii. The server responds with Type set to A (or something similar).

When you finish your session, type bye or quit to close your connection to the server. The server politely responds with Goodbye, and you are returned to your system prompt.

Netiquette

Anonymous FTP servers are usually running as "public services" on machines that have other work to do. Common courtesy—netiquette, as it is referred to on the Net—requires that you respect those servers that request usage to be restricted to off-hours (usually outside of 8 a.m. to 6 p.m. local time). Many larger servers announce that service is available 24 hours a day. However, massive downloads that can be done during off-hours should be done at those hours if possible. Even machines that are dedicated servers can simultaneously serve only a finite number of FTP sessions; file transfers that take a long time preclude others from making use of the system.

Another aspect of netiquette involves the login process. Although many implementations of the FTP server do not enforce the use of your e-mail address as the password, you should get in the habit of supplying the address nonetheless (as mentioned earlier in this chapter, your username followed by an @ will suffice). Because logging your user ID and site is becoming standard procedure, it is not a question of losing anonymity. Rather, the question concerns courtesy. As Jon Granrose repeatedly pointed out in his anonymous FTP listings, FTP is a privilege, not a right.

Essential Commands

The supported FTP commands vary somewhat from one type of server to another, but a core group of essential commands exists and is supported by almost all FTP programs. The best way to determine whether a command is supported is to try it. If it fails, obtain a listing of commands by issuing the help command. This command (which usually has ? as a synonym) lists the commands the local client supports. Once connected to a server, you can type remotehelp to obtain a listing of the supported commands on the server.

For most purposes, however, the list in Table 22.2 (in which curly braces { } indicate optional arguments) should be all you require.

Table 22.2. A list of essential FTP commands.

FTP Command

Description

Common Problems

The following error message captures what is probably the most common problem when trying to FTP to a site:

unknown host

Often enough, the reason for this error is the mistyping of the host address. Check the spelling of the hostname and try again. If you still get an error message and you have access to the ping command, you can verify the existence and accessibility of a host by typing this command:

ping -s {hostname} 1 1

This command sends the named host one byte in one packet. The purpose here is twofold: the command determines whether your host is a valid hostname, and the procedure demonstrates whether the site is reachable from your host. If so, you will have a numeric address to try:

amber ~ > ping -s cerf.net 1 1

PING cerf.net: 1 data bytes

9 bytes from nic.cerf.net (192.102.249.3): icmp_seq=0.

——nic.cerf.net PING Statistics——

1 packets transmitted, 1 packets received, 0% packet loss

The important thing to note is that the numeric address is given (in this example, the numeric address is 192.102.249.3), which can be used in place of the full site name (in this example, nic.cerf.net). If your FTP client does not have access to a nameserver, using the numeric address is often necessary.

Another problem arises because not all sites accept anonymous FTP connections. Just because a site has an FTP server does not mean that anonymous FTP service is enabled. In fact, most UNIX machines are shipped with a functional FTP server. However, as shipped, the server only enables users listed in the system password file to access their own accounts.

Finally, not all anonymous FTP sites accept ftp as the username. Although ftp is becoming as common as anonymous for the login, some servers do not accept it. Make sure that you try the anonymous login before flaming the administrator of a site.

Finding Sites

If you know where to find what you are looking for, you can connect to that site. But how do you find the name of an appropriate site—or worse, the name of a program or file? The complete answer is beyond the scope of this chapter. However, several approaches fall within the realm of FTP.

The usual method to locate files in the FTP domain is the Archie program (discussed in Chapter 24, "Archie: An Archive of Archives"). You should use Archie, but if you find it is impractical for your site, don't despair. Before Archie existed, there were lists of FTP sites that briefly described what collections those sites maintained.

Probably the most comprehensive list was maintained by Jon Granrose at pilot.njin.net. Although very dated, this file is still somewhat useful because it can provide a starting point for searching for general archive sites (for example, mac, X11, and TeX). The file is called ftp-list and can be obtained from any of the following sites:

ftp://ftp.gsfc.nasa.gov/pub/ftp-list

ftp://funet.fi/netinfo/ftp-list

ftp://nic.cerf.net/pub/doc/ftp.list.Z

Jon Granrose apparently last updated the list in December 1991.

NOTE Because IP numbers tend to change more frequently than host names, it is prudent to try the names rather than the numbers listed. Given the date on this list, however, even the host names are likely to be out of date.

More recently, the list was updated by Tom Czarnik. Using a slightly different format, Czarnik compiled a substantial list. However, the last version he edited seems to be from April 1993. The list is currently maintained by Perry Rovers. It can be found at the following sites:

ftp://oak.oakland.edu/pub/msdos/info/ftp-list.zip

ftp://garbo.uwasa.fi/pc/doc-net/ftp-list.zip

The list also can be obtained from the mail-server at MIT. Send mail to mail-server@rtfm.mit.edu with no subject line; for the body of the message, enter this text:

send usenet/news.answers/ftp-list/faq

When using these listings, remember that they are dated. You can obtain more current information by using the Archie services. If Archie is not available, however, the ftp-list can provide a useful starting point for your search.

Mac/PC Usage

If you use a microcomputer (a Mac or a PC) directly connected to the Internet, you can use FTP to transfer files directly to and from your desktop. A good example is Fetch, a program for the Mac that not only downloads programs with a few clicks of a mouse button, it also automatically unbinhexes binaries on the fly! The end result is usually a program that is ready to run or a self-extracting archive (with an .SEA extension) that you can double-click to start. As with all the programs mentioned in this chapter, consult Archie to find the latest version and nearest source.

Although no single program is available for the PC that is the exact equivalent of Fetch, there are some very good packages. One of these is the NCSA Telnet package, which includes an FTP utility. Several suites of TCP/IP software, such as Chameleon or Internet in a Box, contain useful FTP programs.

When transferring files to and from a microcomputer, it is important to remember which is the client and which is the server. With an application such as Fetch, the Mac acts as a client connecting to an FTP server. When you get a file in this case, you are transferring it from the server to your Mac.

On the other hand, when using the FTP function of NCSA Telnet, you are actually running the FTP client on the host to which you connected, and your Mac (or PC) becomes the server. When you get a file in this case, you are getting it from your Mac, not the host to which you connected. The logic seems to be reversed, unless you keep in mind that your microcomputer is the server.

CAUTION By default, the FTP server is enabled while you are running NCSA Telnet. This potentially allows anyone on the Internet to access and download the files on your microcomputer! To avoid this problem, ensure that you have set up an NCSA Telnet password file.

If your microcomputer is not directly connected to the Internet, you can still download software from FTP servers using a two-step process. First, FTP the file from the remote server to a local host. Then use traditional file transfer such as ZModem or Kermit to download from the local host to your PC.

Some of the main sites for Mac or PC archives are listed in Table 22.3. Because of the popularity of these sites, they are commonly not accessible at peak hours. Fortunately, popular sites are mirrored at other sites, so it is often more convenient to use a mirror site.

Table 22.3. Primary FTP sites for Mac and DOS archives.

Site

Specialty

Mirrored By

Tools

As mentioned earlier in this chapter, Fetch is an extremely useful Mac interface to FTP. Similarly, other programs facilitate FTP usage on other platforms. For example, users of OpenWindows can obtain the ftptool distribution. Table 22.4 shows some other FTP tools.

Table 22.4. Examples of various FTP resources.

Tool

OS

Type

The ftptool utility was written as shareware by Mike Sullivan at Sun Microsystems. It has features that provide a significant advantage over the standard UNIX FTP client. Batch mode, for example, provides the capability to select items in various directories and transfer them all at once. The advantage is that you don't have to wait for one file to transfer before moving to another directory. And because ftptool is a windowed application, you can iconify it or move to another window while the batch transfer takes place.

In addition to enabling you to make better use of your time, ftptool also makes intelligent use of computer resources. Specifically, you can use a cache file to store remote directories rather than retrieve them repeatedly; users of the .netrc file will be glad to know that ftptool can read the .netrc (for more details on .netrc, see "Advanced Use," later in this chapter).

For those who prefer (or require) a more generic X-client, there is the moxftp distribution. This name comes from the three versions of the client: mftp (Motif), oftp (OpenWin), and xftp (Athena). Most versions of UNIX can run at least one of these versions.

For the command-line diehard, ncftp provides a much nicer interface than the standard FTP client. The ncftp client features the following :

• No more typing anonymous plus your e-mail address every time you want to FTP anonymously. You don't have to have the site in your .netrc file.
  
  
  
• No more typing complete site names. Sites in your .netrc can be abbreviated. Type o wuar to call wuarchive.wustl.edu.
  
  
  
• Use your pager (such as more) to read remote files (and also compressed files).
  
  
  
• Use your pager to view remote directory listings.
  
  
  
• Transfers feature a progress meter.
  
  
  
• You can keep a log of your actions. See what the path was of that file you downloaded yesterday so that you don't have to look for it today.
  
  
  
• Comes with a built-in mini-nslookup.
  
  
  
• The ls command is ls -CF. Some FTP ls commands were identical to the dir command, but ls is more flexible, so that you can do things like this:
  
  
  

ls -flags directory

• You can use the redial command for a remote host until you connect.
  
  
  
• Don't feel like typing a long filename? Use a wildcard in single-file commands such as get and page.
  
  
  
• Supports colon mode so that you can type ncftp cse.unl.edu:/pub/foo, to copy foo into your current directory.
  
  
  
• You can redisplay the last directory listing without getting it across the network.
  
  
  
• Detects when new mail arrives.
  
  
  
• ncftp is quieter by default; who cares if the PORT command was successful—if you do, turn verbose on. :-)
  
  
  

FTP by Mail

If you do not have direct access to the Internet but can send e-mail, you can still make use of FTP. Several services are available that provide an e-mail interface to FTP. One such service, BITftp, is for users of Bitnet only (other users are rejected). To obtain a help file, send mail to bitftp@pucc.bitnet with help as the one-line message.

Another useful example is FTPmail, which is located at Digital Equipment Corp (DEC) in Palo Alto, California. FTPmail was written by Paul Vixie at DEC, and is in use at the following sites:

ftpmail@decwrl.dec.com

ftpmail@src.doc.ic.ac.uk

ftpmail@cs.uow.edu.au

ftpmail@grasp1.univ-lyon1.fr

ftpmail@ftp.uni-stuttgart.de

As with BITftp, you can get information from these servers by sending the help message to ftpmail@gatekeeper.dec.com. (Table 22.5 contains examples from that help file.)

Table 22.5. Examples of FTPmail requests.

Desired Function

Body of Mail Message

To send an FTP request to the mailserver, you use one connect command and no more than one cd command. The commands are sent in the body of your mail message; the subject line can be blank or used for something like job requests (the mailserver returns the same subject line you send).

Unless you specify otherwise, the results of your request are sent back in chunks of 64,000 characters. Note that binary transfers must be preceded by the binary command, otherwise an ASCII transfer is attempted. When the binary mode is selected, automatic btoa (binary-to-ASCII) encoding takes place (uuencode can be specified).

Advanced Use

Even though many sophisticated tools simplify the use of FTP, knowledge of some advanced FTP features can make the normal interface preferable in some ways. For example, mget and mput provide a means to transfer multiple files; their nominal syntax is as follows:

mget {list of files}

mput {list of files}

However, they also can be used in conjunction with wildcards to ease the difficulties associated with long filenames:

ftp> mget alt*

mget alt.sex.bestiality.barney.Z?

The use of wildcards varies, so remember that the server may have a different syntax than the client. Although the mput command is expanding wildcards on the client side, mget wildcard functionality is based on the server's syntax.

Another useful trick is to use — as the name of the file to be downloaded. For example, if you want to look at a README file, you could type this:

get README —

This command tells the FTP client to output the received file to standard output (your screen). The only problem with this method is that the entire file is dumped to the screen without pause. On a short file, this may be acceptable if you can scroll back your window buffer, but on longer files it can be cumbersome.

Alternatively, if you are on a UNIX system, you can use the pipe (|) operator to read a file directly into a pager. That is, when you use get to retrieve a readable file, you can use |more as the name of the file you want to "save." For example:

get README |more

This syntax downloads the README file, but rather than saving another README file in your current directory, the command pipes the file to screen using the more program. Of course, if you prefer the less pager, feel free to use that program instead: it's more or less the same. :-)

Lastly, note that although the file is not stored on the local system using these techniques, the entire file nonetheless travels across the network. If you have to look at a long file several times, it is probably best to download it and then view it as you normally would. You can view the file without terminating your FTP session by using the shell escape character(!). An example of this technique follows:

get README readme.new

150 ASCII data connection...

1634 bytes received in 0.052 seconds (30 Kbytes/s)

ftp> !more readme.new

Once you are finished reading the file, or if you quit out of the more program, you return to the FTP prompt. If you don't take too long reading the file (generally, you have 15 minutes), you can continue your FTP session. If you do take a long time between sending commands to the remote host, the session is terminated.

To simplify the process of logging in to an FTP site, the UNIX client reads a startup file called .netrc in your home directory. The client scans the file for an entry matching the site to which you are currently connected; if one is found, the information is passed to the server. Note that if the .netrc file contains the password token and the file is readable by anyone other than you, the process aborts.

The format of .netrc is as follows:

machine {ftphost}

login {username}

password {password}

{optional macro}

Here is an alternative format for .netrc:

machine {ftphost} login {username} password {password}

{optional macro}

The FTP client parses the machine tokens for an entry matching the site to which you are trying to connect. Once it finds an entry, it continues to process the entry until another machine token is reached (or an EOF is encountered). The login and password strings are passed to the server, saving you the trouble of entering them each time. However, you should use the password token only on anonymous sites.

Another convenience offered by the .netrc file is its ability to use macros. A practical use of the macro feature is to change the directory; you can also use a macro to get a file. For example, the following .netrc entry logs the user into the site, changes the directory, and gets a file:

machine ftp.cadence.com login anonymous password joe@

macdef init

cd jobs

get joblist

quit

The macdef command functions the same as in interactive use: it defines a macro. The init macro is a special case, however, because it is invoked automatically at login. Therefore, this entry is essentially an automated procedure for downloading the joblist file.

Note that the macdef function requires a null line to mark the end of the macro. This also applies to the last entry in the .netrc—where it is often forgotten.

Another "automagic" function is the built-in compression in the wuarchive server. Compressed files can be automatically uncompressed before downloading. To activate this feature, omit the extension from the compressed file when specifying the get command. For example, if the file is called foo.bar.Z and you specify get foo.bar, the file is uncompressed automatically. This function is not available on the standard servers and is one good argument for installing your own server.

REFERENCE WORKS

CERT, "FTP security notes," FTP at cert.org:/pub/tech_tips/anonymous_ftp, June 1993.

____, "Wuarchive Ftpd Vulnerability," FTP at cert.org:/pub/cert_advisories/CA-93:06.wuarchive.ftpd.vulnerability, June 1993.

____, "AIX FTP Vulnerability," FTP at cert.org:/pub/cert_advisories/CA-92:09.AIX.anonymous.ftp.vulnerability, October 1992.

____, "Ftpd Hole," FTP at cert.org:/pub/cert_advisories/CA-88:01.ftpd.hole, January 1988.

Czarnik, Tom, "ftp.list" FTP at askhp.ask.uni-karlsruhe.de:/pub/info/ftp.list.sites.Z, April 1993.

December, John, "Internet Tools," FTP at ftp.rpi.edu:/pub/communications/internet-tools.txt, December 1993.

Gleason, Mike, "Ncftp Blurb," FTP at world.std.com:/src/network/ncftp/Blurb, September 1993.

Granrose, Jon, "ftp-list" FTP at ftp.gsfc.nasa.gov:/pub/ftp-list, December 1991.

Lemson, David, "Compression Notes," FTP at ftp.cso.uiuc.edu:/doc/pcnet/compression, July 1993.

Postel, J., and J. Reynolds "File Transfer Protocol," RFC 959 FTP at nic.merit.edu:/documents/rfc/rfc0959.txt, October 1985.

Rovers, Perry, aftp-list.FAQ, FTP at oak.oakland.edu:/pub/msdos/info/ftp-list.zip, January 1994.