FTPEXT Working Group Internet Engineering Task Force INTERNET-DRAFT D. Borman Berkeley Software Design, Inc. Expires 21 September 1997 R. Adams UUNET 21 March 1997 REST Command and Extensions to FTP Status of this memo This document is an Internet-Draft. Internet-Drafts are working documents of the Internet Engineering Task Force (IETF), its areas, and its working groups. Note that other groups may also distribute working documents as Internet-Drafts. Internet-Drafts are draft documents valid for a maximum of six months and may be updated, replaced, or obsoleted by other documents at any time. It is inappropriate to use Internet- Drafts as reference material or to cite them other than as ``work in progress.'' To view the entire list of current Internet-Drafts, please check the "1id-abstracts.txt" listing contained in the Internet-Drafts Shadow Directories on ftp.is.co.za (Africa), ftp.nordu.net (Northern Europe), ftp.nis.garr.it (Southern Europe), munnari.oz.au (Pacific Rim), ftp.ietf.org (US East Coast), or ftp.isi.edu (US West Coast). 1. Overview This memo describes changes to FTP [1], which were proposed by Rick Adams in May of 1989, to allow the RESTART mechanism to be used with STREAM mode transfers. Along with this, two new optional commands are added, MODIFICATION TIME (MDTM) and SIZE OF FILE (SIZE). All these changes have been implemented, and are in use today in many production FTP implementations. 2. Motivation The FTP specification has three modes of data transfer, Stream, Block and Compressed. To avoid having to resend the entire file if the file is only partially transferred, both sides need some way to be able to agree on where in the data stream to restart the data transfer. In Block and Compressed mode, the data that is transferred over the data connection is formatted, allowing the imbedding of restart markers into the data stream. The sending side can send a restart marker with whatever information it needs to be able to restart a file transfer at that point. The receiving side can keep a list of these restart markers, and corrolate them with how the file is being saved. To restart the file transfer, the receiver just sends across that last restart marker, and both sides know how to resume the data transfer. Note that there are some flaws in the description of the restart mechanism in RFC 959, see RFC 1123 [2] for the corrections. Borman & Adams Expires 21 September 1997 [Page 1] INTERNET-DRAFT REST Command and Extensions to FTP 21 March 1997 2.1 Restarting in STREAM Mode In Stream mode, the data that is transferred over the data connection is just a stream of unformatted bytes of data, hence explicit restart markers cannot be inserted into the data stream. For this reason, the FTP specification does not provide the ability to do restarts in Stream mode. However, there is not really a need to have explicit restart markers in this case, as restart markers can be implied by the byte offset into the data stream. If the data representation type is IMAGE, for many systems the file will be stored exactly in the same format as it is sent across the data connection. It is very easy for the receiver to determine how much data was previously sent, and notify the sender the byte offset where the transfer should be restarted. If the User FTP is doing the retrieve, it will directly calculate the restart marker, and send that information in the RESTart command. However, if the User FTP is sending the file, it needs to be able to determine how much data was previously sent. A new FTP command is needed to get this information. 2.2 Determining File Size A new FTP command, SIZE OF FILE (SIZE), is used to get the size of a file on the remote side. If the data representation type is not IMAGE, then how the file is stored may not match how it will look when transferred over the wire. It is because of this that the implied restart marker is a byte offset into the data stream, and the SIZE of a file may change depending on what is the current mode of transfer. The receiver will need to read the partially transferred file, do any appropriate conversion, and count the number of bytes that would be generated when sending the file. This is the byte offset into the data stream, and is what is conveyed to the sender. The sender will need to read the file, do the appropriate conversion, and skip throw away as much of the data stream as the sender requested. At this point, both sender and receiver will be at the same point in the data stream, and the transfer of the file can be continued. When doing a restart of a file transfer, it is very possible that the file may have been modified on the senders side between when the file was originally transferred, and when the restart is being done. In this case, it would be better to resend the entire file. The FTP protocol does not provide a mechanism to automatically detect this situation. However, with the addition of one new FTP command, this can be done. 2.3 RESTarting and Source File Modification The new FTP command, MODIFICATION TIME (MDTM), can be used to determine when the file on the remote side of the connection was last modified. When attempting to restart a RETRieve, if the User FTP does a MDTM command, it can check and see if the modification time of Borman & Adams Expires 21 September 1997 [Page 2] INTERNET-DRAFT REST Command and Extensions to FTP 21 March 1997 the source file is more recent than the modification time of the partially transferred file. If it is, then most likely the source file has changed and it would be unsafe to restart in the middle of the file transfer. When attempting to restart a STORe, the User FTP can do a MDTM command to find out the modification time of the partially transferred file. If it is older than the modification time of the file that is about to be STORed, then most likely the source file has changed and it would be unsafe to restart in the middle of the file transfer. Note that this is applicable to any RESTart attempt, regardless of the mode of the file transfer. 3. Description This section describes the specific changes that need to be made to the FTP specification to add the changes described in Section 2. Note that the FTP specification is modified by RFC 1123 [2], specifically RFC 959 has errors in its description of the Restart mechanism. 3.1 MODIFICATION TIME In Section 4.1.3, "FTP SERVICE COMMANDS", insert: MODIFICATION TIME (MDTM) This command returns the time the file named in its argument field was last modified. The time is returned in ISO 3307 "Representation of the Time of Day" format. This format is YYYYMMDDHHmmSS or YYYYMMDDHHmmSS.xxx, where YYYY is the year MM is the month (01-12) DD is the day of the month (01-31) HH is the hour of the day (00-23) mm is the minute of the hour (00-59) SS is the second of the hour (00-59) xxx if present, is a fractional second and may be any length Time is expressed in UTC (GMT), not local time. 3.2 SIZE OF FILE (SIZE) In Section 4.1.3, "FTP SERVICE COMMANDS", insert: SIZE OF FILE (SIZE) This command returns the size of the file named in its argument field. The value returned is in a format suitable for use with the RESTART (REST) command. This format will change depending on the current MODE and TYPE of the connection. E.g. For mode stream and type image, it would be Borman & Adams Expires 21 September 1997 [Page 3] INTERNET-DRAFT REST Command and Extensions to FTP 21 March 1997 a count of the number of octets that would be transferred with the RETRIEVE (RETR) command. This command is normally used in conjunction with the RESTART (REST) command. 3.3 ERROR RECOVER AND RESTART In Section 3.5, "ERROR RECOVERY AND RESTART", second paragraph, replace: The restart procedure is defined only for the block and compressed mode of data transfer. It requires the sender... with: The restart procedure is defined for the block, compressed and stream mode of data transfer. Block and compressed mode require the sender... At the end of Section 3.5, "ERROR RECOVERY AND RESTART", insert: STREAM MODE transfers with FILE STRUcture may be restarted even though no restart marker has been transferred in addition to the data itself. This is done by using the SIZE command followed by the RESTART (REST) command. When using TYPE ASCII or IMAGE, the SIZE command will return the number of bytes that would actually be transferred if the file were to be sent between the two systems. I.e. with type IMAGE, the SIZE normally would be the number of octets in the file. With type ASCII, the SIZE would be the number of characters in the file INCLUDING any characters that would be inserted during the CR-LF expansion. 5. References [1] Postel, J., and Reynolds, J., "FILE TRANSFER PROTOCOL (FTP)", RFC 959, ISI, October 1985 [2] Braden, R., editor, "Requirements for Internet Hosts -- Application and Support", RFC 1123, October 1989 Author's Address David A. Borman Berkeley Software Design, Inc. 4719 Weston Hills Drive Eagan, MN 55123 Phone: (612) 405-8194 Email: dab@bsdi.com Rick Adams UUNET Technologies, Inc. 3110 Fairview Park Drive, Suite 570 Borman & Adams Expires 21 September 1997 [Page 4] INTERNET-DRAFT REST Command and Extensions to FTP 21 March 1997 Falls Church, VA 22042 USA Phone: +1 703 204 8000 Email: rick@uunet.uu.net Borman & Adams Expires 21 September 1997 [Page 5]