M. Saul Internet Draft Mentat Document: draft-saul-ftp-plus-00.txt August 2000 Category: Informational FTP Plus Status of this Memo This document is an Internet-Draft and is in full conformance with all provisions of Section 10 of RFC2026. Internet-Drafts are working documents of the Internet Engineering Task Force (IETF), its areas, and its working groups. Note that other groups may also distribute working documents as Internet- Drafts. Internet-Drafts are draft documents valid for a maximum of six months and may be updated, replaced, or obsoleted by other documents at any time. It is inappropriate to use Internet-Drafts as reference material or to cite them other than as "work in progress". The list of current Internet-Drafts can be accessed at http://www.ietf.org/ietf/lid-abstracts.txt. To view the list Internet-Draft Shadow Directories, see http://www.ietf.org/shadow.html. Forward The terminology used in this document is a bit television centric and thus may be somewhat unusual to the internet community. The reader is referred to a joint task force report of the Society of Motion Picture and Television Engineers (SMPTE) and the European Broadcast Union (EBU) [1] as background material. Also, this document may refer to itself as a "standard", in which case it is referring to its potential status within SMPTE and not necessarily IETF. 1. Introduction The transfer of files containing audio/video essence and metadata is a fundamental and important issue. The computer industry has developed several standards which specify methods for performing file transfer. However, there are problems and requirements unique to the video production industry which are not fully addressed by the existing computer industry standards. To overcome these limitations, and in order to guarantee interoperability in the Saul Informational - February 2001 [Page 1] FTP+ August 2000 interface, network and protocol domain for file transfer, it is recommended that the industry follow the rules and guidelines defined in this document. This document presents methods for adapting current computer industry standards and presents a new standard. The use of the ubiquitous File Transfer Protocol (FTP) is described as it relates to transferring large video and audio files. A new file transfer protocol, FTP+, is defined which provides a number of additional functions. FTP+ builds on FTP and uses the same base set of commands as FTP. FTP+ includes new commands that enable additional features and also provide the ability to embrace new network protocols as they become available. This document contains details of these new FTP+ commands. 2. Scope Certain requirements for transferring video and audio files cannot be met using existing protocols. Some of these requirements include: . Transferring large files whose length exceeds the maximum value which can be expressed within a 32-bit variable. (This can be either 2 or 4 gigabytes, depending on whether the variable is signed or unsigned). . Transferring parts of a file or asset essence. . Using network interfaces other than TCP/IP (for example, XTP [2]). . Performing transfers simultaneously to multiple destinations (point-to-multipoint transfers). . Setting of a maximum transfer rate. This document defines several command sets which can be added to the FTP Protocol in order to facilitate these requirements. It is not necessary for an FTP+ implementation to support all of the command sets. However, an implementation must use the STAT command to indicate which command sets are supported. The following items are not within the scope of this document: . Details of file formats. . File transfer protocols other than FTP. 3. Definition of terms Saul [Page 2] FTP+ August 2000 FTP+ Profile: A set of FTP commands which supports certain features or requirements. Currently two FTP+ Profiles are defined: General (GEN) and XTP. 4. Enhanced FTP: "FTP+" 4.1 Overview of FTP+ The FTP+ enhancements described in this standard are designed to extend the functionality of FTP and to allow for different transport and network layers for data transfer. FTP+ shall be based on (but not limited to) the FTP RFC 959 specification [3]. FTP+ should be considered a strict superset of FTP in the sense that FTP+ shall be 100% compatible with RFC 959 for the GEN profile. Understanding RFC 959 is imperative for understanding FTP+. An FTP server may be included with the OS as installed on most devices. However, an FTP+ server may replace the FTP server. An FTP+ server is simply an FTP server with added capabilities; an FTP client interoperates with an FTP+ server in the same way that it interoperates with a standard FTP server. Likewise, an FTP+ client is an FTP client with added capabilities and can interoperate with FTP servers using the standard FTP protocol commands. The FTP+ client can determine which FTP+ profiles the server supports by parsing the results of the STAT command. Since a standard FTP server won't support any FTP+ profiles, the FTP+ client shall make sure that only standard FTP commands are sent to that particular server. In their basic forms, both FTP and FTP+ use a data path for the actual transfer of data and a control connection for the set up, tear down, and status of the data transfers; however, for FTP+, the data path must use TCP. Both FTP and FTP+ support the normal client/server model and the less used proxy client model. To understand how FTP+ changes the features it has copied from FTP, it is important to recognize the environment in which FTP+ exists. Figure 1 shows how the FTP control path and data path exists in relation to the layers below it. Figure 1 also illustrates how the control path used by FTP+ exists in relation to the lower layers, since FTP+ shall always use TCP for the control path. The FTP+ data path environment is illustrated in Figure 2. Saul [Page 3] FTP+ August 2000 ------------------------------------------------------ |Application layer (Client) Application | | | | | V | |Protocol Layer FTP | | | | | V | |Transport Layer TCP | | | | | V | |Network Layer IP | | | | | V | |Link/Physical Layers Any Link/Physical Layer | ------------------------------------------------------ Figure 1. Protocol, Network, and Transport layers for FTP Client or Server (FTP control and data path, FTP+ control path). Note: The application layer shown in Figures 1 and 2 shall be present only for the client side. ------------------------------------------------------ |Application layer (Client) Application | | | | | V | |Protocol Layer FTP+ | | -------------- | | | | | | V V | |Transport Layer TCP XTP | | -------------- | | | | | | | V | | |Network Layer IP | | | | | | | V V | |Link/Physical Layers IP Link XTP Link | ------------------------------------------------------ Figure 2. Protocol, Network, and Transport layers for FTP+ Client or Server (File-Data path). As shown in Figure 2, the protocol layer may access either the TCP or XTP link for a given file-data transfer session, but the FTP+ enhancements described in this standard are specific to the choice of transport layer used. Ideally, one common set of enhancements would suffice for use with any of the different transport choices. Saul [Page 4] FTP+ August 2000 Unfortunately that would make the actual use of the transfer protocol ambiguous and difficult to use. For instance, some of the enhancements required for FTP+'s full use of XTP are not allowable for any other transport layer and vice versa. Therefore, the FTP+ enhancements include two basic profile sets. Each set is designed with particular transport layer types in mind and provides features related to the respective type. The profile sets are described in 4.2. 4.2 FTP+ Profiles Table 1 shows the two FTP+ Profiles, which are defined as follows: FTP+ (General Profile): This is standard FTP as defined by RFC 959 and RFC 1123 plus general enhanced commands for use over TCP or XTP. When used with XTP, the multipoint and max rate functions that are available with XTP are not available for this profile. XTP offers speed advantages even when used as a direct TCP replacement. FTP+ (XTP Profile): This is the General Profile plus XTP specific related commands. This profile supports all XTP features, including multipoint transfers and transfer max rate setting. Table 1 . FTP+ Profiles --------------------------------------------------- | General Profile | XTP Profile related | | | commands | --------------------------------------------------- | STAT (new return | RATE (new) | | values defined) | MCPT (new) | | XPRT (new command) | MCPV (new) | | XPSV (new command) | MCGC (new) | | RETR (added optional | MCGM (new) | | partial file start,| MCGR (new) | | length params) | MCGS (new) | | STOR (added optional | SITE (new RATE | | partial file start,| parameter defined) | | length params) | | | SIZE (new command) | | --------------------------------------------------- Notes on the commands in Table 1: . Some commands have newly defined return values. For example, the STAT command now returns additional values for statistics for the transfer in progress. . The STOR and RETR commands have new optional parameters for partial file transfer. The RETR command determines how much of the input file will be read; for RETR, the start byte offset and Saul [Page 5] FTP+ August 2000 the length of the desired data are now allowed. The STOR command also allows a start byte offset and length. When an existing file is modified by a write, partial file contents are either over- written by the new data or appended to the end of the file, depending on the start byte offset. . There are new commands to support the multipoint feature of XTP. There is also a new command to support the max rate setting available with XTP. . XPRT, XPSV, MCPT, MCPV are new commands related to the existing PORT and PASV commands. . The exact formats for the new and modified commands are described in Clause 5 of this standard. 4.2.1 Usage of the profiles There is no requirement for an FTP+ server process to support all the profiles. The STAT command will return the supported sets of a particular FTP+ server. The decision of which profile to use depends upon the application requirements. The choice is a function of the transfer rate desired, the distance between the local and remote devices, and the stack support available on a given machine. Switching to a different transport layer will often require a change to the FTP+ Profile as well. 4.2.2 FTP+ file transfer using XTP FTP+ includes support for the Xpress Transport Protocol (XTP). XTP exists at the same layer as TCP. It is in many ways a superset of TCP but is not compatible with it. XTP has maximum-rate and other QoS setting features, whereas TCP has no equivalent functions and will move data as fast as possible. Also XTP supports point-to-multipoint transfers over IP. XTP is an open protocol with a history of use and stability. XTP is being selected to provide functionality that is lacking in TCP but is required to meet user needs. 5. FTP+ protocol specification 5.1 FTP+ components As with FTP, FTP+ utilizes a control channel and a data channel. FTP+ provides additional capabilities for the data channel as well Saul [Page 6] FTP+ August 2000 as for the identification of the asset to be transferred via the control channel. 5.1.1 FTP+ control channel FTP+ shall always employ a TCP connection for the control channel. FTP has been registered to listen for control channel connections on TCP port 21. FTP+ shall also use TCP port 21 for the control channel. FTP+ client programs shall always first establish a TCP connection to this port. 5.1.2 FTP+ data channel FTP+ provides support for XTP transport as well as TCP and also for IPv6 networks in addition to IPv4. With the addition of XTP transport comes the ability for data channels to be multicast channels that involve more than two hosts. As with FTP, FTP+ supports both simple client/server connections as well as "two server" configurations in which data is transferred between two FTP+ servers using two control channels. These "two server" configurations may be of particular interest to automation systems. For example, an external automation program running on a workstation could use a "two server" configuration to initiate a transfer between two video servers or between a video server and library management server. In the case of XTP multicast transfers, the data channel will be "connected" to multiple receiving servers simultaneously. By nature, these multicast transfers are always done as "pushes". That is, each receiving server is instructed to join a multicast group using a particular multicast address. Once every receiver has successfully joined the multicast group, the sending server can write data to this multicast data channel and it will be received by each receiving server. NOTE . File transfer between different operating systems: Care must be used when moving files between machines with different operating systems. For example, the Windows NT File System (NTFS) only supports case-insensitive file names while UNIX systems have case-sensitive names. Also, the file access permissions in NTFS differ from those in UNIX. 5.1.3 FTP+ threads Certain FTP+ implementations may designate a thread to handle data connection requests. This allows the control connection to respond to status requests while a data transfer is in progress. If threads are used, care should be taken that there is no more than one outstanding data request at a time. It is also necessary to synchronize the reading of the control connection between the Saul [Page 7] FTP+ August 2000 threads. While it is not required to implement threads, an FTP+ implementation must indicate whether or not it can respond to status requests while a data connection is in progress. Refer to the STAT command for more details. 5.2 Common features of FTP+ and FTP This clause documents some areas in which FTP+ shall use identical features as specified by FTP in RFC 959. 5.2.1 Error code syntax and interpretation The reply code syntax specified in RFC 959 section 4.2 shall also be used by FTP+. Some of the new FTP+ commands define additional reply codes which adhere to the FTP reply code syntax. These new reply codes are presented in 5.4, "FTP+ Commands," along with the FTP+ command descriptions. 5.2.2 Multi-line control channel responses In some cases it is necessary to send multiple lines of response over the control channel (for example, the output of a STAT command which is returning information about many files). The mechanism for sending a multi-line textual response over the control channel specified in RFC 959 section 4.2 shall be used by FTP+. 5.3 FTP+ commands The FTP+ commands are presented below in two groups: Transfer Protocol commands and System Status commands. An FTP+ server shall choose to implement the FTP+ commands based on a profile which it wishes to support. The command descriptions below include the profile(s) which apply to each command. In the following FTP+ command descriptions, angle brackets "<>" are used as command argument delimiters. Optional arguments are indicated by square brackets []. 5.3.1 Transfer protocol commands These commands are used to allow FTP+ to support transport protocols other than TCP on the data channel. 5.3.1.1 PORT h1,h2,h3,h4,p1,p2 Profile: GEN This is the normal RFC 959 command, duplicated without modification. It is provided so that existing code which performs FTP file Saul [Page 8] FTP+ August 2000 transfer operations can still be used with the FTP+ daemon without modification. The argument is a HOST-PORT specification for the data port to be used in a data connection. There are defaults for both the user and server data ports, and under normal circumstances this command and its reply are not needed. If this command is used, the argument is the concatenation of a 32-bit Internet host address and a 16-bit TCP port address. This address information is broken into 8-bit fields and the value of each field is transmitted as a decimal number (in character string representation). The fields are separated by commas. In the port command: PORT h1,h2,h3,h4,p1,p2 h1 is the high order 8 bits of the Internet host address. 5.3.1.2 XPRT Profile: GEN,XTP The XPRT (eXtendedPoRT) command supplements the RFC 959 command, PORT, serving a similar functional purpose yet offering enhanced capabilities and a different set of arguments. Namely, it specifies to the receiving FTP+ process a network end-point to which a data connection shall be established. However, XPRT is generalized to allow the description of non-IPv4 networks and non-TCP transport addresses for the data connection. The protocol_specifier will describe the transport and network protocols to be used. The end_point_address will be the actual network address for the network and transport given that should be used for the data channel. The protocol_specifier will be given as a "/" (slash) followed by a string and optionally another "/" and another string, with the first string being a transport_specifier and the second optional string being the network_specifier. // The network_specifier values defined are: "IP4" for IPv4 and "IP6" for IPv6. The transport_specifier values defined are "TCP" and "XTP" which specify the transport layer as either TCP or XTP. The network_specifier is optional because XTP supports a native mode in which no additional network protocol is used. However, XTP can be run on top of IP or some other network protocol. The end_point_address specifies the network end-point. This must include both the network and transport endpoints (for example, an IP address and a TCP port number). The syntax of this string is dependent upon the network/transport being used. In the case of "/TCP/IP4", "/XTP/IP4", or "/XTP", it will be an IP address followed by a ":" (colon) and the dotted decimal 32-bit port number. The format of this string will be the same as the PORT command described Saul [Page 9] FTP+ August 2000 in 0. In the case of "/TCP/IP6" or "/XTP/IP6", it will be an address in string representation as specified in RFC 2373 followed by a ":" (colon) and the decimal port number. XPRT examples: . The XPRT command specifies the use of TCP transport over IP version 4 networking and issues a data connection at the IP version 4 network address of 192.10.20.1 using TCP port 3456: XPRT /TCP/IP4 192.10.20.1:3456 . The XPRT command specifies the use of raw XTP transport and issues a data connection to the address 192.148.20.17 at port 1958. (XTP, when used as both transport and network layer, uses the same IP version 4 network address definition and a 2 byte port number, but it does not use IP packets.): XPRT /XTP 192.148.20.17:1958 . The XPRT command specifies the use of TCP transport over IP version 6 networking and issues a data connection at the IP version 6 network address 10A0:0:0:0:0:0.192.20.148.17 using TCP port 3458: XPRT /TCP/IP6 10A0:0:0:0:0:0.192.20.148.17:3458 Upon receipt of a valid XPRT command, the FTP+ process must respond with the standard "200 Command OK". The error code 501 would indicate a syntax error in the parsing of the XPRT arguments (e.g., insufficient arguments or an invalid network or port syntax). The new error code 522 would indicate that the FTP+ process does not support the transport/network combination specified. The text portion of the 522 error must include the list of transport/network protocols combinations supported by the FTP+ implementation, enclosed in parenthesis and separated by commas and followed by language or implementation specific text. For example, the response to an invalid XPRT might be: 522 (/XTP, /TCP/IP4) Supported protocols This would indicate that only raw-mode XTP and TCP over IP version 4 are supported by that FTP+ implementation. 5.3.1.3 PASV Profile: GEN This is the normal RFC 959 command, duplicated without modification. It is provided so that existing code, which performs FTP file Saul [Page 10] FTP+ August 2000 transfer operations, can still be used with the FTP+ daemon without modification. 5.3.1.4 XPSV [] Profile: GEN,XTP The XPSV (eXtendedPaSsiVe) command replaces the RFC 959 command PASV command, serving a similar purpose yet offering enhanced capabilities and a different set of arguments. Namely, it requests that the receiving FTP+ process begin to listen for an incoming data connection and to respond to this command with the network address on which it is listening. In FTP+, the response to a XPSV command will be an address specification corresponding to the transport/network protocol passed as the transport_specifier (see the XPRT command for a description of the syntax of the transport_specifier). This address specification shall be returned in the same format as required for use as the arguments for a corresponding XPRT command. The response to a valid XPSV command must be a "229" code followed by XPRT-format network/transport protocol specification. The optional interface_name may be used when connecting to a multi- homed FTP+ server. In some cases it may be ambiguous as to which network interface the server should begin listening. In such cases this string may be used to specify the interface. See 5.6.3 for a discussion of multi-homed host issues. Here is an example XPRT command followed by a "229" response: XPRT /XTP/IP4 229 (/XTP/IP4 192.20.10.1:2334) Entering passive mode This indicates that the FTP+ process on the host 192.20.10.1 is listening for a data connection on the XTP port 2334. The error code 501 indicates an error in parsing the network or transport arguments (e.g., insufficient arguments). The error code 522 is used to indicate that an invalid transport/network combination was specified. See the XPRT command for a description of how this error code is used. 5.3.1.5 XPRT and XPSV examples Here is an example of how the XPRT and XPSV commands set-up a data connection between two FTP+ processes by a separate client process running on a third host (i.e., a proxy FTP+ session). First the client process establishes TCP control channels to both FTP+ processes and login to both FTP+ servers. Assuming the client wishes to perform an XTP transfer over IPv4, it will then send a XPSV command to the first FTP+ process. It will then use the response to this XPSV command as the arguments of a XPRT command to Saul [Page 11] FTP+ August 2000 the second FTP+ process. At this point a STOR or RETR command can be issued to either FTP+ process and the second FTP+ process will establish the IPv4/XTP connection to the first FTP+ process. This interaction is shown below. Commands are shown in bold text and responses in non-bold Italics. Host 192.20.10.1 Host 192.20.10.2 XPSV /XTP/IP4 229 (/XTP/IP4 192.20.10.1:4623) Entering passive mode XPRT (/XTP/IP4 192.20.10.1:4623) 200 Command Okay 5.3.1.6 STOR [] [] Profile: GEN The STOR command is enhanced by FTP+ to include two new optional arguments, the start_byte_offset and the length of bytes to be transferred. When the start_byte_offset is used, the STOR command will begin writing data to an existing file at the indicated offset. Note that for multicast transfers the length argument can be used to determine the end of file. The reply text for a STOR shall be enhanced by FTP+ to include the number of bytes written by the receiving FTP+ server. This will allow a client program to verify that the entire file was received. The format of this reply message shall be: 226 Transfer completed. 12345678 bytes written. 5.3.1.7 RETR [ [] ] Profile: GEN The RETR command is enhanced by FTP+ to include two new optional arguments, the start_byte_offset and the length of bytes to be transferred. The start_byte_offset indicates the offset to begin reading the input file. 5.3.2 System information commands The following commands are used to provide additional information needed about a FTP+ system's capabilities. 5.3.2.1 STAT Saul [Page 12] FTP+ August 2000 Profile: GEN The STAT command has been expanded to include three forms of responses: . When a pathname is not provided, system information is returned. . When a pathname is provided and a transfer is in progress, the status of the transfer operation is returned. . When a pathname is provided and no transfer is in progress, file attributes are given for the indicated pathname, similar to a LIST command. Responses to the STAT command must be on the control channel. When no pathname is provided, STAT shall return a 211 response code followed by FTP+ server information. This version of the STAT command shall show whether threading is supported, which FTP+ profiles are supported, what transport/network protocol combinations are supported, and whether 64 or 32 bit file sizes are supported. For example, the following reply indicates that the FTP+ implementation supports threading, XTP capabilities, and runs only on IPV4 networks with both TCP and XTP transports using 64 bit files: 211- server.acme.com FTPPLUS server status Version X.X ACME_OS 01/25/98 Connected to foo.bar.com (10.20.1.2) Logged in as username TYPE:BIN,FORM:Nonprint;STRUcture:File;transfer MODE:Stream Threading:YES Profiles:XTP Protocols:/TCP/IP4,/XTP/IP4 Size:64 211 End of status In the above example, the value for threading shall be either YES or NO. (See 5.1.3 for more explanation of FTP threads.) The value for Profiles must be either GEN or XTP. Protocols shall be indicated by a list of protocol_specifiers, separated by commas. The value for Size must be either 64 or 32. The second form of the STAT command response occurs when a pathname is provided and a file transfer is in progress. If threading is not supported then this form will never be used, since the file transfer will have completed before a response is issued. If threading is supported, the format of this response must be: 213 Status of filename: xxx of yyy bytes transferred; estimated time remaining hh:mm:ss Saul [Page 13] FTP+ August 2000 The third form of the STAT command response occurs when a pathname is provided and no file transfer is in progress. If a partial pathname is given, the server may respond with a list of file names or attributes associated with that specification. The response must be: 213- Status of filename: system specific file information (similar to the LIST command) 213 End of status 5.3.2.2 SITE RATE Profile: XTP This extended variant of the RFC 959 SITE command will return some additional information regarding the features provided by the XTP Profile. This information is returned as text responses to these commands. The response to the SITE RATE command shall be: 200 bytes per second maximum rate Where maximum_rate is the decimal number of bytes per second that is largest value accepted as an argument to the FTP+ RATE command. 5.3.2.3 SIZE [network_size] Profile: GEN The SIZE command is useful in obtaining the size in bytes of a remote file in a standardized format. If the network_size parameter is not indicated the network_size shall default to 1, where the remote server sends the size of the file as it appears on the network according to the current TYPE value. The response to the SIZE command shall be: 213 where bytes is the size of the file in bytes. The proper values for the optional parameter network_size are: . 1 - send the file size as it would appear on the network, according to the current TYPE value. For example, if the current type were ASCII, then the size would be equivalent to the length of the file if it were converted to netascii. If the current type were IMAGE, then the size would be the same as reported by the LIST command. Saul [Page 14] FTP+ August 2000 . 0 - send the file size as it appears on the local disk. This option is valuable for reporting the file's status. 5.3.3 Overview of multicast transfers Multicast transfers are supported by XTP, either in "raw" mode or over IP. However, FTP+ has been designed to encompass other transports which may support multicast data transfers. To perform a multicast transfer, one server must first create a multicast group to which other receivers will then join. The multicast group will be identified by a unique address that the sending server determined when it created the group. When the sending server later sends any data to this address, the underlying transport protocol transmits this data in packets that will be received by each member of the group. The XTP protocol was designed with the intention of supporting this type of operation. Specifically, it provides capabilities for multiple receivers to acknowledge the receipt of these multicast packets and to request retransmission of lost packets if necessary. 5.3.4 Multicast transfer commands These commands are used when the XTP option is being supported to provide multicast transfer capabilities. 5.3.4.1 RATE [] Profile: XTP The RATE command is added by FTP+ to allow a client to set the maximum transfer rate that will be used during the data transfer. The rate_value specifies the maximum average transfer rate and burst_value specifies the maximum burst transfer rate. Both of these values are given as decimal bytes per second to be used in a transfer. The SITE RATE command (see 0) may be used to determine the maximum value supported by an FTP+ server. This command is only applicable to XTP, which inherently supports this type of rate throttling. 5.3.4.2 MCGC Profile: XTP The MultiCastGroupClose command shall be used to close the multicast data connection. Normally this connection should be left open until a MCGC command is received. The MCGC command takes no parameters. The response to the MCGC command will be: 220 Passive multicast connection closed Saul [Page 15] FTP+ August 2000 or 220 Multicast connection closed 5.3.4.3 MCGR Profile: XTP The MultiCastGroupReport command may be used to retrieve a list of host IP addresses which are still connected to the group. This is useful for reporting purposes. This command may be issued to the FTP+ sending server any time that the connection is still open. Note: The reply will be on the control channel. The format of the reply will be: 230 host_count ip_address1,ip_address2,...ip_addressn or 425 0 - No group list available 5.3.4.4 MCGS Profile: XTP The MultiCastGroupSecurity command shall be used to send a list of host IP addresses which will be allowed to connect to the group. This is useful for security purposes. This command must be issued to the FTP+ sending server after the MCPT has been issued to all of the receiving FTP+ servers. The parameters for MCGS consist of a host_count followed by a list of host IP addresses; each IP address should be separated by commas. Note: It is acceptable to send a host_count of 0. The format of the reply will be: 200 Multicast Group forming OK or 425 Error in multicast group forming 5.3.4.5 MCGM leaves [Yes | No] Profile: XTP The MultiCastGroupManagment command controls (and shows) what the policy should be for FTP+ servers which attempt to join or leave a multicast group while a transfer is in progress. If set to "No" then the sending server will abort the transfer and will return an error code and message on the control channel. If set to "Yes" and at least one receiver still remains in the group, then the transfer will continue normally. If given with no arguments, the response to the MCGM command should be the current setting. Saul [Page 16] FTP+ August 2000 5.3.4.6 MCPV [end_point_address] Profile: XTP The MultiCastPassiVe command is analogous to the XPSV command (see 0). The argument protocol_specifier is exactly the same. The purpose of the MCPV command is to request that the receiving FTP+ create a multicast group and to return the end_point_address for that group so that it can be sent to the receiving FTP+ servers in a MCPT command. Note: It is possible to provide a multicast address as an optional argument. While currently the only transport that can support multicast transfers is XTP, a protocol_specifier is included so that future transports which support multicast can be included. 5.3.4.7 MCPT Profile: XTP The MultiCastPorT command is analogous to the XPRT command (see 5.4.1.2). The arguments protocol_specifier and end_point_address are exactly the same. The purpose of the MCPT command is to make it unambiguous that the end_point_address specified is a multicast address. As described in 5.4.1.2, the FTP+ server which receives a MCPT command does not simply establish a connection to the end_point_address as in the case of a XPRT command. Instead, it must join the multicast group corresponding to that address. The underlying system calls to perform these two tasks will be different. By having a separate MultiCastPorT command, it is clear (without relying on interpreting the address given) that a multicast connection is desired. 5.3.5 Examples of performing multicast transfers The first example presents the non-proxy case where there is client code running on the sending server which wishes to send a file to three receiving FTP+ servers. First, this client code must get a multicast address (i.e., group) to use. Next, it must establish TCP control connections to each of the receiving FTP+ servers. Then it must issue MCPT commands to each of the receiving FTP+ servers, for example: MCPT /XTP/IP 224.0.0.2:1234 200 Command Okay The above command indicates that the multicast IP address 224.0.0.2 and XTP port 1234 will be used for this transfer. For each file that the client sends, it shall do the following: Saul [Page 17] FTP+ August 2000 a. Get the size of the file to send and use it as the length parameter to the STOR command. b. Send a STOR command to each of the receiving FTP+ servers. c. After the end of file has been reached (as currently indicated by the length parameter to STOR), wait for a reply on the control channel from each server in the group. When the client is finished with the multicast data channel, it shall issue an MCGC command to each receiving server and then close the local multicast data channel. The second example presents the proxy-case in which the client code is running on a machine not participating in either the sending or receiving of the data. This client code first establishes TCP control channels to each of the involved FTP+ servers. Next, it issues a MCPV command to the sending FTP+ server, which then returns the multicast address that has been assigned for this transfer. For example: MCPV /XTP/IP 229 (/XTP 224.0.0,1:4623) Entering passive mode At this point, the RATE command may be issued to the sending FTP+ Server. Now the client sends MCPT commands to each of the receiving FTP+ servers using the multicast address returned by the MCPV command, as in the first example. Next, the MCGS command is issued, allowing the sending FTP+ server to decide who will be in the group. For each file that will be sent: . A SIZE command must be issued to the sending FTP+ Server, using the size returned as the length parameter to the STOR command. . A STOR command must be sent to each of the receiving FTP+ servers. . A RETR command must be sent to the sending FTP+ servers, to start the actual transfer. When the transfers are completed, it is possible to issue an MCGR command to get a report of which Servers are still in the group. To complete this example, the client issues a MCGC command to all of the servers in the group. Note: The RATE and MCGR commands are optional, but all of the rest of the commands are mandatory. 5.4 Writing an FTP+ server A particular implementation of FTP+ will likely not support all profiles, transports and network protocols. To do so would entail a Saul [Page 18] FTP+ August 2000 very large and complex piece of software. Instead, the writer of an FTP+ server may wish to follow these guidelines for implementing an FTP+ server. The FTP+ server can be logically divided into a Core Component that receives most of the GEN Profile commands involved in defining the FTP+ server's mode of operation and Transport Components which are responsible for read/writing data and sending/receiving the data over a particular transport. 5.5 Miscellaneous FTP+ issues 5.5.1 Detecting partial asset transfer FTP as specified in RFC 959 has difficulty with Stream mode in detecting the difference between a broken data connection and the end of data because the receiving FTP server sees both cases as a closed connection. Block mode offers a solution to this problem, but Block mode is not supported by all FTP implementations. If an FTP+ server crashes in the process of performing a transfer, the fact that the control channel will also break may be used to detect that the whole file may not have been transferred. When the data channel breaks while the sending server is still writing data, the sending server will detect this condition and will send an error message over the control channel. However, when just the data channel breaks near the end of a transfer it is possible that the writer may have written all bytes into system buffers, but all of these buffers have not yet been received and acknowledged by the receiving server. To solve this problem, FTP+ enhances the reply response to the STOR command to include the number of bytes written to permanent storage and enhancing the RETR command to include the number of bytes written to the network. Then the client code FTP+ may verify that this number matches the number of bytes it sent or received. 5.5.2 Restarting failed transfers RFC 959 provides Block mode as a way to "checkpoint" data transfers. This is very important when two machines do not have the same byte size or use Record Structures. However, since 8-bit byte sizes are now universal and video asset essence is universally stored as a byte stream, this problem can, in a large degree, be solved by knowing how many bytes a server has successfully written. To this end, STOR and RETR shall include in their reply codes the number of bytes written or transmitted even when the transfer is interrupted by a data channel connection failure. Then, using the starting_byte_offset optional argument to STOR and RETR, a client can restart a failed transfer. In the case of an FTP+ server which has crashed, the SIZE command can be used to determine the size of Saul [Page 19] FTP+ August 2000 the file that was created on the receiving server to determine how much of it was successfully transferred. 5.5.3 Multi-homed hosts When an FTP+ server is connected to more than one type of network, there may be problems performing certain types of transfers. One solution is to state that the interface on which the control channel has been established should be the interface for all data connections. For FTP+, this does not work well because of the variety of data channels supported and the lack of simple TCP connections on each type of transport protocol. The difficulty can be seen in the case of a server which has three Ethernet interfaces, one which supports TCP/IP and the other two supporting raw XTP. In this instance, if a client program then wishes to initiate a two-server transfer between this server and another FTP+ server using XTP on one of the XTP Ethernet interfaces, it must state on which interface it wishes this server to listen for a connection (i.e., on which interface it should perform the XPSV command). The additional argument, interface_name to the XPSV command addresses this problem. It assumes, however that the client program knows the "name" of the interface on which it wants the server to listen. 5.5.4 Raw-mode XTP The use of XTP without IP is optional. The benefits of using "raw- mode" XTP are limited at the present time. In the future it is possible that "raw-mode" XTP might take advantage of the underlying physical media, for example using ATM QoS to implement transfer rate setting. However, at the time of this writing, no such capabilities exist. "Raw-mode" XTP does offer some small increase in efficiency at the present time, but this is not a significant factor. 6. Acknowledgments This document is a result of the output of the joint SMPTE and EBU Task Force, and many people were involved in the high level requirements for this effort. The original draft of this document was prepared for review and standardization in SMPTE and is concurrently undergoing review there through the additional work of Paul Stevens and Al Kovalick. The author also wishes to thank Michael Dolan for reformatting this document making normal IETF review possible. 7. Security Considerations The security impact of this work has not been considered. Saul [Page 20] FTP+ August 2000 8. Full Copyright Statement This document and translations of it may be copied and furnished to others, and derivative works that comment on or otherwise explain it or assist in its implementation may be prepared, copied, published and distributed, in whole or in part, without restriction of any kind, provided that the above copyright notice and this paragraph are included on all such copies and derivative works. However, this document itself may not be modified in any way, such as by removing the copyright notice or references to the Internet Society or other Internet organizations, except as needed for the purpose of developing Internet standards in which case the procedures for copyrights defined in the Internet Standards process must be followed, or as required to translate it into languages other than English. The limited permissions granted above are perpetual and will not be revoked by the Internet Society or its successors or assigns. This document and the information contained herein is provided on an "AS IS" basis and THE INTERNET SOCIETY AND THE INTERNET ENGINEERING TASK FORCE DISCLAIMS ALL WARRANTIES, EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO ANY WARRANTY THAT THE USE OF THE INFORMATION HEREIN WILL NOT INFRINGE ANY RIGHTS OR ANY IMPLIED WARRANTIES OF MERCHANTABILITY OR FITNESS FOR A PARTICULAR PURPOSE. 9. References [1] SMPTE/EBU, "Joint EBU / SMPTE Task Force on User Requirements for the Exchange of Television Programme Material as Bit Streams", http://www.smpte.org/engr/ebumeet1.html [2] XTP Forum, "Xpress Transport Protocol", ftp://dancer.ca.sandia.gov/pub/xtp4.0 [3] IETF RFC 959, File Transfer Protocol (FTP). [4] IETF RFC 1122, Requirements for Internet Hosts. [5] IETF RFC 2373, IP Version 6 Addressing Architecture. 10. Author's Address Mike Saul c/o Mentat 315 Gayley Ave, Ste 315 Los Angeles, CA 90024 Saul [Page 21] FTP+ August 2000 USA Email: mike@mentat.com Saul [Page 22]