INTERNET-DRAFT R. Weber S. Wilson (Expires: August 14, 2001) B. Snively E. Valdevit J. Tien D. Banks Brocade FCIP Frame Encapsulation Enhancement Proposals Status of this Memo This document is an Internet-Draft and is in full conformance with all provisions of Section 10 of RFC2026 [1]. Internet-Drafts are working documents of the Internet Engineering Task Force (IETF), its areas, and its working groups. Note that other groups may also distribute working documents as Internet-Drafts. Internet-Drafts are draft documents valid for a maximum of six months and may be updated, replaced, or obsoleted by other documents at any time. It is inappropriate to use Internet-Drafts as reference material or to cite them other than as "work in progress." The list of current Internet-Drafts can be accessed at http://www.ietf.org/ietf/1id-abstracts.txt. The list of Internet-Draft Shadow Directories can be accessed at http://www.ietf.org/shadow.html. This Internet-Draft will expire on August 14, 2001. Abstract This draft is for consideration by the FCIP team in the IPS working group. Improvements to Fibre Channel Over TCP/IP (FCIP) [2] frame encapsulation mechanisms are proposed. The objectives of the improvements are two fold: increasing the reliability of Fibre Channel frame detection, and assisting with the handling of Fibre Channel E_D_TOV and R_A_TOV time-out processing. Weber, et.al. Expires August 14, 2001 [Page 1] Internet-Draft FCIP Framing Enhancement Proposals February 2001 Table of Contents 1. Introduction . . . . . . . . . . . . . . . . . . . . . . . . . 3 2. The Delimiter . . . . . . . . . . . . . . . . . . . . . . . . . 4 2.1 Delimiter Stuffing . . . . . . . . . . . . . . . . . . . . . . 4 2.2 Delimiter Overhead . . . . . . . . . . . . . . . . . . . . . . 5 2.3 Delimiter Stuffing Alternatives . . . . . . . . . . . . . . . . 5 3. FCIP Header . . . . . . . . . . . . . . . . . . . . . . . . . . 6 3.1 Header word . . . . . . . . . . . . . . . . . . . . . . . . . . 7 3.2 Time Stamp (2 words). . . . . . . . . . . . . . . . . . . . . . 8 4. FC SOF through FC EOF . . . . . . . . . . . . . . . . . . . . . 9 5. FCIP Trailer. . . . . . . . . . . . . . . . . . . . . . . . . . 9 5.1 FCIP Checksum . . . . . . . . . . . . . . . . . . . . . . . . 10 6. Related FCIP Changes . . . . . . . . . . . . . . . . . . . . 10 6.1 Loss of FCIP synchronization . . . . . . . . . . . . . . . . 10 6.2 Values in relevant header fields . . . . . . . . . . . . . . 11 6.3 Performance Considerations . . . . . . . . . . . . . . . . . 11 7. References . . . . . . . . . . . . . . . . . . . . . . . . . 11 8. Author's Addresses . . . . . . . . . . . . . . . . . . . . . 12 9. Full Copyright Statement . . . . . . . . . . . . . . . . . . 12 Weber, et.al. Expires August 14, 2001 [Page 2] Internet-Draft FCIP Framing Enhancement Proposals February 2001 1. Introduction FCIP revision 1 uses the following encapsulation of Fibre Channel frames for transmission over TCP/IP: 32-bit word +----------------------------+ 0 | FCIP Header | +----------------------------+ 1 | FC SOF | +----------------------------+ 2 | FC Header | +----------------------------+ n | FC Payload | +----------------------------+ n+1 | FC CRC | +----------------------------+ n+2 | FC EOF | +----------------------------+ Fig. 1 - Rev 1 FCIP Frame Encapsulation If the length information in the FCIP header is suspect, this structure requires recipients to recognize the following pattern for identifying Fibre Channel frames: +----------------------------+ | FC EOF | +----------------------------+ | FCIP Header | +----------------------------+ | FC SOF | +----------------------------+ Fig. 2 - FCIP Frame Encapsulation Identification Pattern Since the FC EOF and FC SOF patterns are chosen to communicate information and not for frame delineation, locating an FCIP Header is problematic. For example, the common case FC SOF will be 0x'36000000' and the common case FC EOF will be 0x'41000000' (Class 3 normal start of frame and end of frame). Out of eight bytes six are zero bytes. Weber, et.al. Expires August 14, 2001 [Page 3] Internet-Draft FCIP Framing Enhancement Proposals February 2001 This proposal suggests that the FCIP Encapsulation shown in Figure 1 be replaced with the following. word +----------------------------+ 0 | Delimiter | +----------------------------+ 1 | Delimiter | +----------------------------+ 2 | Delimiter | +----------------------------+ 3 | Delimiter | +----------------------------+ 4 | FCIP Header (3 words) | +----------------------------+ 7 | FC SOF | +----------------------------+ 8 | FC Header | +----------------------------+ n | FC Payload | +----------------------------+ n+1 | FC CRC | +----------------------------+ n+2 | FC EOF | +----------------------------+ n+3 | FCIP Trailer (2 words) | +----------------------------+ Fig. 3 - Proposed FCIP Frame Encapsulation Changes are also proposed for the format of the FCIP Header word. The sections that follow discuss each of the proposed changes. 2. The Delimiter As proposed, each encapsulated Fibre Channel frame is preceded by four Delimiter words, in addition to the FCIP Header and FC SOF words already described in FCIP revision 1. Each Delimiter word shall contain 0x'FCFCFCFC'. The Delimiter value and four word (16 byte) repetition increase the probability that the frame encapsulation Delimiter will be unique and not occur "naturally" in the FC Payload. 2.1 Delimiter Stuffing Since is still possible for the four-word Delimiter pattern to appear in the FC Payload, a word-stuffing mechanism is proposed to ensure that no transmitted frame encapsulation contains the Delimiter Weber, et.al. Expires August 14, 2001 [Page 4] Internet-Draft FCIP Framing Enhancement Proposals February 2001 pattern in any part of the byte stream other than immediately before an FCIP Header. When a transmitter detects four or more consecutive words containing the Delimiter pattern in the FC Payload, the transmitter shall add one more Delimiter word of 0x'FCFCFCFC' to the group. A four consecutive word repetition is increased to five, five to six, and so on. FCIP receivers shall detect any five word or larger consecutive repetitions of the Delimiter and remove one word from the repetition. A five consecutive word repetition will be reduced to four, six to five, and so on. 2.2 Delimiter Overhead The Delimiter overhead is fixed at four words unless the Delimiter is embedded in the FC Payload. Since the size of Fibre Channel frames varies, the relative amount of this overhead depends on the Fibre Channel data stream. Assuming an approximately constant stream of 1028 byte (257 word) frames, the Delimiter overhead is 1.5%. When the presence of the Delimiter in the FC Payload necessitates Delimiter stuffing, the worst case occurs when the data pattern is Delimiter+Delimiter+Delimiter+Delimiter+other. In this case, an additional Delimiter must be stuffed for every five data words, producing a 20% overhead. However, this is truly a pathological case. 2.3 Delimiter Stuffing Alternatives As recommended, the following two alternative byte stuffing algorithms have been considered. o HDLC-like Framing with Octet-stuffed framing [3] o Consistent Overhead Byte Stuffing [4] Rejection of these alternatives is based on the anticipated used of FCIP and its Fibre Channel counterpart FC-BB-2 [5]. FCIP and FC-BB-2 are planned to provide a LAN tunnel that allows 10 Gbs (Giga bits per second) Fibre Channel fabrics to tunnel through a 10 Gbs Ethernet LAN. The 10 Gbs technology for both Fibre Channel and Ethernet employs for parallel byte lanes each operating at about 3 Gbs to achieve an aggregate 10 Gbs transfer rate. In short, 1 Gbs Fibre Channel and Ethernet is a word oriented interface not a byte oriented interface. Weber, et.al. Expires August 14, 2001 [Page 5] Internet-Draft FCIP Framing Enhancement Proposals February 2001 To support efficient 10 Gbs hardware implementations of the proposed FCIP encapsulation mechanism, the proposed Delimiter/stuffing mechanism is word oriented, like the underlying data transmission technology. The proposed Delimiter stuffing mechanism most closely resembles the Bit-stuffed framing alternative to Octet-stuffed framing described in [3], an algorithm simpler than any of the recommended alternatives. In bit-stuffed framing, a zero bit is inserted after every detected sequence of five contiguous one bits. Bit-stuffed framing is designed for serial transmission and correlates well to the "word- stuffed framing" proposed here for what is fundamentally a word transmission mechanism (10 Gbs Fibre Channel and Ethernet). The Octet-stuffed framing mechanism is more complex because escape characters are inserted in the stream to identify data uses of the frame delimiter and of course the escape character itself. Simple repetition of the frame delimiter seems simpler and would eliminate the additional complexity of an escape character. Delimiter repetition is the model for this proposal. Consistent Overhead Byte Stuffing (COBS) is a more complex algorithm than any of those discussed thus far, employing preprocessing of the entire data buffer in order to tightly bound the worst case overhead produced by stuffing. The COBS algorithm is designed for systems where the byte stuffing overhead might run afoul of link usage constrains, perhaps even legally imposed constraints. These are not conditions faced by FCIP. 3. FCIP Header The FCIP revision 1 Header is a single word whose format is as follows. |3 3 2 2 2 2 2 2 2 2 2 2 1 1 1 1 1 1 1 1 1 1 | |1 0 9 8 7 6 5 4 3 2 1 0 9 8 7 6 5 4 3 2 1 0 9 8 7 6 5 4 3 2 1 0 | +-------+-------+---------------------------+--------------------+ | VER. |HDR Len| Reserved | FCIP Frame Length | +-------+-------+---------------------------+--------------------+ Fig. 4 - Rev 1 FCIP Header Format Weber, et.al. Expires August 14, 2001 [Page 6] Internet-Draft FCIP Framing Enhancement Proposals February 2001 The proposed FCIP Header is three words as follows. +----------------------------+ | Header word | +----------------------------+ | Time Stamp integer | +----------------------------+ | Time Stamp fraction | +----------------------------+ Fig. 5 - Proposed FCIP Header Format The Header word contains most of the information found in the single revision 1 FCIP Header word (see 3.1). The two Time Stamp words are proposed to help FCIP end-points support the Fibre Channel E_D_TOV (Error Detect Timeout) and R_A_TOV (Resource Allocation Timeout) requirements. 3.1 Header word The proposed Header word format is as follows. |3 3 2 2 2 2 2 2 2 2 2 2 1 1 1 1 1 1 1 1 1 1 | |1 0 9 8 7 6 5 4 3 2 1 0 9 8 7 6 5 4 3 2 1 0 9 8 7 6 5 4 3 2 1 0 | +-------+---+-------------------+-----------+--------------------+ | VER. |Rsv| FCIP Frame Length | Reserved | -FCIP Frame Length | +-------+---+-------------------+-----------+--------------------+ Fig. 6 - Proposed Header word Format As an additional check on framing integrity, it is proposed that a ones complement of the FCIP Frame Length be added and the header be restructured for computational convenience. The HDR Len field has been removed because it contains no useful information in the FCIP Header Format identified by VER. equals 1. The HDR Len value is fixed at 1. Since any FCIP header changes that would result in information being conveyed by the HDR Len field would also have to change the contents of the VER. field, it seems sensible to postpone addition of a HDR Len field until a time when its purpose is known. This also allows the potential for making other uses the bits currently assigned to the HDR Len field in the event that other needs are identified. It should also be noted that the FCIP Frame Length includes any stuffed words added to the frame as a result of the Delimiter stuffing described in 2.1 and the two FCIP Trailer words described in section 5. As with FCIP revision 1, the FCIP Frame Length is in units of 32-bit words, not in units of bytes. Weber, et.al. Expires August 14, 2001 [Page 2] Internet-Draft FCIP Framing Enhancement Proposals February 2001 -FCIP Frame Length is the ones complement of FCIP Frame Length and FCIP receivers may compare the two as an additional verification that an FCIP Frame Header is being processed. It is proposed that the FCIP Frame Length be moved to bits 16-25 to simplify the computation of the FCIP Header. After computing and storing bits 16-31, a one's compliment can be taken of the 16-bit value and the high order 6 bits of that value masked to zero with the result being the bits 0-15 of the proposed FCIP Header. 3.2 Time Stamp (2 words) The two Time Stamp words are proposed to help FCIP end-points support the Fibre Channel E_D_TOV (Error Detect Timeout) and R_A_TOV (Resource Allocation Timeout) requirements. The complete 64-bit time stamp composed of a 32-bit integer and a 32-bit fraction. The integer and fraction Time Stamp words match the time format found in the proposed FC-GS-4 (Generic Services) Time Service [6]. The proposed FC-GS-4 Time Service format has been chosen for this proposal because synchronization of a Fabric clock is a key element of the FC-GS-4 proposal. Basing the FCIP time stamp on a Fabric service avoids reinventing the wheel. Since both ends of a FCIP connection are members of a Fibre Channel Fabric, the FC-GS-4 Time Service clock should be available to the FCIP header generator. It is important to note that the FC-GS-4 Time Service proposal is based on Network Time Protocol (Version 3) [7] and Simple Network Time Protocol (SNTP) Version 4 [8]. The proposed two 32-bit word format is aligned with the current version of SNTP. The Time Stamp integer field contains the time in seconds relative to the epoch, 00:00:00 January 1,1990. The Time Stamp fraction field contains the fractional part of the second identified by the Time Stamp integer field. In the FC-GS-4 Time Service proposal the fractional part is optional, but that will not be the case with FC-GS-4 Time Services are used for FCIP because accuracy to greater than one second is required for the proposed FCIP Time Stamp. A value of zero in the Time Stamp integer field indicates that no standard time stamp is provided by the sender and the receiver shall ignore the contents of the Time Stamp integer field. When the Time Stamp integer field contains a zero, the contents of the Time Stamp fraction field are not specified by the standard. Weber, et.al. Expires August 14, 2001 [Page 8] Internet-Draft FCIP Framing Enhancement Proposals February 2001 The use of the Time Stamp is as follows. Sender Receiver +-------------+ +-------------+ | |--+--------------+-->| | | | | Delimiter... | | | | | | FCIP Header | | | | Clock | | Time Stamp | | [Clock] | | | | ... | | | | | | Payload | | | +-------------+ +--------------+ +-------------+ Note: Time Stamp is the sender's clock. Fig. 7 - Proposed FCIP Time Stamp Usage When the Sender prepares an FCIP encapsulation, it stores the current value of its clock in the Time Stamp integer and fraction words. When the encapsulation arrives at the Receiver the Time Stamp values are compared to the current values in the synchronized clock. If the difference indicates that too much time has elapsed since the encapsulation was sent based on the R_A_TOV and E_D_TOV requirements placed on the Receiver, then the encapsulated Fibre Channel frame is discarded. 4. FC SOF through FC EOF Except for the Delimiter stuffing described in 2.1, no changes are proposed for the FC SOF, FC Header, FC Payload, FC CRC, or FC EOF elements of the FCIP revision 1 encapsulation. 5. FCIP Trailer Following the FC EOF word, a two word FCIP Trailer is proposed with the following format. +----------------------------+ | Reserved | +----------------------------+ | FCIP Checksum | +----------------------------+ Fig. 8 - Proposed FCIP Trailer Format The reserved word is proposed for future standardization and for compatibility with existing implementations. Weber, et.al. Expires August 14, 2001 [Page 9] Internet-Draft FCIP Framing Enhancement Proposals February 2001 5.1 FCIP Checksum It is proposed that a checksum word be added in the FCIP Trailer with the following format. |3 3 2 2 2 2 2 2 2 2 2 2 1 1 1 1 1 1 1 1 1 1 | |1 0 9 8 7 6 5 4 3 2 1 0 9 8 7 6 5 4 3 2 1 0 9 8 7 6 5 4 3 2 1 0 | +-------------------------------+--------------------------------+ | -Checksum | Checksum | +-------------------------------+--------------------------------+ Fig. 9 - Proposed FCIP Checksum Format The basic checksum (in bits 0 - 15) is computed as a 16-bit simple sum of all the preceding elements of the FCIP encapsulation, starting with the first Delimiter and ending with the reserved word that immediately precedes the checksum. The -checksum value is the ones complement of the basic checksum and is provided as an additional FCIP encapsulation integrity check. 6. Related FCIP Changes 6.1 Loss of FCIP synchronization The description of loss of synchronization in 4.4 needs to be revised to reflect the changes in this proposal. Replacement of the whole section with the following is proposed. The following FCIP Encapsulation elements provide verification that the FCIP devices communicating over a particular TCP connection are synchronized with each other: o The Delimiter and Delimiter Stuffing (see 2) o The FCIP Frame Length and -FCIP Frame Length in the FCIP Header (see 3) o The Checksum and -Checksum at the end of each encapsulation (see 7) These elements also provide substantial mechanisms for reestablishing synchronization when data loss, network congestion, or other error conditions in the TCP byte stream cause synchronization to be lost or questioned. If synchronization is lost reasonable efforts to reestablish it are recommended. If a communicating pair of FCIP devices cannot reestablish synchronization the receiving FCIP device shall reset the TCP connection (set the RST bit). Weber, et.al. Expires August 14, 2001 [Page 10] Internet-Draft FCIP Framing Enhancement Proposals February 2001 6.2 Values in relevant header fields All of section 5.3 describes IP and TCP header fields as if they are part of the FCIP encapsulation. This is incorrect. All of these fields belong to networking layers to which FCIP is an upper layer protocol. Section 5.3 needs to be rewritten to describe the various elements as parameters established by FCIP in its interactions with the TCP/IP lower layers. 6.3 Performance Considerations Is the following statement in section 9 plausible? "In the case where there is no data link fragmentation, each FC frame has a one-to-one mapping to a TCP segment; the TCP segment has a one-to-one mapping to an IP datagram; and the IP datagram has a one-to-one mapping to a data link frame. This has the tendency to further improve the throughput performance." Is is possible for FCIP to have any assurances regarding how its encapsulated Fibre Channel Frames are processed by the TCP and IP layers? 7. References: [1] Bradner, S., "The Internet Standards Process -- Revision 3", BCP 9, RFC 2026, October 1996. [2] Rajagopal, M., et.al., "Fibre Channel Over TCP/IP (FCIP)", draft-ietf-ips-fcovertcpip-01.txt, November 2000. [3] Simpson, W., "PPP in HDLC-like Framing", RFC 1662, July 1994. HDLC is an acronym for High-level Data Link Control. [4] Cheshire, S. and Baker, M., "Consistent Overhead Byte Stuffing", IEEE/ACM Transactions On Networking, pgs. 159-172, Vol. 7, No. 2, April 1999. [5] NCITS T11/Project 1466-D, "Fibre Channel Backbone -2", (FC-BB-2). No project revision is available, however, the FC-BB-2 project proposal is available from ftp://ftp.t11.org/t11/admin/project_proposals/00-432v1.pdf. [6] Wilson, S., "Fibre Channel Time Service -- Revision 0.3", T11/01-052v0, available from: ftp://ftp.t11.org/t11/pub/fc/gs-4/01-052v0.pdf. Weber, et.al. Expires August 14, 2001 [Page 11] Internet-Draft FCIP Framing Enhancement Proposals February 2001 [7] Mills, D., "Network Time Protocol (Version 3)", RFC 1305, March 1992. [8] Mills, D., "Simple Network Time Protocol (SNTP) Version 4 for IPv4, IPv6 and OSI", RFC 2030, October 1996. 8. Author's Addresses Ralph O. Weber Steve Wilson Representing Brocade Comm. Brocade Communications ENDL Texas 1745 Technology Drive PMB 178 San Jose, CA 95110 18484 Preston Road US Suite 102 Dallas, TX 75252 Phone" +1 408 487 8128 US EMail: swilson@brocade.com Phone: +1 214 912 1373 EMail: rweber@brocade.com Robert Snively Ezio Valdevit Brocade Communications Brocade Communications 1745 Technology Drive 1745 Technology Drive San Jose, CA 95110 San Jose, CA 95110 US US Phone: +1 408 487 8135 Phone: +1 408 487 8110 EMail: rsnively@brocade.com EMail: ezio@brocade.com Joe Tien Dave Banks Brocade Communications Brocade Communications 1745 Technology Drive 1745 Technology Drive San Jose, CA 95110 San Jose, CA 95110 US US Phone: +1 408 487 8040 Phone: +1 408 487 8137 EMail: tien@brocade.com EMail: dcb@brocade.com 9. Full Copyright Statement Copyright (C) The Internet Society (2000). All Rights Reserved. This document and translations of it may be copied and furnished to others, and derivative works that comment on or otherwise explain it or assist in its implementation may be prepared, copied, published and distributed, in whole or in part, without restriction of any Weber, et.al. Expires August 14, 2001 [Page 12] Internet-Draft FCIP Framing Enhancement Proposals February 2001 kind, provided that the above copyright notice and this paragraph are included on all such copies and derivative works. However, this document itself may not be modified in any way, such as by removing the copyright notice or references to the Internet Society or other Internet organizations, except as needed for the purpose of developing Internet standards in which case the procedures for copyrights defined in the Internet Standards process must be followed, or as required to translate it into languages other than English. The limited permissions granted above are perpetual and will not be revoked by the Internet Society or its successors or assigns. This document and the information contained herein is provided on an "AS IS" basis and THE INTERNET SOCIETY AND THE INTERNET ENGINEERING TASK FORCE DISCLAIMS ALL WARRANTIES, EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO ANY WARRANTY THAT THE USE OF THE INFORMATION HEREIN WILL NOT INFRINGE ANY RIGHTS OR ANY IMPLIED WARRANTIES OF MERCHANTABILITY OR FITNESS FOR A PARTICULAR PURPOSE. Weber, et.al. Expires August 14, 2001 [Page 13]