IP Storage Working Group Rod Mullendore INTERNET DRAFT Charles Monia Expires May 2001 Josh Tseng Nishan Systems November 2000 iFCP - A Protocol for Internet Fibre Channel Storage Networking Status of this Memo This document is an Internet-Draft and is in full conformance with all provisions of Section 10 of RFC2026. Internet-Drafts are working documents of the Internet Engineering Task Force (IETF), its areas, and its working groups. Note that other groups may also distribute working documents as Internet- Drafts. Internet-Drafts are draft documents valid for a maximum of six months and may be updated, replaced, or obsoleted by other documents at any time. It is inappropriate to use Internet- Drafts as reference material or to cite them other than as "work in progress." The list of current Internet-Drafts can be accessed at http://www.ietf.org/ietf/1id-abstracts.txt The list of Internet-Draft Shadow Directories can be accessed at http://www.ietf.org/shadow.html. Comments Comments should be sent to the ips mailing list (ips@ece.cmu.edu) or to the author(s). 1. Abstract This document specifies iFCP, a gateway-to-gateway protocol for the implementation of a Fibre Channel fabric in which TCP/IP switching and routing elements replace Fibre Channel components. The protocol enables the attachment of existing Fibre Channel storage products to an IP network by supporting the subset of fabric services required by such devices. 2. About This Document 2.1 Conventions used in this document The key words "MUST", "MUST NOT", "REQUIRED", "SHALL", "SHALL NOT", "SHOULD", "SHOULD NOT", "RECOMMENDED", "MAY", and "OPTIONAL" in this document are to be interpreted as described in RFC-2119 [1]. 2.2 Purpose of this document This is a standards-track document, which specifies a protocol for the implementation of Fibre Channel transport services on a TCP/IP network. Some portions of this document contain material from standards controlled by NCITS T10 and T11. This material is included here for informational purposes only. The authoritative information is given in the appropriate NCITS standards document. The authoritative portions of this document specify the protocol for mapping standards-compliant fibre Channel storage and adapter implementations to TCP/IP. This mapping includes sections of this document which describe the "iFCP Layer" (see section 4.1). 3 iFCP Introduction Mullendore, Monia, Tseng 1 iFCP November 2000 iFCP is a gateway-to-gateway protocol, which provides Fibre Channel fabric services to FCP-based Fibre Channel devices over a TCP/IP network. iFCP uses TCP to provide congestion control, error detection and recovery. iFCP's primary objective is to allow interconnection and networking of existing Fibre Channel devices at wire speeds over an IP network. The protocols and method of frame translation described in this document permit the transparent attachment of Fibre Channel storage devices to an IP-based fabric by means of lightweight gateways. The protocol achieves this transparency through a translation process that allows normal frame traffic to pass through the gateway directly, with provisions for intercepting and emulating the fabric services required by an FCP device. 3.1 Definitions Terms needed to clarify the concepts presented in this section are presented here. Fibre Channel Network _ A fabric and all attached Fibre Channel devices. Fabric _ The part of a Fibre Channel network which provides the transport services defined in the FC-FS specification. A fabric may be implemented in the IP framework by means of the architecture and protocols discussed in this document. FC-2 _ The Fibre Channel transport services layer described in the FC-FS specification. FCP Portal - An IP-addressable entity representing the point at which a logical or physical FCP device is attached to the IP network. N_PORT _ An iFCP or Fibre Channel entity representing the interface to Fibre Channel device functionality. This interface implements the Fibre Channel N_PORT semantics specified in the FC-FS standard [FC-FS]. N_PORT fabric address - The address of an N_PORT within the Fibre Channel fabric. N_PORT Network Address - The address of an N_PORT in the IP fabric. This address consists of the IP address of the FCP Portal and teh N_PORT ID of the directly-attached Fibre Channel device. F_Port - The interface used by an N_PORT to access Fibre Channel fabric and fabric services functionality. iFCP _ The protocol discussed in this document. Mullendore, Monia, Tseng 2 iFCP November 2000 Logical FCP Device _ The abstraction representing a single Fibre Channel device as it appears on an iFCP network. iSNS _ The protocol by which storage name services are implemented. Resolution of Fibre Channel network object names is provided by an iSNS name server. N_PORT Session - An association created when two N_PORTS have executed a PLOGI operation. It is comprised of the N_PORTs and the associated set of TCP connections that carry traffic between them. iFCP Frame - The frame inserted into the TCP stream which contains the Fibre Channel frame and iFCP header. 3.2 The iFCP Network Model The following diagram shows a Fibre Channel fabric with attached devices. These are connected to the fabric through N_PORT and F_PORT interfaces, whose behavior is specified in [FGS]. Within the Fibre Channel device domain, fabric-addressable entities consist of other N_PORTs and devices internal to the fabric that perform the fabric services defined in [FGS]. In this case, the N_PORT Fibre Channel addresses are 24-bit quantities that are unique within the scope of the FC fabric. N_PORTs that perform fabric services are assigned well-known addresses starting at the top end of the 24-bit Fibre Channel address space. Fibre Channel Network +--------+ +--------+ | FC | | FC | | Device | | Device | |........| |........| Fibre Channel | N_PORT |<------>| N_PORT | Device Domain +---+----+ +----+---+ ^ | | | +---+----+ +----+---+ | | F_PORT | | F_PORT | | ==========+========+========+========+============== | Fabric & | | | Fabric Services | v | | Fibre Channel +--------------------------+ Fabric Domain Mullendore, Monia, Tseng 3 iFCP November 2000 An iFCP Network with iFCP Gateways Fibre Channel Devices Fibre Channel Devices +--------+ +--------+ +--------+ +--------+ | FC | | FC | | FC | | FC | | Device | | Device | Fibre | Device | | Device | Fibre |........| |........| Channel |........| |........| Channel | N_PORT | | N_PORT |<--------->| N_PORT | | N_PORT | Device +---+----+ +---+----+ Traffic +----+---+ +----+---+ Domain | | | | ^ +---+----+ +---+----+ +----+---+ +----+---+ | | F_PORT | | F_PORT | | F_PORT | | F_PORT | | =+========+==+====================+========+==+========+========== | iFCP Layer |<--------->| iFCP Layer | | |....................| ^ |....................| | | FCP Portal | | | FCP Portal | v +--------+-----------+ | +----------+---------+ IP Fabric | Control | | Data | | | | | |<------Encapsulated Frames------->| | +------------------+ | | | | | +------+ IP Network +--------+ | | +------------------+ The above diagram shows the equivalent iFCP fabric implementation. Here, Fibre Channel devices are connected to the fabric through F_PORTs implemented as part of the edge switch or gateway. At the N_PORT interface, the network appears as a Fibre Channel fabric. N_PORT to N_PORT communications that traverse a TCP/IP network require the intervention of the iFCP layer. This is done through the following operations on Fibre Channel frames: a) For outbound frames, translate Fibre Channel fabric addresses imbedded in FC frames to N_PORT Network Addresses. b) For inbound frames, translate N_PORT Network Addresses to Fibre Channel fabric addresses. c) Redirect requests for fabric-supplied link services addressed to one of the well-known Fibre Channel N_PORT addresses. d) Generate control data in response to certain link service requests or fabric control operations as described in section 7.1. e) Encapsulate Fibre Channel frames for injection into the TCP/IP network and de-encapsulate Fibre Channel frames received from the TCP/IP network. f) Direct de-encapsulated, FC frames to the appropriate N_PORT. Mullendore, Monia, Tseng 4 iFCP November 2000 After address translation on outbound FC frames, the emulation process generates two types of encapsulated FC frame traffic: a) FC frames to be passed between N_PORTs unchanged, except for the address field modifications described below. b) Augmented FC frames generated in response to N_PORT Link Service requests. These frames are passed between the iFCP layers. For N_PORT link service requests, iFCP may modify the payload to perform these operations in an IP fabric. Control traffic is also generated in order to perform certain peer- to-peer operations among FCP Portals, such as TCP/IP connection setup and management. 3.3 Fibre Channel Frame Address Translation An iFCP gateway is responsible for the mapping between N_PORT Fibre Channel addresses and N_PORT addresses in the IP fabric. In the IP fabric, the address of a N_PORT has two components: the IP address of the FCP Portal to which the Fibre Channel device is attached and an N_PORT ID that is unique within the scope of the FCP Portal. When an FC frame is in transit, the IP components of the source and destination FCP Portals are implied from the TCP/IP connection context. The source and destination N_PORT IDs are stored in the corresponding N_PORT address fields that are part of the FC frame. The iFCP gateway is responsible for assigning Fibre Channel N_PORT IDs to directly-attached N_PORTs. For external N_PORTs, the iFCP gateway is also responsible for assigning an internal key used to look up the N_PORT network address for the external device. To perform this function a gateway or edge switch maintains a table which maps frame traffic to the appropriate TCP/IP connection and N_PORT ID of all external N_PORTs with active sessions. The gateway builds the store of N_PORT network addresses for external devices in the IP fabric by: a) Intercepting name service requests issued by directly-attached N_PORTs and redirecting them to the iSNS name server or, b) Intercepting incoming N_PORT login requests from external Fibre Channel devices. The TCP/IP connection context to a given FCP Portal may be established during fabric initialization. Subsequently, in response to name server requests, the iSNS server returns the IP address and N_PORT ID pair. The IP address is mapped Mullendore, Monia, Tseng 5 iFCP November 2000 to the connection context. After saving the context and N_PORT ID, the iFCP layer then creates a 24-bit key that is returned to the local N_PORT as the Fibre Channel address of the external device. For outbound frames, the table of external N_Port network addresses are referenced to map the Destination N_PORT address key and Source N_PORT ID to a TCP connection identifier and N_PORT ID. The translation process for outbound frames is shown below. Raw Fibre Channel Frame +--------+-----------------------------------+ +--------------+ | | Destination N_PORT Address Key |---->| Lookup TCP | +--------+-----------------------------------+ | connection | | | Source N_PORT ID |---->| and N_PORT ID| +--------+-----------------------------------+ +------+-------+ | | | TCP | Control information | | Conn | and Payload | | & +--------------------------------------------+ | N_PORT | ID | After Address Translation and TCP/IP Encapsulation | +--------------------------------------------+ Conn | | iFCP Encapsulation |<-----------+ | Header | Context | +========+===================================+ | | | Destination N_PORT ID |<-----------+ +--------+-----------------------------------+ | | Source N_PORT ID | +--------+-----------------------------------+ | | | Control information | | and Payload | +--------------------------------------------+ For inbound frames, the store regenerates the internal address key from the TCP connection context and N_PORT ID contained in the encapsulated FC frame. The translation process for inbound frames is shown below. Mullendore, Monia, Tseng 6 iFCP November 2000 Network Format of Inbound Frame +--------------------------------------------+ Conn. +---------+ | iFCP Encapuslation Header |------>| Address | | |Context| Key | +========+===================================+ | Lookup | | | Destination N_PORT ID | | | +--------+-----------------------------------+ | | | | Source N_PORT ID |------>| | +--------+-----------------------------------+ +-----+---+ | | |Address | Control information | | Key | and Payload | | +--------------------------------------------+ | | | | Frame after Address Translation and Decapsulation | +--------+-----------------------------------+ | | | Destination N_PORT ID | | +--------+-----------------------------------+ | | | Source N_PORT Key |<------------+ +--------+-----------------------------------+ | | | Control information | | and Payload | +--------------------------------------------+ 3.4 iFCP Layered Services The following diagram shows the functional layers for host devices that support FCP. As shown, iFCP provides a set of layered services that transparently provide the transport services required by FCP devices. Using the iFCP framework, any existing host FCP implementation will execute with no modifications required. The iFCP protocol layer consists of the data transport services and iFCP-specific Link Services. This layer provides transport services specific to Fibre Channel devices as specified in [FC-PH], [FC-PH-2], and [FC-PH-3]. This is illustrated in the following diagram, which shows the IP Fabric consisting of the TCP/IP network and the iFCP Layer. The IP Fabric provides the transport services for FCP, and is a direct replacement for the transport services provided by a Fibre Channel fabric. Meanwhile, the components in the Fibre Channel Device Domain remain unchanged. Mullendore, Monia, Tseng 7 iFCP November 2000 +---------------------------------------+ - - - - - - - | Storage & Backup Applications | +---------------------------------------+ | Operating System | Application +--------------------+ | Layer | SCSI | | +--------------------+ | - - - - - - - | FCP | | FC-4 Layer +------------+-------+------------------+ - - - - - - - | | Link Services | | +--------------------------+ FC-2 Layer ^ | | | | N_PORT - F_PORT Interface | Fibre Channel | | Device Domain <=============================================================> | | IP Fabric | iFCP Data Transport Service | | | | v | +---------------+ | |iFCP Specific | iFCP Layer | |Link Services | +-----------------------+---------------+ - - - - - - | | | TCP | Transport | | Layer +---------------------------------------+ - - - - - - | | | IP | Network | | Layer +---------------------------------------+ - - - - - - | | | Physical Transport | Link Layer | | +---------------------------------------+ - - - - - - In the figure shown above, each layer leverages the services of the layer below it. 3.4.1 Application Layer This includes the operating system, Storage and Backup applications, and the SCSI driver. This layer interfaces with FCP and Link Services in the FC-2 and FC-4 layers. 3.4.2 FC-4 Layer (FCP) FCP is the Fibre Channel FC-4 layer application protocol used to communicate with devices implementing the SCSI-3 command set and architectural model. Basically, FCP divides each SCSI I/O operation into a series of information units to be transferred between the initiator and target. Mullendore, Monia, Tseng 8 iFCP November 2000 3.4.3 FC-2 Layer The FC-2 Layer provides the facilities for Link Services and transfer of Fibre Channel information units as described below. 3.4.3.1 Link Service Messages Fibre Channel defines a series of link services defined in Fibre Channel Physical and Signaling Interface specification (FC-PH, FC- PH-2, FC-PH-3). These Link Service Messages provide a set of defined functions that allow a Fibre Channel port to send control information, or to request another port to perform a specific function. Some Link Service messages reference services provided internally within the Fibre Channel fabric. 3.4.3.2 N_PORT Interface This is an interface which provides access to Fibre Channel device functionality. The N_PORT interface is responsible for segmentation and reassembly of information units from Fibre Channel frames. 3.4.3.3 F_PORT Interface This is the interface through which the N_PORT accesses the Fibre Channel fabric. 3.4.4 iFCP Layer The iFCP layer provides three essential services for FCP-based storage products: a) Transport of Fibre Channel frames and Link Service messages between N_PORTs. b) Support for special Link Service messages needed by iFCP to manage the transmission of storage data on a IP network. c) Augmentation of some Link Service messages with additional data needed in the iFCP environment. The iFCP layer maps Fibre Channel frames to a predetermined TCP connection for transport. Additionally, many link service messages can similarly be transported without modification over a TCP connection. 4. iFCP Protocol 4.1 Overview 4.1.1 iFCP Transport Services Mullendore, Monia, Tseng 9 iFCP November 2000 The iFCP transport services map the Fibre Channel frames comprising each FCP IU and Link Service message to a predetermined TCP connection for transport across an IP network. When receiving in- band FCP-based storage data from the network, the iFCP layer transports, and delivers each resulting frame to the appropriate N_PORT via the F_PORT. The iFCP layer never interprets the frame contents. For incoming iFCP frames with control data, iFCP interprets the augmented information in the iFCP header, modifies the frame content accordingly, and may forward the resulting frame to the N_PORT for further processing. For out-bound Fibre Channel frames that require control data, the iFCP layer creates the augmented information based on frame content, modifies the frame content, then transmits the resulting Fibre Channel frame with augmented iFCP header through the appropriate TCP connection. 4.1.2 iFCP Support for Link Services Some Link Service messages reference constructs which are specific to the Fibre Channel fabric environment, but are irrelevant in the context of an IP fabric. When iFCP encounters such messages, it will augment the information in the payload by adding additional information in the iFCP header. The receiving iFCP layer will reference the augmented information in order to reconstruct the original Link Service message. The reconstructed frames are then forwarded to the N_PORT for further processing. Section 7.1 describes augmented Link Services. 4.3 Mandatory FC-2 Functionality [To be specified] 4.4 FC-2 Functionality Not Supported [To be specified] 4.5 Optional FC-2 Functionality [To be specified] 5. iFCP Encapsulation of FCP and Link Services Messages This section describes the formatting of iFCP messages. iFCP encapsulates two types of payload--Frames comprising FCP Information Units and Link Service Messages. Mullendore, Monia, Tseng 10 iFCP November 2000 The iFCP frame encapsulation format is shown below. The iFCP header encapsulates the iFCP payload containing the Fibre Channel frame. The length of the iFCP frame depends on the size of the encapsulated Fibre Channel frame and the amount of augmentation data (if any) in the iFCP header. Each Fibre Channel frame, in turn, is comprised of a Fibre Channel header and payload whose format is specified in the appropriate Fibre Channel standards [FC- PH]. Byte MSb LSb Offset 7 6 5 4 3 2 1 0 +----------------------------------+ - - - - - - - - - - - 0 | iFCP Header | N Bytes ^ +----------------------------------+ - - - - - - - - | | | ^ iFCP N / Fibre Channel Header / 24 Bytes | Frame | (described in section 5.1) | | | +----------------------------------+ Fibre | N+24 | | Channel | / Fibre Channel Payload / L Bytes Frame | | | | | +----------------------------------+ - - - - - - - - - - - Total Length = 24 + N + L 5.1 iFCP Header The iFCP header for TCP is shown below. |3 3 2 2 2 2 2 2 2 2 2 2 1 1 1 1|1 1 1 1 1 1 0 0 0 0 0 0 0 0 0 0| |1 0 9 8 7 6 5 4 3 2 1 0 9 8 7 6|5 4 3 2 1 0 9 8 7 6 5 4 3 2 1 0| +-------+-------+---------------+-------------------------------+ | CLASS | VERS | HEADER MARKER | iFCP FLAGS | +-------+-------+---------------+-------------------------------+ | iFCP HEADER LENGTH | iFCP DATA LENGTH | +---------------------------------------------------------------+ | iFCP AUGMENTATION DATA (if present) | +---------------------------------------------------------------+ CLASS - This 4-bit field indicates the class of service. Only the values 2 and 3 are allowed and are equivalent to the class of service for Fibre Channel. VERS - This 4-bit field indicates the iFCP protocol version. The version number shall be 1. iFCP FLAGS - This 16-bit field contains status flags that indicate various parameters for the data frame. Bit Field iFCP Flag Description --------- --------- ----------- 15 SEQUENCE END Indicates if frame terminates a sequence. For Class 3 sequences, the Mullendore, Monia, Tseng 11 iFCP November 2000 FCP_Portal sets this bit on the last frame of the sequence. In Class 2 sequences, this bit is set by the recipient in the ACK frame that terminates the sequence. 1=Last frame of sequence, 0=not 14 SEQUENCE START Indicates if frame is the first frame of a sequence. 1=First frame of sequence, 0=not 13-3 RESERVED 2 TCP ELS Indicates frame contains a TCP-specific Link Service message. 1 AUGMENTATION Indicates the iFCP Augmentation Data PRESENT field is present in the iFCP header. 1=Present, 0=Not 0 COMPLIANCE This bit must be enabled. 1=Enabled, 0=Not HEADER MARKER - This 8-bit field shall be 0xAA. iFCP HEADER LENGTH - A 2-byte field giving the length in bytes of the iFCP HEADER, including the iFCP AUGMENTATION DATA field (if present). iFCP DATA LENGTH - A 2-byte field giving the length in bytes of the iFCP payload. This includes the FCP or Link Service headers, and is always a multiple of 4. This field does not include the iFCP Header. iFCP AUGMENTATION DATA - This is a variable-length field containing augmentation data required for transport of some Link Service messages over TCP/IP. 5.2 Fibre Channel Frame Header The Fibre Channel frame header defined in FC-PH is used when transporting FCP and Link Service frames in an IP fabric. Use of the Fibre Channel frame header simplifies the connection of Fibre Channel devices to a iFCP storage network. In iFCP, the only modification to the basic Fibre Channel frame header is that the contents of D_ID and S_ID fields are replaced with Destination and Source N_PORT IDs as described in section 3.3. The figure below shows the format of the header used for both FCP and Link Service frames. The contents of D_ID and S_ID fields have been replaced by the Destination and Source N_PORT IDs. Mullendore, Monia, Tseng 12 iFCP November 2000 3 3 3 2 2 2 2 2 2 2 2 2 1 1 1 1 1 1 1 1 1 1 0 0 0 0 0 0 0 0 0 0 |1 0 9 8 7 6 5 4|3 2 1 0 9 8 7 6|5 4 3 2 1 0 9 8|7 6 5 4 3 2 1 0| +---------------+-----------------------------------------------+ | R_CTL | Destination N_PORT ID | +---------------+-----------------------------------------------+ | CS_CTL | Source N_PORT ID | +---------------+-----------------------------------------------+ | Type | F_CTL | +---------------+---------------+-------------------------------+ | SEQ_ID | DF_CTL | SEQ_CNT | +---------------+---------------+-------------------------------+ | OX_ID | RX_ID | +-------------------------------+-------------------------------+ | PARAMETER | +---------------------------------------------------------------+ Other than Source and Destination N_PORT ID, descriptions of each of the above fields can be found in [FC-PH]. 5.2.1 Destination N_PORT ID This 24-bit value is stored in the Fibre Channel D_ID field. The destination IP address, combined with the Destination N_PORT ID, specifies the unique N_PORT network address of the device receiving the Fibre Channel frame. 5.2.2 Source N_PORT ID This 24-bit value is stored in the Fibre Channel S_ID field. The source IP address, combined with the Source N_PORT ID specifies the unique N_PORT network address of the device originating the Fibre Channel frame. 6. TCP Stream Transport of iFCP Frames TCP connections MAY be established between FCP_Portals that have discovered each other through a naming service or through manual configuration. If a TCP connection is not maintained between the FC_Portals, then a change in the status of remote N_PORTs must be discovered through a central name server authority. Multiple TCP connections may exist between pairs of FCP Portals. Such connections are either "bound" or "unbound". An unbound connection is a TCP connection that is not actively supporting an N_PORT login session. Pre-existing TCP connections between FCP Portals remain unbound and uncommitted until a CBIND message (see section 7.2.2) has been transmitted through them. When the iFCP layer detects a Port Login (PLOGI) message creating a login session between a pair of N_PORTs, it will select an existing TCP connection (bound or unbound) or establish a new TCP connection, and send the CBIND message down that TCP connection. Mullendore, Monia, Tseng 13 iFCP November 2000 This allocates the TCP connection to that PLOGI login session. Using a previously-bound TCP connection binds that connection to more than one N_PORT session. [Currently, this model only allocates at most one TCP connection per PLOGI session. The use of multiple connections per PLOGI session is TBD]. 6.1 TCP Session Models iFCP supports two models for TCP sessions used for transporting Fibre Channel frames between FCP Portals. 6.1.1 Single Connection (SC) This model uses a single TCP connection to transport both SCSI Commands and related data. All Fibre Channel frames betweeen unique pairs of N_PORTs are transported over the same TCP connection. 6.1.2 Multiple Connections (MC) This section TBD. 6.1.4 Session Model Support Support for the TCP Single Connection Model (SC) is required, while support for the Multiple Connections model (MC) will be optional. 6.2 TCP Port Numbers An FCP Portal uses a single port number to receive TCP connection requests for iFCP over TCP. All TCP connections established between FCP Portals must be directed to the registered well known port number assigned by the IANA. Note that control and data connections are established to the same port number with CBIND messages used to distinguish the connection type. An FCP Portal may use any TCP port number consistent with its implementation of the TCP/IP stack to initiate a TCP connection, but each port number must be unique. 7. Link Services The link services provide a set of functions that allow a port to send control information or request another port to perform a specific function. Each Link Service message (response and reply) is carried by a Fibre Channel sequence, and can be segmented into multiple frames. Mullendore, Monia, Tseng 14 iFCP November 2000 The iFCP Layer is responsible for transporting Link Service messages across the IP fabric. This includes mapping Link Service messages appropriately from the domain of the Fibre Channel transport to that of the IP network. This process may involve manipulation of field values as the Link Service message travels to and from the IP and Fibre Channel fabrics. It also may involve the inclusion of additional control data through use of the AUGMENTATION DATA field in the iFCP header in order to make the Link Service message significant in the IP fabric domain. [Editor's Note: The method for processing multi-frame Link Service messages will be detailed in a subsequent draft] 7.1 Augmented Link Service Messages The following Link Service Messages must be augmented with additional control data carried in the iFCP header. When the iFCP header encapsulates one of these Link Service messages in the iFCP payload, the AUGMENTATION PRESENT bit must be enabled in the iFCP FLAGS field, and the iFCP HEADER LENGTH must be adjusted appropriately to account for the additional augmentation data in the iFCP header. In addition, many ACC responses to the following Link Service messages must also have control data carried in the encapsulating iFCP header. Link Service Message LS_COMMAND Short Name -------------------- ---------- ---------- Port Login 0x03000000 PLOGI Read Exchange Status Block 0x08000000 RES Read Sequence Status Block 0x09000000 RSS Request Sequence Initiative 0x0A000000 RSI Reinstate Recovery Qualifier 0x12000000 RRQ Read Exchange Concise 0x13000000 REC Third Party Process Logout 0x24 TPRLO Discover Address 0x52000000 ADISC The above Link Service messages use N_PORT fabric addresses in their message format, and must have the corresponding World Wide Port name (WWPN) carried in the AUGMENTATION DATA field of the iFCP header. The N_PORT fabric address field shall then be cleared while the Link Service message is carried in the iFCP frame. Upon receipt of the iFCP frame, the iFCP layer shall use the WWPN data in the iFCP header to replace the original N_PORT fabric address with the appropriate value. The following sections describe the contents and format of the iFCP AUGMENTATION DATA field in the iFCP header. Note that if appropriate, the augmentation may also apply to the corresponding ACC or LS_RJT reply. 7.1.1 Port Login (PLOGI) Mullendore, Monia, Tseng 15 iFCP November 2000 PLOGI provides the mechanism for establishing a login session between two N_PORTs. The PLOGI request carries information identifying the originating N_PORT, including specification of its capabilities and limitations. If the destination N_PORT accepts the login request, it sends an accept (an ACC frame with PLOGI payload), specifying its capabilities and limitations. This exchange establishes the operating environment for the two N_PORTs. The following figure is duplicated from FC-PH, and shows the PLOGI message format for both request and accept (ACC) response. A port will reject a PLOGI request by transmitting an LS_RJT message, which contains no payload. Byte MSb LSb Offset 7 6 5 4 3 2 1 0 +----------------------------------+ 0 | LS_COMMAND | 4 Bytes +----------------------------------+ 4 | COMMON SERVICE PARAMETERS | 16 Bytes +----------------------------------+ 20 | PORT NAME | 8 Bytes +----------------------------------+ 28 | NODE NAME | 8 Bytes +----------------------------------+ 36 | CLASS 1 SERVICE PARAMETERS | 16 Bytes +----------------------------------+ 52 | CLASS 2 SERVICE PARAMETERS | 16 Bytes +----------------------------------+ 68 | CLASS 3 SERVICE PARAMETERS | 16 Bytes +----------------------------------+ 86 | CLASS 4 SERVICE PARAMETERS | 16 Bytes +----------------------------------+ 102 | VENDOR VERSION LEVEL | 16 Bytes +----------------------------------+ Total Length = 118 Details on the above fields, including common and class-based service parameters, can be found in [FC-PH]. The above PLOGI message is transported by the iFCP layer without modification. However, additional accompanying information must be carried in the encapsulating iFCP header. The iFCP AUGMENTED DATA field in the iFCP header shall contain the following information: Byte MSb LSb Offset 7 6 5 4 3 2 1 0 +----------------------------------+ 0 | DEVICE TYPE | 4 Bytes +----------------------------------+ Total Length = 4 Mullendore, Monia, Tseng 16 iFCP November 2000 DEVICE TYPE - This field contains a value indicating the type of device. The allowed values are shown in the following table. A Parallel SCSI type is assumed to be attached by a SCSI-iFCP Router, while a Fibre Channel device is assumed to be attached by an iFCP Edge Switch. iSCSI and mFCP devices are native IP-based storage devices. DEVICE TYPE Bit Field Device Type Values --------- ------------------ 7:4 RESERVED 3 iSCSI DEVICE 2 mFCP DEVICE 1 FIBRE CHANNEL DEVICE 0 PARALLEL SCSI DEVICE 7.1.2 Third Party Process Logout (TPRLO) TPRLO provides a mechanism for an N_PORT (third party) to remove one or more login sessions that exists between the destination N_PORT and other N_PORTs specified in the command. This command includes one or more TPRLO LOGOUT PARAMETER PAGEs, each of which when combined with the destination N_PORT identifies a SCSI login session which shall be terminated by the command. Byte MSb LSb Offset 7 6 5 4 3 2 1 0 +----------------------------------+ 0 | LS_COMMAND | 1 Byte +----------------------------------+ 1 | PAGE LENGTH (0x10) | 1 Byte +----------------------------------+ 2 | PAYLOAD LENGTH (0x14) | 2 Bytes +----------------------------------+ 4 | TPRLO LOGOUT PARAMETER PAGE 1 | 2-4 Bytes +----------------------------------+ | . . . . | M Bytes +----------------------------------+ | TPRLO LOGOUT PARAMETER PAGE N | 2-4 Bytes +----------------------------------+ Total Length = Variable Each TPRLO LOGOUT PARAMETER PAGE identifies a remote N_PORT which when combined with the destination N_PORT identifies a SCSI session to be terminated. The TPRLO LOGOUT PARAMETER PAGE is of the following format: Mullendore, Monia, Tseng 17 iFCP November 2000 Byte MSb LSb Offset 7 6 5 4 3 2 1 0 +----------------------------------+ 0 | TYPE CODE | 1 Byte +----------------------------------+ 1 | TYPE CODE EXTENSION | 1 Byte +----------------------------------+ 2 | TPRLO FLAGS | 2 Bytes +----------------------------------+ 4 | ORIG PROCESS ASSOC (if present) | 4 Bytes +----------------------------------+ 8 | RESP PROCESS ASSOC (if present) | 4 Bytes +----------------------------------+ 12 | RESERVED | 1 Byte +----------------------------------+ 13 | THIRD PARTY ORIGINATOR N_PORT ID | 3 Bytes +----------------------------------+ When the iFCP header contains a TPRLO message (including the ACC response), the iFCP AUGMENTED DATA field will contain the PORT_NAME(s) (WWPN) identifying the N_PORT described by the equivalent TPRLO LOGOUT PARAMETER PAGE(s). If more than one TPRLO LOGOUT PARAMETER PAGE is contained in the Link Service message, equivalent additional PORT_NAME shall also be carried in the iFCP AUGMENTED DATA field. PORT_NAMEs shall be listed in the same order as the equivalent TPRLO LOGOUT PARAMETER PAGEs in the original Link Service message. The following diagram describes the PORT_NAME entries in the iFCP AUGMENTATION DATA field in the iFCP header: Byte MSb LSb Offset 7 6 5 4 3 2 1 0 +----------------------------------+ 0 | 3rd PARTY ORIG PORT NAME 1 | 8 Bytes +----------------------------------+ 8 |3rd PTY ORIG PORT NAME 2 (if pres)| 8 Bytes +----------------------------------+ 16 | . . . . | +----------------------------------+ Additionally, the THIRD PARTY ORIGINATOR N_PORT ID field in each TPRLO LOGOUT PARAMETER PAGE shall be cleared when it is carried by the iFCP header. This applies to both the original Link Service message and the ACC response. When the iFCP layer receives a TPRLO message with AUGMENTATION DATA in the iFCP header, it shall use the latter to replace the THIRD PARTY ORIGINATOR N_PORT ID in the original Link Service message, before forwarding it on to the upper Fibre Channel layers. Additional information on TPRLO can be found in [FC-PH-2]. Mullendore, Monia, Tseng 18 iFCP November 2000 7.1.3 Reinstate Recovery Qualifier (RRQ) RRQ is a request to release resources previously made unavailable as a result of an ABTS or ABTX. The following shows the format of an RRQ request message. Byte MSb LSb Offset 7 6 5 4 3 2 1 0 +----------------------------------+ 0 | LS_COMMAND | 4 Bytes +----------------------------------+ 4 | RESERVED | 1 Bytes +----------------------------------+ 5 | EXCHANGE ORIGINATOR S_ID | 3 Bytes +----------------------------------+ 8 | RRQ OX_ID | 2 Bytes +----------------------------------+ 10 | RRQ RX_ID | 2 Bytes +----------------------------------+ Total Length = 12 Upon transmitting the RRQ Link Service message to the IP network, the iFCP layer shall clear the EXCHANGE ORIGINATOR S_ID field. Furthermore, the iFCP layer shall add the EXCHANGE ORIGINATOR to the iFCP AUGMENTED DATA field in the iFCP header. The EXCHANGE ORIGINATOR is a 4-byte field set to 0x00000001 to indicate that the RRQ originator is also the originator of the exchange for which the Recovery Qualifier is being reinstated. This field is set to 0x00000000 to indicate that the RRQ recipient is the originator of the exchange for which the Recovery Qualifier is being reinstated. RRQ OX_ID - The OX_ID of the exchange for which the Recovery Qualifier is being reinstated. RRQ RX_ID - The RX_ID of the exchange for which the Recovery Qualifier is being reinstated. Upon receipt of the RRQ Link Service message from the IP network, the iFCP layer shall use the N_PORT fabric addresses (in the Fibre Channel header) and the EXCHANGE ORIGINATOR value (in the iFCP header) to replace the EXCHANGE ORIGINATOR S_ID field before forwarding the message on to the upper Fibre Channel layers. The ACC reply contains only the LS_COMMAND field as the payload, and does not require additional augmentation data. An LS_RJT reply may be sent in lieu of ACC, to indicate that the RRQ was rejected. Mullendore, Monia, Tseng 19 iFCP November 2000 7.1.4 Read Exchange Concise (REC) REC is a request for information on an exchange. The following shows the format of an REC request. Byte MSb LSb Offset 7 6 5 4 3 2 1 0 +----------------------------------+ 0 | LS_COMMAND | 4 Bytes +----------------------------------+ 4 | RESERVED | 1 Bytes +----------------------------------+ 5 | EXCHANGE ORIGINATOR S_ID | 3 Bytes +----------------------------------+ 8 | REC OX_ID | 2 Bytes +----------------------------------+ 10 | REC RX_ID | 2 Bytes +----------------------------------+ Total Length = 12 Upon transmitting the REC Link Service message to the IP network, the iFCP layer shall clear the EXCHANGE ORIGINATOR S_ID field. Furthermore, the iFCP layer shall add the EXCHANGE ORIGINATOR to the iFCP AUGMENTED DATA field in the iFCP header. The EXCHANGE ORIGINATOR is a 4-byte field set to 0x00000001 to indicate that the REC originator is also the originator of the exchange for which status is being requested. This field is set to 0x00000000 to indicate that the REC recipient is the originator of the exchange for which status is being requested. REC OX_ID - The OX_ID of the exchange for which information is being requested. REC RX_ID - The RX_ID of the exchange for which information is being requested. RX_ID may be 0xFFFF, indicating RX_ID is unassigned. Upon receipt of the REC Link Service message from the IP network, the iFCP layer shall use the N_PORT fabric addresses (in the Fibre Channel header) and the EXCHANGE ORIGINATOR value (in the iFCP header) to replace the EXCHANGE ORIGINATOR S_ID field before forwarding the message on to the upper Fibre Channel layers. The following shows the format of the ACC payload in response to REC. Mullendore, Monia, Tseng 20 iFCP November 2000 Byte MSb LSb Offset 7 6 5 4 3 2 1 0 +----------------------------------+ 0 | LS_COMMAND (0x02000000) | 4 Bytes +----------------------------------+ 4 | REC OX_ID | 2 Bytes +----------------------------------+ 6 | REC RX_ID | 2 Bytes +----------------------------------+ 8 | RESERVED | 1 Byte +----------------------------------+ 9 | EXCHANGE ORIGINATOR N_PORT ID | 3 Bytes +----------------------------------+ 12 | RESERVED | 1 Byte +----------------------------------+ 12 | EXCHANGE RESPONDER N_PORT ID | 3 Bytes +----------------------------------+ 16 | DATA TRANSFER COUNT | 4 Bytes +----------------------------------+ 20 | E_STAT | 4 Bytes +----------------------------------+ Total Length = 24 Upon transmitting the ACC response to the IP network, the iFCP layer shall clear the EXCHANGE ORIGINATOR N_PORT ID and EXCHANGE RESPONDER N_PORT ID fields when encapsulating the ACC message into iFCP. Furthermore, the iFCP AUGMENTED DATA field in the encapsulating iFCP header shall contain the EXCHANGE RESPONDER field, which is identical to the EXCHANGE ORIGINATOR value used to augment the original REC request message. REC OX_ID - The OX_ID of the exchange for which information is being returned. It should be identical to the REC OX_ID field in the REC request. REC RX_ID - The RX_ID of the exchange for which information is being returned. This field should also be identical to the REC RX_ID field in the REC request. When receiving the ACC response message from the IP network, the iFCP layer shall use the N_PORT fabric addresses (in the Fibre Channel header) and EXCHANGE RESPONDER (in the iFCP header) to replace the EXCHANGE ORIGINATOR N_PORT ID and EXCHANGE RESPONDER N_PORT ID fields before passing the Link Service message on to the upper Fibre Channel layers. DATA TRANSFER COUNT - The number of bytes received by a destination N_PORT for a write command, or the number of bytes received for a read command. E_STAT (Exchange Status) - Of the bits defined in this field, two are particularly significant to iFCP: Mullendore, Monia, Tseng 21 iFCP November 2000 Bit Field Description Significance --------- ----------- ------------ 30 SEQUENCE INITIATIVE 0 = Other port holds sequence init 1 = This port holds sequence init 29 Completion 0 = Open 1 = Closed An LS_RJT reply may be sent in lieu of ACC, to indicate that the REC was rejected. 7.1.5 Discover Address (ADISC) ADISC allows devices to exchange name (Port and Node) information. This allows verification of Port and Node addresses. The following shows the format of the ADISC request message. The ACC response is identical except the command code is replaced with the ACC code (0x02000000), and the PORT_NAME and NODE_NAME fields are those of the responder. Byte MSb LSb Offset 7 6 5 4 3 2 1 0 +----------------------------------+ 0 | LS_COMMAND | 4 Bytes +----------------------------------+ 4 | RESERVED | 1 Byte +----------------------------------+ 5 | HARD ADDRESS | 3 Bytes +----------------------------------+ 8 | ORIGINATOR/RESPONDER PORT NAME | 8 Bytes +----------------------------------+ 16 | ORIGINATOR/RESPONDER NODE NAME | 8 Bytes +----------------------------------+ 24 | RESERVED | 1 Byte +----------------------------------+ 25 | ORIGINATOR N_PORT ID | 3 Bytes +----------------------------------+ Total Length = 28 The HARD ADDRESS field has no meaning for iFCP. The field may contain non-zero values in the request message, but shall be zeroed in the ADISC response. Upon transmission to the IP network, the ORIGINATOR N_PORT ID shall be zeroed in both the request and ACC response messages. Upon receipt of the request or ACC response from the IP network, the iFCP layer shall use the ORIGINATOR/RESPONDER PORT_NAME and NODE_NAME to replace the ORIGINATOR N_PORT ID with the appropriate value, before forwarding the message on to upper Fibre Channel layers. Mullendore, Monia, Tseng 22 iFCP November 2000 The following parameters are used for the ADISC request message. ORIGINATOR PORT NAME - This field contains the World Wide Port Name (WWPN) of the iFCP port from which the ADISC request is being originated. ORIGINATOR NODE NAME - This field contains the WWPN of the N_PORT from which the ADISC request is being originated. 7.1.6 Request Exchange Status (RES) RES requests the status of a specific exchange. The RES request is sent in a new exchange. The following shows the RES message format. Byte MSb LSb Offset 7 6 5 4 3 2 1 0 +----------------------------------+ 0 | LS_COMMAND (0x08000000) | 4 Bytes +----------------------------------+ 4 | RESERVED | 1 Byte +----------------------------------+ 7 | EXCHANGE ORIGINATOR S_ID | 3 Bytes +----------------------------------+ 8 | RES OX_ID | 2 Bytes +----------------------------------+ 10 | RES RX_ID | 2 Bytes +----------------------------------+ Total Length = 12 Upon transmitting the RES Link Service message to the IP network, the iFCP layer shall clear the EXCHANGE ORIGINATOR S_ID field. Furthermore, the iFCP layer shall add the EXCHANGE ORIGINATOR to the iFCP AUGMENTED DATA field in the iFCP header. The EXCHANGE ORIGINATOR is a 4-byte field. When 0x00000001, itindicates that the RES requester is the originating N_PORT of the exchange for which status is being requested. When 0x00000000, indicates that the RES recipient is the originator of the exchange for which status is being requested. RES OX_ID - Specifies the OX_ID of the exchange for which status is being requested. RES_RX_ID - Specifies the RX_ID of the exchange for which status is being requested. If the RES recipient does not have an Exchange Status Block for the specified exchange, it shall respond with an LS_RJT message with a reason code of "UNABLE TO PERFORM COMMAND REQUEST". Mullendore, Monia, Tseng 23 iFCP November 2000 Upon receipt of the RES Link Service message from the IP network, the iFCP layer shall use the N_PORT fabric addresses (in the Fibre Channel header) and the EXCHANGE ORIGINATOR value (in the iFCP header) to replace the EXCHANGE ORIGINATOR S_ID field before forwarding the message on to the upper Fibre Channel layers. The payload for the ACC response is a variable-length block of information called the EXCHANGE STATUS BLOCK. The Exchange Status Block (ESB) is used throughout an exchange to track the exchange's progress and identify which ports holds sequence initiative. The ESB has the following format shown below. Byte MSb LSb Offset 7 6 5 4 3 2 1 0 +----------------------------------+ 0 | ESB OX_ID | 1 Byte +----------------------------------+ 2 | ESB RX_ID | 2 Bytes +----------------------------------+ 4 | RESERVED | 1 Byte +----------------------------------+ 5 | ORIGINATOR N_PORT ID | 3 Bytes +----------------------------------+ 4 | RESERVED | 1 Byte +----------------------------------+ 5 | RESPONDER N_PORT ID | 3 Bytes +----------------------------------+ 6 | E_STAT | 4 Bytes +----------------------------------+ | RESERVED | 4 Bytes +----------------------------------+ 8 | SERVICE PARAMETERS | 112 Bytes +----------------------------------+ 88 | OLDEST SHORT SSB | 8 Bytes +----------------------------------+ 96 | INTERMEDIATE SHORT SSB | X*8 Bytes +----------------------------------+ 96 + X*8 | NEWEST SHORT SSB | 8 Bytes +----------------------------------+ Total Length = 104 + X*8 ESB OX_ID - The OX_ID for the exchange status saved in the ESB. ESB RX_ID - The RX_ID for the exchange status saved in the ESB. Upon transmitting the ACC reply to the IP network, the iFCP layer shall clear the ORIGINATOR N_PORT_ID and RESPONDER N_PORT ID fields. Furthermore, the iFCP layer shall add a 4-byte EXCHANGE ORIGINATOR to the iFCP AUGMENTED DATA field in the iFCP header. When the EXCHANGE ORIGINATOR field is 0x00000001, this indicates that the port sending the RES message (or receiving the ACC reply) is the originator of the exchange whose status is saved in the ESB. Mullendore, Monia, Tseng 24 iFCP November 2000 When the EXCHANGE ORIGINATOR is 0x00000000, this indicates that the port receiving the RES message (or sending the ACC reply) is the originator of the exchange whose status is saved in the ESB. Upon receipt of the ACC reply from the IP network, the iFCP layer shall use the N_PORT fabric addresses (in the Fibre Channel header) and the EXCHANGE ORIGINATOR value (in the iFCP header) to replace the ORIGINATOR and RESPONDER N_PORT ID fields before forwarding the message on to the upper Fibre Channel layers. More information on RES and the Exchange Status Block (ESB) can be found in [FC-PH]. 7.1.7 Request Sequence Initiative (RSI) RSI allows a sequence recipient to request sequence initiative be passed for an open exchange. The RSI recipient responds in the following manner if the RSI request identifies an open exchange. The following shows the format of the RSI request message. Byte MSb LSb Offset 7 6 5 4 3 2 1 0 +----------------------------------+ 0 | LS_COMMAND (0x12000000) | 4 Bytes +----------------------------------+ 4 | RESERVED | 1 Byte +----------------------------------+ 7 | EXCHANGE ORIGINATOR S_ID | 3 Bytes +----------------------------------+ 8 | RSI OX_ID | 2 Bytes +----------------------------------+ 10 | RSI RX_ID | 2 Bytes +----------------------------------+ Total Length = 12 Upon transmitting the RSI Link Service message to the IP network, the iFCP layer shall clear the EXCHANGE ORIGINATOR S_ID field. Furthermore, the iFCP layer shall add a 4-byte EXCHANGE ORIGINATOR to the iFCP AUGMENTED DATA field in the iFCP header. The EXCHANGE ORIGINATOR field, when 0x00000001, indicates that the RSI requester is the originator of the exchange for which initiative is being requested. When EXCHANGE ORIGINATOR is set to 0x00000000, this indicates the RSI recipient is the originator of the exchange for which initiative is being requested. RSI OX_ID - Specifies the OX_ID of the exchange for which sequence initiative is being requested. RSI RX_ID - Specifies the RX_ID of the exchange for which sequence initiative is being requested. Mullendore, Monia, Tseng 25 iFCP November 2000 Upon receipt of the RSI message from the IP network, the iFCP layer shall use the N_PORT fabric addresses (in the Fibre Channel header) and the EXCHANGE ORIGINATOR value (in the iFCP header) to replace the EXCHANGE ORIGINATOR S_ID field before forwarding the message on to the upper Fibre Channel layers. An ACC response is sent after the sequence initiative has been transferred, or if it already has been transferred. The ACC response only has the 4-byte LS_COMMAND field as the payload. An LS_RJT response may be sent if the RSI parameters do not specify an open exchange (invalid OX_ID/RX_ID combination). 7.1.8 Read Sequence Status (RSS) RSS requests the status of a specific sequence in an exchange. The following shows the format of the RSS request. Byte MSb LSb Offset 7 6 5 4 3 2 1 0 +----------------------------------+ 0 | LS_COMMAND (0x09000000) | 4 Bytes +----------------------------------+ 4 | SEQ_ID | 1 Byte +----------------------------------+ 5 | EXCHANGE ORIGINATOR S_ID | 3 Bytes +----------------------------------+ 8 | RSS OX_ID | 2 Bytes +----------------------------------+ 10 | RSS RX_ID | 2 Bytes +----------------------------------+ Total Length = 12 Upon transmitting the RSS Link Service message to the IP network, the iFCP layer shall clear the EXCHANGE ORIGINATOR S_ID field. Furthermore, the iFCP layer shall add a 4-byte EXCHANGE ORIGINATOR to the iFCP AUGMENTED DATA field in the iFCP header. The EXCHANGE ORIGINATOR field, when 0x00000001, indicates that the RSS requester is the originator of the exchange for which sequence status is being requested. When EXCHANGE ORIGINATOR is set to 0x00000000, this indicates the RSS recipient is the originator of the exchange for which sequence status is being requested. SEQ_ID - Specifies the SEQ_ID of the sequence for which status is being requested. RSS OX_ID - Specifies the OX_ID of the exchange for which sequence status is being requested. RSS RX_ID - Specifies the RX_ID of the exchange for which sequence status is being requested. Mullendore, Monia, Tseng 26 iFCP November 2000 Upon receipt of the RSS message from the IP network, the iFCP layer shall use the N_PORT fabric addresses (in the Fibre Channel header) and the EXCHANGE ORIGINATOR value (in the iFCP header) to replace the EXCHANGE ORIGINATOR S_ID field before forwarding the message on to the upper Fibre Channel layers. The ACC response payload for an RSS request consists of a single 16-byte field, the Sequence Status Block (SSB), shown below. If the RSS recipient does not have a Sequence Status Block, it shall respond with an LS_RJT with a reason code of "UNABLE TO PERFORM COMMAND REQUEST". More information on the Sequence Status Block can be found in [FC-PH]. 7.2 TCP Link Service Messages TCP Link Service Messages are used only to manage TCP connections. They are different from other ELS messages in that they are only passed between peer FCP Portals, and are only processed within the iFCP layer. The response to the TCP Link Service Message (if any) will echo the original request. The LS_COMMAND value for the response remains the same as that used for the request. Additionally, the ABTS request shall never be generated for any TCP Link Service Message. The Link Service frame carrying a TCP ELS message is identified by the TCP ELS bit being set in the iFCP FLAGS field of the iFCP header. Additionally, the TYPE field is 0x01 and R_CTL field is 0x22 for the request, and 0x23 for the reply. The following lists the TCP Link Service messages and their corresponding LS_COMMAND values. Request LS_COMMAND Short Name iFCP Support ------- ---------- ---------- ------------ Control Connection Bind 0xE0 CBIND REQUIRED Unbind Connection 0xE4 UNBIND OPTIONAL TCP Message 0xE8 TCPMSG REQUIRED Network Connection Interfaces 0xED NINTF REQUIRED 7.2.1 Network Connection Interfaces (NINTF) NINTF allows an FCP Portal to request information on other network interfaces that may be used to establish connections. This extended link service will return the number of network interfaces available, and an interface descriptor record for a single interface. Since each NINTF request returns information on one interface, multiple NINTF requests are required to obtain information on more than one interface. The following shows the format of the NINTF request message. Mullendore, Monia, Tseng 27 iFCP November 2000 Byte MSb LSb Offset 7 6 5 4 3 2 1 0 +----------------------------------+ 0 | LS_COMMAND (0xED000000) | 4 Bytes +----------------------------------+ 4 | USER INFO | 4 Bytes +----------------------------------+ 8 | INTERFACE KEY | 2 Bytes +----------------------------------+ 10 | RESERVED | 2 Bytes +----------------------------------+ Total Length = 12 USER INFO - Contains any data desired by the requester. The value will be simply returned by the recipient as in the FLUSH reply. INTERFACE KEY - Contains an index to the interface for which the NINTF message is querying. Each interface at the destination shall be sequentially numbered beginning with 1. If the number of interfaces supported by the message recipient is unknown, then this field shall contain 0. In the NINTF response, the recipient will indicate the number of interfaces supported. Each of these interfaces can be referenced in subsequent NINTF messages by the sender by setting the INTERFACE KEY value up to the highest- numbered interface. The following shows the format of the NINTF response. Byte MSb LSb Offset 7 6 5 4 3 2 1 0 +----------------------------------+ 0 | LS_COMMAND (0xED000000) | 4 Bytes +----------------------------------+ 4 | USER INFO | 4 Bytes +----------------------------------+ 8 | RESERVED | 3 Bytes +----------------------------------+ 11 | INTERFACES AVAILABLE (A) | 1 Byte +----------------------------------+ 12 | INTERFACE RECORDS | X Bytes +----------------------------------+ Total Length = X + 12 USER INFO - The 4-byte field is the same value as the USER INFO in the NINTF request. The recipient echoes this value back to the sender, and does not perform any operation using this field. INTERFACES AVAILABLE (A) - This parameter specifies the number of additional network interfaces that may be used to establish data connections attached to the control connections. This parameter may be 0 indicating that data connections attached to this control connection must use the same network interface as the control Mullendore, Monia, Tseng 28 iFCP November 2000 connection. The value stored in this field also specifies the number (A) of network interface records that are present at the end of the message. INTERFACE RECORDS - This field contains A interface records describing each of the network interfaces. An interface record consists of 5 parameters as shown in below. Byte MSb LSb Offset 7 6 5 4 3 2 1 0 +----------------------------------+ 0 | RECORD LENGTH | 1 Byte +----------------------------------+ 1 | IP ADDRESS TYPE | 1 Byte +----------------------------------+ 2 | INTERFACE HANDLE | 2 Bytes +----------------------------------+ 4 | RESERVED | 4 Bytes +----------------------------------+ 8 | INTERFACE SPEED | 4 Bytes +----------------------------------+ | IP ADDRESS | X-12 Bytes +----------------------------------+ Total Length = X RECORD LENGTH - Defines the total length, in bytes, of the INTERFACE RECORD, including the RECORD LENGTH field. This value shall be a multiple of 4 bytes. IP ADDRESS TYPE - Defines the type of address in the IP ADDRESS field. 0x01 indicates IPv4, 0x02 indicates Ipv6. INTERFACE HANDLE - This 16-bit field contains an identifying number (i.e., handle) assigned to the interface by the destination N_PORT. INTERFACE SPEED - This parameter specifies the data rate of the interface in bits per second. The value in this field is the data rate divided by 1024. For example, a value of 1024 indicates a data rate of 1048576 bits per second. IP ADDRESS - This field contains the IP address of the network interface for which information is being returned. If the address type is N bytes long and the field is larger than N, the address shall be in the first N bytes of the field with the remainder of the field set to 0. 7.2.2 Control Connection Bind (CBIND) The CBIND message binds an N_PORT login session to a specific TCP connection. In the CBIND request message, the source and destination N_Port is identified by its N_PORT network address. The following shows the format of the CBIND request. Mullendore, Monia, Tseng 29 iFCP November 2000 Byte MSb LSb Offset 7 6 5 4 3 2 1 0 +----------------------------------+ 0 | LS_COMMAND (0xE0000000) | 4 Bytes +----------------------------------+ 4 | USER INFO | 4 Bytes +----------------------------------+ 8 | SOURCE PORT NAME | 8 Bytes +----------------------------------+ Length = 16 USER INFO - Contains any data desired by the requester. This info is echo-ed back by the recipient. SOURCE PORT NAME - Contains the originating N_PORT's World Wide Port Name (WWPN). The FCP Portal uses this to verify that there is no pre-existing N_PORT session between the source and destination N_PORTs. [The response to this error condition will be handled in a future release of this specification] The following shows the format of the CBIND response. Byte MSb LSb Offset 7 6 5 4 3 2 1 0 +----------------------------------+ 0 | LS_COMMAND (0xE0000000) | 4 Bytes +----------------------------------+ 4 | USER INFO | 4 Bytes +----------------------------------+ 8 | DESTINATION PORT NAME | 8 Bytes +----------------------------------+ 16 | RESERVED | 2 Bytes +----------------------------------+ 18 | CBIND STATUS | 2 Bytes +----------------------------------+ 20 | RESERVED | 2 Bytes +----------------------------------+ 22 | CONTROL CONNECTION HANDLE | 2 Bytes +----------------------------------+ Total Length = 24 USER INFO - Contains the same value received in the USER INFO field of the CBIND request message. DESTINATION PORT NAME - Contains the destination N_PORT's World Wide Port Name (WWPN). CBIND STATUS - Indicates success or failure of the CBIND request. CBIND values are shown below. Mullendore, Monia, Tseng 30 iFCP November 2000 Value Description ----- ----------- 0 Successful - No Other Status 1-15 Reserved 16 Failed - Unspecified Reason 17 Failed - No Such device 18 Failed - N_PORT session already exists 19 Failed - Lack of Resources Others Reserved CONTROL CONNECTION HANDLE (CHANDLE) - Contains a value assigned by the FCP Portal to identify the control connection 7.2.3 Data Connection Bind (DBIND) [This section is TBD] 7.2.4 Unbind Data Connection (UNBIND) UNBIND is used to release a bound TCP data connection and return it to the pool of unbound TCP connections. This message is transmitted in the data connection that is to be unbound. The UNBIND message is an implied FLUSH. The following is the format of the UNBIND request message. Byte MSb LSb Offset 7 6 5 4 3 2 1 0 +----------------------------------+ 0 | LS_COMMAND (0xE4000000) | 4 Bytes +----------------------------------+ 4 | USER INFO | 4 Bytes +----------------------------------+ 8 | CONNECTION CONTROL HANDLE | 4 Bytes +----------------------------------+ 12 | RESERVED | 8 Bytes +----------------------------------+ Total Length = 20 CONTROL CONNECTION HANDLE (CHANDLE) - Contains a value assigned by the FCP Portal to identify the control connection The following shows the format of the UNBIND response message. Mullendore, Monia, Tseng 31 iFCP November 2000 Byte MSb LSb Offset 7 6 5 4 3 2 1 0 +----------------------------------+ 0 | LS_COMMAND (0xE4000000) | 4 Bytes +----------------------------------+ 4 | USER INFO | 4 Bytes +----------------------------------+ 8 | CONNECTION CONTROL HANDLE | 48 Bytes +----------------------------------+ 16 | RESERVED | 10 Bytes +----------------------------------+ 26 | UNBIND STATUS | 2 Bytes +----------------------------------+ 28 | RESERVED | 2 Bytes +----------------------------------+ 30 | CONNECTION HANDLE | 2 Bytes +----------------------------------+ Total Length = 32 UNBIND STATUS - Indicates the success or failure of the UNBIND request. Value Description ----- ----------- 0 Successful - No Other Status 1-15 Reserved 16 Failed - Unspecified Reason 17 Failed - No Such device 18 Failed - Control Connection ID Invalid Others Reserved CONNECTION HANDLE (CHANDLE) - Contains a value assigned by the FCP Portal to identify an active control connection 7.2.5 TCP Message (TCPMSG) TCPMSG sends an error message to another iFCP port. TCPMSG differs from other messages in that there is no reply to TCPMSG (both the first and last sequence in a exchange). The primary purpose for TCPMSG is to generate a message informing an iFCP port that a fatal FCP/TCP protocol error was detected, and all connections (control and data) established with the iFCP port are being closed. TCPMSG can also be used to send "Informative" or "Warning" messages that may be used for debugging or diagnostic purposes. The format of the TCPMSG request message follows. Mullendore, Monia, Tseng 32 iFCP November 2000 Byte MSb LSb Offset 7 6 5 4 3 2 1 0 +----------------------------------+ 0 | LS_COMMAND (0xEE000000) | 4 Bytes +----------------------------------+ 4 | RESERVED | 4 Bytes +----------------------------------+ 8 | ERROR CODE | 2 Bytes +----------------------------------+ 10 | TCPMSG FLAGS | 1 Byte +----------------------------------+ 11 | MSG LENGTH (L) | 1 Byte +----------------------------------+ 12 | MSG | L Bytes +----------------------------------+ Total Length = L + 12 ERROR CODE - Specifies one of the predefined error messages shown in the following table. This field is valid only if the FATAL bit is 1 and MSG LENGTH is 0 in the TCPMSG FLAGS field. Error Code Message ---------- ------- 0x0001 Loss of Synchronization on Control Connection 0x0002 Loss of Synchronization on Data Connection 0x0003 Received FCP_CMND IU on Data Connection 0x0004 RESERVED 0x0005 Received FCP_CONF IU on Data Connection 0x0006 RESERVED 0x0007 Received Unexpected Link Service Message on Data Connection 0x0008 Received Unexpected Data on Control Connection 0x0009 Received Unexpected Data on Data Connection Others RESERVED TCPMSG FLAGS - This field contains 3 bit flags that specify how the recipient should interpret the received message. Bit Field Flag Description --------- ---- ----------- 7:3 RESERVED 2 INFORMATIVE Indicates the message is informative, usually for debugging purposes. These messages may be discarded. 1 WARNING Indicates the message is a warning. Processing of warning messages is required and implementation-specific. 0 FATAL Indicates a fatal protocol error has been detected. Sender is terminating Mullendore, Monia, Tseng 33 iFCP November 2000 login sessions with the recipient and closing all TCP connections. The recipient shall implicit logout the sender of the message and close TCP connections to the sender. A WARNING or INFORMATIVE message shall not cause the recipient to alter the operating environment. When more than one TCPMSG FLAG bit is set, the message shall be considered Fatal. When no flags are set, the message shall be discarded. MSG LENGTH - Specifies the length in bytes of the MSG field. The length must be a multiple of 4 and can be a value of between 0 and 128. A value of 0 indicates the MSG field is not present. 8 Error Detection and Recovery Procedures for iFCP 8.1 Overview [FCP-2], [FC-PH], and [FC-PH-2] define error detection and recovery procedures. These Fibre Channel-defined mechanisms continue to be available in the iFCP environment. 8.2 Timer Definitions 8.2.1 Error_Detect_Timeout (E_D_TOV) E_D_TOV is "a reasonable timeout value for detection of a response to a time event". The default value specified by FC-PH of 10 seconds will be also used as the iFCP default value. E_D_TOV is the maximum time allowed between the transmission of consecutive data frames within a sequence. For Class 2 service, E_D_TOV specifies the maximum time interval between transmission of a frame, and receipt of the ACK for that frame. [The policy for setting E_D_TOV for an IP fabric is TBS] 8.2.2 Resource Allocation Timeout (R_A_TOV) R_A_TOV is defined in FC-PH-2 as "the maximum transit time within a fabric to guarantee that a lost frame will never emerge from the fabric". A value of 2 x R_A_TOV is the minimum time that the originator of an ELS request or FC-4 ELS request shall wait for the response to that request. [The policy for setting R_A_TOV for an IP fabric is TBS] 8.2.3 Resource Recovery Timer (RR_TOV) [The content of this section is TBD] Mullendore, Monia, Tseng 34 iFCP November 2000 8.3 TCP Error Recovery Issues A failed TCP connection will result in a dropped N_PORT session. [The remainder of this section is TBD] 8.4 iFCP Protocol Error iFCP protocol errors between FCP Portals shall be considered fatal errors resulting in the termination of the login sessions and closing of the TCP sessions. An iFCP protocol error occurs when Fibre Channel frames are sent on the wrong TCP connection. One example of a protocol error is receiving an FCP_CMND IU on the data connection. If an iFCP port detects an iFCP/TCP protocol error on a connection, the port shall transmit a TCPMSG message on the control connection (if one exists) with the appropriate error code. The FCP_Portal shall then implicitly log out and close all TCP connections established with the iFCP port, and ignore all data received on these TCP connections until they are reopened. [The information returned to the N_PORT upon occurence of an iFCP protocol error will be specified in the next revision of this specification] 9. Security 9.1 Overview As with any other IP-based network, an iFCP storage network has security issues which must be addressed with the appropriate security policies and enforcement resources. There are various levels of security paradigms which when applied appropriately to an iFCP network can provide sufficient levels of security, including data integrity, authentication, and privacy, depending on user needs. 9.2 Physical Security Most existing SCSI and Fibre Channel interconnections are deployed in private, physically isolated environments where hostile entities are not provided access to the SCSI and Fibre Channel interconnects. This is the most basic security mechanism, and may be a sufficient model in some cases for an iFCP network. 9.3 Controlling Access A second level of security is the use of zoning. Zoning specifies which devices are allowed to communicate, and is similar in concept Mullendore, Monia, Tseng 35 iFCP November 2000 to VLAN (Virtual Local Area Network) technology. Zoning information is maintained in a Name Server. 9.4 Authentication and Encryption Where additional levels of data integrity and privacy are required for iFCP, existing IPSec specifications can be applied to iFCP. Because IPSec is a layer-3 technology and has no knowledge of TCP, UDP, or higher-level protocols such as iFCP and FCP, it can be applied transparently to iFCP. The following IETF documents describe the operational framework and automatic keying mechanisms for IPSec. RFC2401 Security Architecture for the Internet Protocol RFC2402 IP Authentication Header RFC2406 IP Encapsulating Security Payload RFC2407 The Internet IP Security Domain of Interpretation for ISAKMP RFC2408 Internet Security Association and Key Management Protocol (ISAKMP) RFC2409 The Internet Key Exchange (IKE) 9.5 Storage Firewalls Firewalls are a common and proven methodology for securing access to IP-based networks, and they can be appropriate for use in IP- based storage networks as well. A firewall is a choke point through which all transit traffic must transit in order to pass between two separate networks. Since all iFCP traffic uses a well- known IANA-assigned TCP port number, it can easily be recognized and inspected. Access to storage resources can be secured by setting up a single gateway through which all outside non-secured traffic must pass through in order to access resources in the storage network. Such a firewall can be a proxy host operating at the session or application layer, requiring authentication before allowing traffic to pass. It can also be a stateful inspection gateway which understands the iFCP protocol, and can passively inspect and discover security threats as they transit the gateway. A third option is to use a standard router access control list to filter authorized traffic based upon static parameters such as IP addresses and TCP port numbers. 10. Glossary 10.1 Definitions address identifier - An address value used to identify source (S_ID) or destination (D_ID) of a frame. application client - An object that is the source of SCSI commands. Mullendore, Monia, Tseng 36 iFCP November 2000 application client buffer offset - Offset in bytes from the start or base address of the application client's data buffer to the location for the transfer of the first byte of a data delivery service request. base address - The address of the lowest address byte to be transferred to or from an application client buffer. command - A request describing a unit of work to be performed by a device server. command descriptor block - A structure used to communicate a command from an application client to a device server. device server - An object within the logical unit which executes SCSI tasks and enforces the rules for task management. FCP I/O operation - An unlinked SCSI command, a series of linked SCSI commands, or a task management function. FCP_Port - An N_PORT or NL_Port that supports the SCSI Fibre Channel Protocol. fully qualified exchange identifier - A token used to uniquely identify a FCP I/O operation. information unit - An organized collection of data specified by the FCP to be transferred as a single sequence by the Fibre Channel service interface. initiator - A SCSI device containing application clients that originate device service requests and task management functions to be processed by a target SCSI device. logical unit - A target resident entity that implements a device model and executes SCSI commands sent by an application client. logical unit identifier - Identifier used by an initiator to reference the logical unit. logical unit number - An encoded 64-bit identifier for a logical unit. NL_Port - An N_PORT that contains arbitrated loop functions associated with the Fibre Channel Arbitrated Loop topology. N_PORT - A hardware entity that supports FC-PH. It may act as an originator, a responder, or both. Originator - The logical function associated with an N_PORT responsible for originating an Exchange. Mullendore, Monia, Tseng 37 iFCP November 2000 Originator Exchange Identifier (OX_ID) - An identifier assigned by an Originator to identify an Exchange and meaningful only to the Originator. Port Login (PLOGI) - The Fibre Channel Extended Link Service (ELS) that exchanges identification and operation parameters between an originating N_PORT and a responding N_PORT. recovery abort - FC-PH protocol that recovers FCP_Port resources by terminating the Exchange and FCP I/O Operation. request byte count - Number of bytes to be moved by a data delivery service request. Responder - The logical function in an N_PORT responsible for supporting the Exchange initiated by the Originator in another N_PORT. Responder Exchange Identifier (RX_ID) - An identifier assigned by a Responder to identify an exchange, and meaningful only to the Responder. Source_Identifier - The address identifier used to indicate the source port of the transmitted frame. SCSI device - A device that originates or services SCSI commands. tag - The initiator-specified component of the task identifier. target - A SCSI device that receives SCSI commands and directs such commands to one or more logical units for execution. target identifier - Address of up to 64 bits by which a target is identified. task - An object within the logical unit representing the work associated with a command or group of linked commands. task attribute - The queuing specification for a task (SIMPLE, ORDERED, HEAD OF QUEUE, ACA). task identifier - The information uniquely identifying a task task management function - A peer-to-peer confirmed service provided by a task manager that can be invoked by an application client to affect the execution of one or more tasks. 10.2 Abbreviations and Acronyms ABTS Abort Sequence ACA Automatic Contingent Allegiance Mullendore, Monia, Tseng 38 iFCP November 2000 AH Authentication Header AL_PA Arbitrated Loop Physical Address API Application Program Interface BLS Basic Link Service CDB Command Data Block CRC Cyclic Redundancy Checksum CRN Command Reference Number DHCP Dynamic Host Configuration Protocol DOI Domain of Interpretation HBA Host Bus Adapter ELS Extended Link Service EOF End of Frame EPDC Enable Precise Delivery Checking ESP Encapsulating Security Payload FC Fibre Channel FC-AL Fibre Channel Arbitrated Loop FC-FLA Fibre Channel Fabric Loop Attachment FCP Fibre Channel Protocol for SCSI HBA Host Bus Adapter IANA Internet Assigned Numbers Authority ICMP Internet Control Message Protocol IEEE The Institute of Electrical and Electronic Engineers IKE Internet Key Exchange I/O Input/Output IP Interet Protocol ISAKMP Internet Security Association and Key Management Protocol IU Information Unit JBOD Just a Bunch of Disks LAN Local Area Network LSb Least Significant Bit LSB Least Significant Byte LUN Logical Unit Number MAN Metropolitan Area Network MPLS Multi-Protocol Label Switching MSb Most Significant Bit MSB Most Significant Byte MTU Maximum Transmission Unit NAA Network Address Authority NOP No Operation OUI Organizationally Unique Identifier OX_ID Originator Exchange ID RDMA Remote Direct Memory Access RX_ID Responder Exchange ID RAID Redundant Arrays of Inexpensive Disks SAM SCSI Architecture Model SAM-2 SCSI Architecture Model - 2 SAN Storage Area Network SCSI Small Computer Systems Interface SDB Stream Data Block SNS Storage Name Server SOF Start of Frame Mullendore, Monia, Tseng 39 iFCP November 2000 TBD To Be Determined TCP Transport Control Protocol UDP User Datagram Protocol ULA Universal LAN MAC Address ULP Upper Level Protocol (or Upper Layer Protocol) VPN Virtual Private Network WAN Wide Area Network WWN World Wide Name WWNN World Wide Node Name WWPN World Wide Port Name 11. References 11.1 Relevant SCSI (T10) Specifications The following documents are available from: Global Engineering, 15 Inverness Way East, Englewood, CO 80112-5704. Telephone (800) 854-7179 or (303) 792-2181, Fax: (303) 792-2192 [SAM] SCSI-3 Architecture Model (SAM), ANSI X3.270-1996 [SAM-2] SCSI Architecture Model-2 (SAM-2), Project 1157-D, revision 11 [SPC] SCSI Primary Commands (SPC), ANSI X3.301-1997 [SPC-2] SCSI Primary Commands-2 (SPC-2), Project 1236-D, revision 16 [FCP] Fibre Channel Protocol for SCSI (FCP), ANSI X3.269- 1996 [FCP-2] Fibre Channel Protocol for SCSI, Second Revision (FCP- 2), Project 1144D, revision 04 11.2 Relevant Fibre Channel (T11) Specifications The following documents are available from: Global Engineering, 15 Inverness Way East, Englewood, CO 80112-5704. Telephone (800) 854-7179 or (303) 792-2181, Fax: (303) 792-2192 [FC-PH] Fibre Channel Physical and Signaling Interface (FC-PH) Rev 4.3, ANSI X3.230:1994 [FC-PH-2] Fibre Channel Physical and Signaling Interface (FC-PH- 2) Rev 7.4, ANSI X3.297:1997 [FC-PH-3] Fibre Channel Physical and Signaling Interface (FC-PH- 3) Rev 9.4, ANSI X3.303:1998 [FC-FG] Fibre Channel Generic Requirements (FC-FG) Rev 3.5, ANSI X3.289:1996 Mullendore, Monia, Tseng 40 iFCP November 2000 [FC-GS-2] Fibre Channel Generic Services (FC-GS-2) Rev 5.2, ANSI NCITS 288 [FC-AL] Fibre Channel Arbitrated Loop (FC-AL) Rev 4.5, ANSI X3.272:1996 [FC-AL-2] Fibre Channel Arbitrated Loop (FC-AL-2) Rev 7.0, NCITS 332:1999 [FC-PLDA] Fibre Channel Private Loop SCSI Direct Attachment (FC- PLDA), NCITS TR-19:1998 [FC-FLA] Fibre Channel Fabric Loop Attachment (FC-FLA), NCITS TR-20:1998 [FC-TAPE] Fibre Channel Tape and Tape Medium Changers (FC-TAPE), NCITS TR-24:1999 11.3 Relevant RFC Documents [RFC768] User Datagram Protocol [RFC791] Internet Protocol, DARPA Internet Program Protocol Specification [RFC1146] TCP Alternate Checksum Options [RFC2401] Security Architecture for Internet Protocol [RFC2402] IP Authentication Header [RFC2406] Encapsulating Security Protocol (ESP) [RFC2407] The Internet IP Security Domain for ISAKMP [RFC2408] Internet Security Association and Key Management Protocol (ISAKMP) [RFC2409] The Internet Key Exchange (IKE) [RFC2460] Internet Protocol, Version 6 (IPv6) Specification 11.4 Other Reference Documents Fibre Channel, Gigabit Communications and I/O for Computer Networks, Alan F. Beener, McGraw-Hill, ISBN 0-07-005669-2 The Fibre Channel Consultant, A Comprehensive Introduction, Robert W. Kembel, Northwest Learning Associates, ISBN 0-931836-82-6 Mullendore, Monia, Tseng 41 iFCP November 2000 The Fibre Channel Consultant, Arbitrated Loop, Rober W. Kembel, Connectivity Solutions, a division of Northwest Learning Associates, ISBN 0-931836-84-0 12. Author's Addresses Charles Monia Rod Mullendore Josh Tseng Nishan Systems 3850 North First Street San Jose, CA 95134 Phone: 408-519-3986 Email: cmonia@nishansystems.com Mullendore, Monia, Tseng 42 iFCP November 2000 Full Copyright Statement "Copyright (C) The Internet Society (date). All Rights Reserved. This document and translations of it may be copied and furnished to others, and derivative works that comment on or otherwise explain it or assist in its implmentation may be prepared, copied, published and distributed, in whole or in part, without restriction of any kind, provided that the above copyright notice and this paragraph are included on all such copies and derivative works. However, this document itself may not be modified in any way, such as by removing the copyright notice or references to the Internet Society or other Internet organizations, except as needed for the purpose of developing Internet standards in which case the procedures for copyrights defined in the Internet Standards process must be followed, or as required to translate it into 1 Bradner, S., "Key words for use in RFCs to Indicate Requirement Levels", BCP 14, RFC 2119, March 1997 Mullendore, Monia, Tseng 43