INTERNET DRAFT draft-hufferd-ips-iser-sctp-ib-00.txt John Hufferd Mike Ko IBM Corporation Yaron Haviv Voltaire Ltd December, 2004 Expires: June, 2005 Generalization of iSER for SCTP, Infiniband and other Network Protocols Status of this Memo By submitting this Internet-Draft, I certify that any applicable patent or other IPR claims of which I am aware have been disclosed, or will be disclosed, and any of which I become aware will be disclosed, in accordance with RFC 3668. Internet-Drafts are working documents of the Internet Engineering Task Force (IETF), its areas, and its working groups. Note that other groups may also distribute working documents as Internet- Drafts. Internet-Drafts are draft documents valid for a maximum of six months and may be updated, replaced, or obsoleted by other documents at any time. It is inappropriate to use Internet-Drafts as reference material or to cite them other than as "work in progress." The list of current Internet-Drafts can be accessed at http://www.ietf.org/1id-abstracts.html. The list of Internet-Draft Shadow Directories can be accessed at http://www.ietf.org/shadow.html. J.Hufferd et. al. Expires June 2005 1 Internet-Draft Generalization of the iSER Specification December 2004 Abstract The iSCSI Extensions for RDMA document [iSER] currently specifies the RDMA data transfer capability for [iSCSI] over iWARP/TCP. This document generalizes the iSER document to permit it to be used with other RDMA capable protocols such as iWARP/SCTP, InfiniBand, etc. It also describes what should be defined in the InfiniBand Trade Association and what are appropriate for IETF. J.Hufferd et. al. Expires June 2005 2 Internet-Draft Generalization of the iSER Specification December 2004 Table of Contents 1 Motivation..................................................5 2 Overall generalizations needed within the iSER specification6 2.1 Generalization of Definitions...............................6 2.1.1 The iWARP term............................................6 2.1.2 The RNIC term.............................................7 2.1.3 Steering Tag (STag).......................................7 2.1.4 Inbound RDMA Read Queue Depth (IRD) & Outbound RDMA Read Queue Depth (ORD)................................................7 2.1.5 RDMA Protocol (RDMAP).....................................8 2.1.6 RDMAP Stream..............................................8 2.1.7 RDMAP Message.............................................8 2.2 The following should be placed/updated in Acronym Section...9 2.3 Connection Establishment, Login, and Transition to iSER.....9 2.4 Security considerations.....................................9 2.5 Adjustments to Appendix....................................10 2.6 Add Appendix B.............................................10 3 Architectural discussion of iSER over InfiniBand...........11 3.1 The Host side of the InfiniBand iSCSI & iSER connections...11 3.2 The Storage side of iSCSI & iSER mixed network environment.13 3.3 Discovery processes for an InfiniBand Host.................14 3.4 IBTA Connection specifications.............................15 4 Preferred changes to iSCSI Discovery Data..................16 5 The Appropriate Standardization Organizations for iSER/IB..17 5.1 The IETF ips Working Group Standardization.................17 5.2 The InfiniBand Trade Association (IBTA) Standardization....17 5.3 IBTA and optional InfiniBand Features......................17 5.4 ServiceID and IP Port number...............................17 6 IANA Considerations........................................18 7 References.................................................19 7.1 Informative References.....................................19 8 Appendix...................................................20 8.1 Additional detailed [iSER] document modification...........20 8.1.1 Adjustment to Section 2.1 Motivation.....................20 8.1.2 Adjustment to Section 2.2 Architectural Goals............20 8.1.3 Adjustment to Section 2.3 Protocol Overview..............20 8.1.4 Adjustment to Section 2.4 RDMA services and iSER.........21 8.1.5 Adjustment to Section 2.7 iSCSI/iSER Layering............21 8.1.6 Generalization of Other iSER Sections....................22 8.1.7 Adjustments to 13.2 Informational References.............25 9 Author’s Address...........................................27 10 Acknowledgments............................................28 11 Full Copyright Statement...................................29 J.Hufferd et. al. Expires June 2005 3 Internet-Draft Generalization of the iSER Specification December 2004 Table of Figures Figure 2 iSCSI, and iSER on IB..................................12 Figure 3 Storage Controller with iSCSI, iWARP/TCP, iWARP/SCTP, and IB connections..................................................14 Figure 1 Generic example of iSCSI/iSER Layering in Full Feature Mode ................................................................22 J.Hufferd et. al. Expires June 2005 4 Internet-Draft Generalization of the iSER Specification December 2004 1 Motivation Currently the work to define iSCSI extensions for RDMA [iSER] only considers using the iWARP protocol suite over the TCP layer. While this objective meets the short term requirement since iSCSI is defined only for TCP, there is a huge benefit to generalize a standardized [iSER] so that it can be used with other types of RDMA capable layers now and in the future, including the following: . iWARP over SCTP . InfiniBand (with reliable connections, RC) The interest in using [iSER] for InfiniBand is based on exploiting the iSCSI protocol features and its discovery and management protocol instead of using the SCSI RDMA Protocol (SRP) which lacks the management and discovery support. Furthermore, with an iSCSI based protocol, the Storage Professional and/or Administrator only needs to understand and support a single basic protocol, which has similar implementations across a suite of different network types (iWARP/TCP, iWARP/SCTP, InfiniBand, etc.). It was to enable this vision and desire for a single Storage Protocol that the proposed generalizations to [iSER] were created. J.Hufferd et. al. Expires June 2005 5 Internet-Draft Generalization of the iSER Specification December 2004 2 Overall generalizations needed within the iSER specification This section will specify changes/adjustments that should be considered to the wordage in the iSER document to make it more general. The goal of these changes is not to modify the basic operation of iSCSI/iSER when operating on the TCP version of iWARP, but to change/adjust the wordage in such a way that iSCSI/iSER can be layered over a different RDMA-capable protocol layer, such as an SCTP version of iWARP, or InfiniBand. The details of many of the suggested changes can be found in the Appendix of this document. 2.1 Generalization of Definitions It is required that some of the terminology be clarified as to applicability of the terms to the actual LLP used. 2.1.1 The iWARP term As currently defined, the iWARP term has a strong TCP centric bias. We will introduce a new, more generic term, known as RDMA-Capable Protocol (RCP) to denote the protocol layer that provides the RDMA functionality for iSER. The following term will be added to the Definition section: RDMA-Capable Protocol – The protocol or protocol suite that provides the RDMA functionality, e.g., iWARP, Infiniband, etc. With these new definitions, the "iWARP" term will be generalized as follows: 1. Whenever the term "iWARP protocol suite" occurs in the iSER draft, it will be replaced by "RDMA-Capable Protocol". In addition, the phrase "such as the iWARP protocol suite" will be added where necessary. 2. Whenever the term "iWARP layer" occurs in the iSER draft, it will be replaced by "RDMA-capable protocol layer". In addition, the phrase "such as the iWARP Layer" will be added where necessary. 3. Whenever the term "iWARP" is used as an adjective in other context, it will be replaced with just RDMA, or "RDMA-Capable", whichever is appropriate. E.g., "iWARP functionality" will be replaced with "RDMA functionality". J.Hufferd et. al. Expires June 2005 6 Internet-Draft Generalization of the iSER Specification December 2004 4. Whenever the term "iWARP" is used as a shorthand for the iWARP protocol suite, it will be replaced by "RDMA-capable protocol". 5. Whenever iWARP is used as a specific implementation example intended for TCP only, such as "iWARP Message Format" in the appendix, it will be changed to iWARP/TCP. 2.1.2 The RNIC term The term "RNIC" has been generally accepted by the industry to mean an RDMA-enabled Network Interface Controller for the IP world. So to generalize iSER for any RDMA-capable protocol layer, we will introduce a new term known as RDMA-Capable Controller, defined as follows: . RDMA-Capable Controller – A network I/O adapter or embedded controller with RDMA functionality. E.g., for TCP/IP, this can be an RNIC, and for Infiniband, this could be a HCA (Host Channel Adapter) or TCA (Target Channel Adapter). Within the body of the iSER document the term RDMA-Capable Controller will be used whenever the intention is to talk about a general controller that provides RDMA functionality. In addition, the clause "such as an RNIC" will be added as necessary. Within the body of the iSER document, the term RNIC is left unchanged if it specifically or implicitly refers to TCP/IP. 2.1.3 Steering Tag (STag) The Steering Tag (STag) term needs to have its definition extended so that it applies to both a Tag for a Remote Buffer, and the Tag for a Local Buffer. The following should be considered as a replacement for the existing one in the definition section. Steering Tag (STag) - An identifier of a Tagged Buffer on a Node (Local or Remote) as defined in [RDMAP] and [DDP]. For other RDMA- Capable protocol layer, the Steering Tag may be known by different names. For example, for Infiniband, a Remote STag is known as an R- Key, and a Local STag is known as an L-Key. 2.1.4 Inbound RDMA Read Queue Depth (IRD) & Outbound RDMA Read Queue Depth (ORD) To generalize on the terms Inbound RDMA Read Queue Depth (IRD) and the Outbound RDMA Read Queue Depth (ORD) for other RDMA-Capable J.Hufferd et. al. Expires June 2005 7 Internet-Draft Generalization of the iSER Specification December 2004 protocol layers, the following should be added to the definition for IRD: "For other RDMA-Capable protocol layer, the term "IRD" may be known by a different name. For example, for Infiniband, the equivalent for IRD is the Responder Resources". For ORD, the following should be added: "For other RDMA-Capable Protocol Layer, the term "ORD" may be known by a different name. For example, for Infiniband, the equivalent for ORD is the Initiator Depth." 2.1.5 RDMA Protocol (RDMAP) In the body of the document the term "RDMA-Capable Protocol", or "RCP" should be used whenever any RDMA wire protocol or RDMA protocol stack is applicable. Only when the document intends to explicitly address a specific wire protocol would the term [RDMAP] be used. 2.1.6 RDMAP Stream The following should be considered for inclusion in the definition section to replace "RDMAP Stream": . RCP Stream - A single bidirectional association between the peer RDMA-capable protocol layers on two Nodes over a single transport-level stream. For TCP or SCTP, an RCP Stream is also known as an RDMAP Stream. For iSER/TCP, the association is created when the connection transitions to iSER-assisted mode following a successful Login Phase during which iSER support is negotiated. In the body of the document, the term "RCP Stream" will be used in place of "RDMAP Stream". 2.1.7 RDMAP Message The following should be considered for inclusion in the definition section to replace "RDMAP Message": . RCP Message – The sequence of packets of the RDMA-capable protocol which represent a single RDMA operation or a part of RDMA Read Operation. For TCP or SCTP, an RCP Message is also known as an RDMAP Message. In the body of the document, the term "RCP Message" will be used in place of "RDMAP Message". The exception is when the term "RDMAP Message" is used to describe the iSER Hello and HelloReply Messages. Here "RDMAP Message" will be replaced by "iSER Message" in order to accommodate LLPs that have message delivery capability, such as SCTP J.Hufferd et. al. Expires June 2005 8 Internet-Draft Generalization of the iSER Specification December 2004 or [IB]. The iSCSI layer may use that messaging capability immediately after connection establishment before enabling iSER- assisted mode. In this case the iSER Hello and HelloReply Messages are not the first RCP Messages, but they are the first iSER Messages. 2.2 The following should be placed/updated in Acronym Section HCA Host Channel Adapter IB InfiniBand IPoIB IP over InfiniBand LLP Lower Layer Protocol SCTP Stream Control Transmission Protocol TCA Target Channel Adapter 2.3 Connection Establishment, Login, and Transition to iSER The discussion of connection establishment and the use of a messaging protocol for exchanging Login Request and Login Response Messages should be discussed for SCTP and IB, along with the transitioning of an IB connection to iSER mode, the suggested detail changes can be found (along with others) in the Appendix of this document. 2.4 Security considerations The discussion of Security should specify that all non IP protocols will define their own requirements for IPsec. However the iSCSI requirements for IPsec are still required wherever an iSER Message enters an IP environment from a non IP one (such as IB). Further the iSCSI/iSER requirement for IPsec on IP based protocols such as TCP and SCTP will continue to require IPsec as a must implement, but optional to use feature. The suggested changes can be found (along with others) in the Appendix of this document. J.Hufferd et. al. Expires June 2005 9 Internet-Draft Generalization of the iSER Specification December 2004 2.5 Adjustments to Appendix Rename the current appendix “Appendix A” Modify the term “iWARP” to “iWARP/TCP” in every section heading in this Appendix. 2.6 Add Appendix B An appendix (Appendix B) should be considered for the iSER draft to explain how an InfiniBand RC connection can be used to carry iSER protocols. The content of any such appendix should be similar in nature to that presented in section 3 of this document. J.Hufferd et. al. Expires June 2005 10 Internet-Draft Generalization of the iSER Specification December 2004 3 Architectural discussion of iSER over InfiniBand The following is an explanation of how an InfiniBand network (with Gateways) would be structured. It is intended to provide an insight on how iSER is used in an InfiniBand environment and be generally informational. This information is NOT being proposed as text for the main body of the iSER document. It may be considered for a NON Normative informational Appendix within the iSER document, but its primary purpose is to help put the idea of an iSER operating on InfiniBand into perspective for the readers of this document. 3.1 The Host side of the InfiniBand iSCSI & iSER connections The following figure (2) defines the topologies in which iSCSI and iSER will be able to operate on an InfiniBand Network. J.Hufferd et. al. Expires June 2005 11 Internet-Draft Generalization of the iSER Specification December 2004 +---------+ +---------+ +---------+ +---------+ +--- -----+ | Host | | Host | | Host | | Host | | Host | | | | | | | | | | | +---+-+---+ +---+-+---+ +---+-+---+ +---+-+---+ +---+-+---+ |HCA| |HCA| |HCA| |HCA| |HCA| |HCA| |HCA| |HCA| |HCA| |HCA| +-v+ +-v-+ +-v-+ +-v-+ +-v-+ +-v-+ +-v-+ +-v-+ +-v-+ +-v-+ | +------|-----+-----|-----+-----|-----+-----|-----+---> To IB IB| IB | IB | IB | IB | SubNet2 SWTCH +-v-----------v-----------v-----------v-----------v---------+ | InfiniBand Switch for Subnet1 | +---+----+-----+----+-----+-----+------+----+-----------v---+ |TCA | |TCA | | TCA | | TCA| | +----+ +----+ +-----+ +----+ | IB / IB \ / IB \ / IB \ / \ +--+--v--+--+ | iSER | | iSER | | iSER | | IPoIB | | | TCA | | | Gateway| | Gateway| | Gateway | | Gateway| | +-----+ | | to | | to | | to | | to | | Storage | | iSCSI | |iSER/TCP| |iSER/SCTP| | IP | | Controller| | TCP | | iWARP | | iWARP | |Ethernet| +-----+-----+ +---v----| +---v----| +----v----+ +----v---+ | EN | EN | EN | EN +----------+-----------+------------+----> to IP based storage Ethernet links that carry iSCSI or iWARP Figure 1 iSCSI, and iSER on IB J.Hufferd et. al. Expires June 2005 12 Internet-Draft Generalization of the iSER Specification December 2004 In Figure 2, the Host systems are connected via the InfiniBand Host Channel Adapters (HCAs) to the InfiniBand links. With the use of IB switch(es), the InfiniBand links connect the HCA to InfiniBand Target Channel Adapters (TCAs) located in gateways or Storage Controllers. An iSER-capable IB-IP Gateway converts the iSER Messages encapsulated in IB protocols to either standard iSCSI, or iSER Messages for iWARP. An IPoIB Gateway converts InfiniBand IPoIB protocols to IP protocols, and in the iSCSI case, permits iSCSI to be operated on an IB Network between the Hosts and the IPoIB Gateway. 3.2 The Storage side of iSCSI & iSER mixed network environment Figure 3 shows a storage controller that has four different portal groups: one supporting only iSCSI (TPG-4), one supporting iSER/iWARP/TCP or iSCSI (TPG-2), one supporting iSER/iWARP/SCTP (TPG-3), and one supporting iSER/IB (TPG-1). J.Hufferd et. al. Expires June 2005 13 Internet-Draft Generalization of the iSER Specification December 2004 | | | | | | | | +--+--v--+-------+--v--+--------+--v--+-------+--v--+--+ | | IB | |iWARP| |iWARP| | EN | | | | RNIC| | TCP | |SCTP | | NIC | | | |(TCA)| | RNIC| |RNIC | | | | | +-----| +-----+ +-----+ +-----+ | | TPG-1 TPG-2 TPG-3 TPG-4 | | 9.1.3.3 9.1.2.4 9.1.2.5 9.1.2.6 | | | | Storage Controller | | | +------------------------------------------------------+ Figure 2 Storage Controller with iSCSI, iWARP/TCP, iWARP/SCTP, and IB connections The normal iSCSI portal group advertising processes (for SLP, iSNS, or SendTargets commands) are available to a Storage Controller. 3.3 Discovery processes for an InfiniBand Host An InfiniBand Host system can gather portal group IP address from SLP, iSNS, or the SendTargets discovery processes by using TCP/IP via IPoIB. After obtaining one or more remote portal IP addresses, the Initiator uses the standard IP mechanisms to resolve the IP address to a local outgoing interface and the destination hardware address (Ethernet MAC or IB GID of the target or a gateway leading to the target). If the resolved interface is an IPoIB network interface, then the target portal can be reached through an InfiniBand fabric. In this case the Initiator can establish an iSCSI/TCP or iSCSI/iSER session with the Target over that InfiniBand interface, using the Hardware Address (InfiniBand GID) obtained through the standard Address Resolution (ARP) processes. If more than one IP address are obtained through the discovery process, the Initiator should select a Target IP address that is on J.Hufferd et. al. Expires June 2005 14 Internet-Draft Generalization of the iSER Specification December 2004 the same IP subnet as the Initiator if one exists. This will avoid a potential overhead of going through a gateway when a direct path exist. In addition a user can configure manual static IP route entries if a particular path to the target is preferred. 3.4 IBTA Connection specifications It is expected that the InfiniBand Trade Association (IBTA) only needs to define: . Means for permitting a Host to establish an iSCSI/iSER connection with a peer InfiniBand end-node, and indicating when that end- node does not support iSER, so the Host would be able to fall back to iSCSI/TCP over IPoIB. . Means for permitting the Host to establish connections with IB iSER connections on Storage Controllers or IB iSER connected Gateways in preference to IPoIB connected Gateways/Bridges or connections to Target Storage Controllers that also accept iSCSI via IPoIB. The IBTA may also decide to specify how iSER may be implemented on connections dealing features that are optional for IB implementations. (See section 5.3) Everything else that is needed is defined in the generalizations made to the iSER specification in this document. The following are implementation issues and need not be standardized by IBTA: . How implementations determine which iSCSI/iSER portal group to use. (Basing the decision on new information that may be placed in the iSCSI discovery information is not required, and simple trial selection is acceptable.) . How implementations determine how to best handle the concept of multiple connections per session as it deals with multiple IB Addresses:Ports per Portal Group. J.Hufferd et. al. Expires June 2005 15 Internet-Draft Generalization of the iSER Specification December 2004 4 Preferred changes to iSCSI Discovery Data The iSER/iWARP/SCTP initiator has similar problems in locating the appropriate Portal Group similar to an InfiniBand initiator. That is, it must check each discovered Portal Group IP address to see if it will accept an iSER/iWARP/SCTP connection. Even though it is possible for the InfiniBand Host initiator or an iSER/iWARP/SCTP initiator to pick an appropriate connection, the approach may not be optimal, and it takes up needless resources and time by attempting to connect to each portal Group. Therefore, it is useful to have a connection type associated with the Portal Group Tag that will permit the most appropriate connections to be made without needless connection tries and failures. For these reasons we will also submit updates to the [iSCSI] (SendTargets), [SLP], and [iSNS] Drafts/RFCs that will add an additional tag following the Portal Group Number. The types will be documented by IANA, with the following definitions. Portal Group Type IANA Portal Group Value iSCSI 0, or blank iSER/iWARP/TCP 1 iSER/iWARP/SCTP 2 iSER/iWARP/IB 3 If other connection types are defined later they will also need to obtain an IANA Portal Group type value. The syntax of the parameter will be IP Address:Port[, PG#[, Type#]] If there is no Type# specified it will be assumed to be an iSCSI capable Portal, or one that must be tested via the connection and Login Process. J.Hufferd et. al. Expires June 2005 16 Internet-Draft Generalization of the iSER Specification December 2004 5 The Appropriate Standardization Organizations for iSER/IB 5.1 The IETF ips Working Group Standardization iSER document will be updated to generalize the use of iSER for other RDMA-capable protocol layers such as iSER over IB or iSER over iWARP/SCTP. (See the suggested words in section 2 and the Appendix of this document.) There may also be an informational appendix to explain how the InfiniBand Path connection will be done and how the Login parameters will be handled in an InfiniBand environment. 5.2 The InfiniBand Trade Association (IBTA) Standardization The InfiniBand Trade Association (IBTA) will be asked to standardize the iSER Connection process that permits the selection of the best path from the Host. (See section 3.4 “IBTA Connection specifications” above.) 5.3 IBTA and optional InfiniBand Features Since the IBTA has made key capabilities optional that are currently required by iSER, such as ZBTO (a.k.a. ZBVA in InfiniBand) and SendInvSE, it is required that IBTA documents how it will support an iSER connection process (perhaps including the Hello/HelloReply messages) that permits the iSER implementation to understand the limitations of the remote peer. It is also required that IBTA documents how a version of iSER, which understands the limitations of its peer, should operate in that environment. It should be noted that this places a requirement on the iSER/IB to iSER/iWARP (or iSCSI) Gateway to support the ZBTOs (ZBVAs) on the IP side of the Gateway. This means that it must keep the Virtual Addresses associated with every outstanding STag, so that it can convert the VA to and from the ZBTO required by the iSER/iWARP peer. 5.4 ServiceID and IP Port number The IBTA will be asked to standardize the iSER ServiceID, and how the ServiceID can be added to the IP port number during the connection process. J.Hufferd et. al. Expires June 2005 17 Internet-Draft Generalization of the iSER Specification December 2004 6 IANA Considerations The following additional items will require registration with IANA before the resulting draft can be approved to become an RFC: None are known at this time. J.Hufferd et. al. Expires June 2005 18 Internet-Draft Generalization of the iSER Specification December 2004 7 References 7.1 Informative References [DA] M. Chadalapaka et al., "Datamover Architecture for iSCSI", IETF Internet-draft, draft-ietf-ips-da-00.txt (work in progress), September 2004 [DDP] H. Shah et al., "Direct Data Placement over Reliable Transports", IETF Internet-draft draft-ietf-rddp-ddp-03.txt (work in progress), August 2004 [IPSEC] S. Kent et al., "Security Architecture for the Internet Protocol", RFC 2401, November 1998 [iSCSI] J. Satran et al., "iSCSI", RFC 3720, April 2004 [iSER] M. Ko et. al., ”iSCSI Extensions for RDMA Specification”, IETF Internet-draft draft-ko-iwarp-iser-00.txt [iSNS] Josh Tseng et. al., Internet Storage Name Service (iSNS), IETF Internet-draft, draft-ietf-ips-isns-22.txt [MPA] P. Culley et al., "Marker PDU Aligned Framing for TCP Specification", IETF Internet-draft draft-ietf-rddp-mpa-01.txt (work in progress), July 2004 [RDMAP] R. Recio et al., "An RDMA Protocol Specification", IETF Internet-draft draft-ietf-rddp-rdmap-01.txt (work in progress), October 2003 [SAM2] T10/1157D, SCSI Architecture Model - 2 (SAM-2) [SLP] M. Bakke et. al., Finding iSCSI Targets and Name Servers Using SLP, IETF Internet-draft, draft-ietf-ips-iscsi-slp-09.txt [TCP] Postel, J., "Transmission Control Protocol", STD 7, RFC 793, September 1981 [VERBS] J. Hilland et al., "RDMA Protocol Verbs Specification", RDMAC Consortium Draft Specification draft-hilland-iwarp-verbs- v1.0-RDMAC, April 2003 J.Hufferd et. al. Expires June 2005 19 Internet-Draft Generalization of the iSER Specification December 2004 8 Appendix 8.1 Additional detailed [iSER] document modification The new terms introduced in the subsections under section 2.1 will replace the existing ones in the [iSER] document where appropriate. In addition, the following changes and clarifications are needed. 8.1.1 Adjustment to Section 2.1 Motivation The fourth paragraph should be adjusted such as: Supporting direct data placement is the main function of an RDMA- capable protocol. An RDMA-Capable Controller (such as a NIC enhanced with the RDMAP/DDP functions layered on top of MPA/TCP or SCTP, or an Infiniband Host Channel Adapter or Target Channel Adapter) can be used by any application that has been extended to support RDMA. 8.1.2 Adjustment to Section 2.2 Architectural Goals The following are changes that should be considered for the numbered paragraphs: 1. Provide an RDMA data transfer model for iSCSI that enables direct in order or out of order data placement of SCSI data into pre- allocated SCSI buffers while maintaining in order data delivery. 5. Allow initiator and target implementations that utilize generic RDMA-capable controllers such as RNICs and implement iSCSI and iSER in software (not require iSCSI or iSER specific assists in the RDMA- capable protocol or RDMA-capable controller). 6. Require full and only generic RDMA-capable protocol functionality at both the initiator and the target. 8.1.3 Adjustment to Section 2.3 Protocol Overview The following is a suggestion for changes that should be considered for the paragraph number 6: 6. The RDMA-capable protocol guarantees data integrity. (For example, for TCP, iWARP includes a CRC-enhanced framing layer (called MPA) on top of TCP; for SCTP, the CRCs are included in the SCTP protocol; and for Infiniband, the CRCs are included in the J.Hufferd et. al. Expires June 2005 20 Internet-Draft Generalization of the iSER Specification December 2004 Reliable Connection mode.) For this reason, iSCSI header and data digests are negotiated to "None" for iSCSI/iSER sessions. 8.1.4 Adjustment to Section 2.4 RDMA services and iSER Additional generalization wordage is needed. The following is a change that should be considered for the first paragraph: iSER is designed to work with software and/or hardware protocol stacks providing the protocol services defined in RDMA-capable protocol documents such as [RDMAP], [IB], etc. 8.1.5 Adjustment to Section 2.7 iSCSI/iSER Layering The layering wordage needs additional generalization and the example needs to be made more general. The following is suggested wordage and a suggested replacement for figure 1: “iSCSI Extensions for RDMA (iSER) is layered between the iSCSI layer and the RDMA-capable protocol layer. Figure 1 shows an example of the relationship between SCSI, iSCSI, iSER, RDMA-capable protocol layers such as iWARP and [IB], and the underlying transports such as TCP, SCTP, or [IB].” J.Hufferd et. al. Expires June 2005 21 Internet-Draft Generalization of the iSER Specification December 2004 +------------------------------------------------+ | SCSI | +------------------------------------------------+ | iSCSI | DI ------> +------------------------------------------------+ | iSER | +------------------------+------------+----------+ | RDMAP | | | +------------------------+ | | | DDP | | Other | +--------------+---------+ InfiniBand | Possible | | MPA | | (RC) | RCP | +--------------+ SCTP | | | | TCP | | | | +--------------+---------+------------+----------+ | IP | [IB] (LLP) |Other LLP | +------------------------+------------+----------+ Figure 3 Generic example of iSCSI/iSER Layering in Full Feature Mode 8.1.6 Generalization of Other iSER Sections The following are changes that should be considered for section 4.1 Interactions with the iWARP Layer (which should change to Interaction with the RDMA-Capable Protocol) . In the next to last * paragraph consider replacing that paragraph with the following: o “* For LLPs operating in the stream mode, such as TCP, the RDMA-capable protocol implementation supports the enabling J.Hufferd et. al. Expires June 2005 22 Internet-Draft Generalization of the iSER Specification December 2004 of the RDMA mode after Connection establishment and the exchange of Login parameters in stream mode. For LLPs that have message delivery capability, such as SCTP or [IB], the iSCSI Layer may use that messaging capability immediately after connection establishment before enabling iSER-assisted mode. The native messaging facility of such an LLP may be used for the Login parameter exchanges.” The following are changes that should be considered for section 4.2 Interactions with the Transport Layer . “The iSER Layer does not interface with the transport layer (e.g., TCP, SCTP or [IB]) directly. During Connection setup, the iSCSI Layer is responsible for setting up the Connection. If the login is successful, the iSCSI Layer invokes the Enable_Datamover Operational Primitive to request the iSER Layer to transition to the iSER- assisted mode for that iSCSI connection. See section 5.1 on iSCSI/iSER Connection setup. After transitioning to iSER-assisted mode, the RDMA-capable protocol layer and the underlying LLP are responsible for maintaining the Connection and reporting to the iSER Layer of any Connection failures. ” 8.1.6.1 Adjustments to 5.1 iSCSI/iSER Connection Setup The following is a new Section 5.1 paragraph (insert after paragraph 1) that should be considered . “When a messaging capability is supported by the underlying transport (e.g. SCTP or InfiniBand), the messaging capability may be used by both the initiator and the target to exchange the iSCSI Login Request and Login Response PDUs. The method for establishing the actual connection is protocol specific and outside the scope of this specification.” The following are changes that should be considered for the last paragraph in 5.1 . “When the RDMAExtensions key is negotiated to "Yes", the HeaderDigest and the DataDigest keys MUST be negotiated to "None" on all iSCSI/iSER connections participating in that iSCSI session. This is because, for an iSCSI/iSER connection, the RDMA-capable protocol provides a CRC based error detection for all iSER Messages.” J.Hufferd et. al. Expires June 2005 23 Internet-Draft Generalization of the iSER Specification December 2004 8.1.6.2 Adjustment to Section 5.1.1 Initiator Behavior The following are changes that should be considered for the 11th paragraph of section 5.1.1 Initiator Behavior. . “3. If necessary, the iSER Layer MUST enable the RDMA-capable protocol and transition the connection to iSER-assisted mode. (Some RDMA-capable protocol, such as [IB], does not require special enablement for RDMA support.)” 8.1.6.3 Adjustment to Section 5.1.2 Target Behavior In section 5.1.2 all the references to “iWARP” should be replaced with “the RDMA-capable protocol”. Also in Section 5.1.2, the paragraph numbered as “4.” The following should be considered as a replacement . “4. After sending the final SCSI Login Response PDU, the iSER Layer MUST enable the RDMA-capable protocol if necessary and transition the connection to iSER-assisted mode. (Some RDMA- capable protocol, such as [IB], does not require special enablement for RDMA support.) Note that for TCP, the final SCSI Login Response PDU is sent in byte stream mode.” And the last paragraph in Section 5.1.2 should consider the following for a replacement . “Note: In the above sequence, the operations as described in bullets 3 and 4 must be performed atomically for iWARP connections. Failure to do this may result in race conditions.” The following are changes that should be considered for the second paragraph of 5.1.3 iSER Hello Exchange. (It tolerates connections that might already be in RDMA mode when the Hello Exchanges were sent.) . “In response to the iSER Hello Message, the iSER Layer at the target MUST return the iSER HelloReply Message as the first RCP Message sent by the target after the connection transitions into iSER-assisted mode. The iSER HelloReply Message is used by the iSER Layer at the target to declare iSER parameters to the initiator. See section 9.4 on iSER Header Format for iSER HelloReply Message.” J.Hufferd et. al. Expires June 2005 24 Internet-Draft Generalization of the iSER Specification December 2004 8.1.6.4 Adjustments to Section 11 Security Considerations The following paragraphs should be considered as the replacement paragraphs for Section 11 Security Considerations. “When iSER is layered on top of an RDMA-capable protocol layer and provides the RMDA extension to the iSCSI protocol, the security considerations of iSER are similar to that of the underlying RDMA- capable protocol layer. For iWARP, this is described in [RDMAP]. If iSER is layered on top of the iWARP protocol Stack (TCP or SCTP), all the Security protocol mechanisms described in [iSCSI] may be deployed for an iSCSI/iSER connection. If the IPsec mechanism is used, then it MUST be established before the connection transitions to iSER-assisted mode. If iSER is layered on top of a non-IP based RDMA-capable protocol layer, the non-IPsec security protocol mechanisms described in [iSCSI] MAY be deployed for an iSCSI/iSER connection. The authorized standards organization for that network’s protocols (such as InfiniBand Trade Associations) is responsible for determining the capability and requirement of that environment on the implementation of IPsec. If iSER is layered on top of a non-IP protocol, the IPsec protocols and features, as specified in [iSCSI] MUST be implemented at any point where the iSER protocol enters the IP network (e.g., via gateways). 8.1.7 Adjustments to 13.2 Informational References Add the following references: [SCTP] R. Stewart, et. al., “Stream Control Transmission Protocol”, RFC 2960 October 2000 (Updated by RFC3309) [IB] InfiniBand Architecture Specification Volume 1 Release 1.2, October 2004 J.Hufferd et. al. Expires June 2005 25 Internet-Draft Generalization of the iSER Specification December 2004 [IPoIB] H.K. Chu et al, "Transmission of IP over InfiniBand", IETF Internet-draft draft-ietf-ipoib-ip-over-infiniband-07.txt (work in progress), August, 2004 J.Hufferd et. al. Expires June 2005 26 Internet-Draft Generalization of the iSER Specification December 2004 9 Author’s Address John Hufferd IBM Corp. 5600 Cottle Rd. San Jose, CA 95120, USA Phone: +1-408-256-0403 Email: hufferd@us.ibm.com Mike Ko IBM Corp. 650 Harry Rd. San Jose, CA 95120, USA Phone: +1-408-927-2085 Email: mako@us.ibm.com Yaron Haviv Voltaire Ltd. 9 Hamanofim St. Herzelia 46725, Israel Phone: +972.9.9717655 Email: yaronh@voltaire.com J.Hufferd et. al. Expires June 2005 27 Internet-Draft Generalization of the iSER Specification December 2004 10 Acknowledgments David Black EMC Corporation 176 South St. Hopkinton, MA 01748, USA Phone: +1-508-293-7953 Email: black_david@emc.com Mallikarjun Chadalapaka Hewlett-Packard Company 8000 Foothills Blvd. Roseville, CA 95747-5668, USA Phone: +1-916-785-5621 Email: cbm@rose.hp.com Mike Krause Hewlett-Packard Company 43LN 19410 Homestead Road Cupertino, CA 95014, USA Phone: +1-408-447-3191 Email: krause@cup.hp.com Alex Nezhinsky Voltaire Ltd. 9 Hamanofim St. Herzelia 46725, Israel Phone: +972.9.9717637 Email: alexn@voltaire.com Renato J. Recio IBM Corp. 11501 Burnett Road Austin, TX 78758, USA Phone: +1-512-838-3685 Email: recio@us.ibm.com Tom Talpey Network Appliance 375 Totten Pond Road Waltham, MA 02451, USA Phone: +1-781-768-5329 EMail: thomas.talpey@netapp.com J.Hufferd et. al. Expires June 2005 28 Internet-Draft Generalization of the iSER Specification December 2004 11 Full Copyright Statement Copyright (C) The Internet Society (year). This document is subject to the rights, licenses and restrictions contained in BCP 78, and except as set forth therein, the authors retain all their rights." This document and the information contained herein are provided on an "AS IS" basis and THE CONTRIBUTOR, THE ORGANIZATION HE/SHE REPRESENTS OR IS SPONSORED BY (IF ANY), THE INTERNET SOCIETY AND THE INTERNET ENGINEERING TASK FORCE DISCLAIM ALL WARRANTIES, EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO ANY WARRANTY THAT THE USE OF THE INFORMATION HEREIN WILL NOT INFRINGE ANY RIGHTS OR ANY IMPLIED WARRANTIES OF MERCHANTABILITY OR FITNESS FOR A PARTICULAR PURPOSE. The IETF takes no position regarding the validity or scope of any Intellectual Property Rights or other rights that might be claimed to pertain to the implementation or use of the technology described in this document or the extent to which any license under such rights might or might not be available; nor does it represent that it has made any independent effort to identify any such rights. Information on the procedures with respect to rights in RFC documents can be found in BCP 78 and BCP 79. Copies of IPR disclosures made to the IETF Secretariat and any assurances of licenses to be made available, or the result of an attempt made to obtain a general license or permission for the use of such proprietary rights by implementers or users of this specification can be obtained from the IETF on-line IPR repository at http://www.ietf.org/ipr. The IETF invites any interested party to bring to its attention any copyrights, patents or patent applications, or other proprietary rights that may cover technology that may be required to implement this standard. Please address the information to the IETF at ietf- ipr@ietf.org. J.Hufferd et. al. Expires June 2005 29