STORM A. Kanevsky, Ed. Internet-Draft Dell Inc. Updates: 5043, 5044 (if approved) C. Bestler, Ed. Intended status: Standards Track Nexenta Systems Expires: March 13, 2012 R. Sharp Intel S. Wise Open Grid Computing September 10, 2011 Enhanced RDMA Connection Establishment draft-ietf-storm-mpa-peer-connect-07 Abstract This document updates RFC5043 and RFC5044 by extending MPA negotiation for RDMA connection establishment. The first enhancement extends RFC5044, enabling peer-to-peer connection establishment over MPA/TCP. The second enhancement extends both RFC5043 and RFC5044, by providing an option for standardized exchange of RDMA-layer connection configuration. Status of this Memo This Internet-Draft is submitted in full conformance with the provisions of BCP 78 and BCP 79. Internet-Drafts are working documents of the Internet Engineering Task Force (IETF). Note that other groups may also distribute working documents as Internet-Drafts. The list of current Internet- Drafts is at http://datatracker.ietf.org/drafts/current/. Internet-Drafts are draft documents valid for a maximum of six months and may be updated, replaced, or obsoleted by other documents at any time. It is inappropriate to use Internet-Drafts as reference material or to cite them other than as "work in progress." This Internet-Draft will expire on March 13, 2012. Copyright Notice Copyright (c) 2011 IETF Trust and the persons identified as the document authors. All rights reserved. This document is subject to BCP 78 and the IETF Trust's Legal Provisions Relating to IETF Documents (http://trustee.ietf.org/license-info) in effect on the date of Kanevsky, et al. Expires March 13, 2012 [Page 1] Internet-Draft Enhanced RDMA Connection Establishment September 2011 publication of this document. Please review these documents carefully, as they describe your rights and restrictions with respect to this document. Code Components extracted from this document must include Simplified BSD License text as described in Section 4.e of the Trust Legal Provisions and are provided without warranty as described in the Simplified BSD License. Table of Contents 1. Introduction . . . . . . . . . . . . . . . . . . . . . . . . . 3 1.1. Summary of changes from RFC 5044 . . . . . . . . . . . . . 3 1.2. Summary of changes from RFC 5043 . . . . . . . . . . . . . 4 2. Requirements Language . . . . . . . . . . . . . . . . . . . . 4 3. Definitions . . . . . . . . . . . . . . . . . . . . . . . . . 4 4. Motivations . . . . . . . . . . . . . . . . . . . . . . . . . 6 4.1. Standardization of RDMA Read Parameter Configuration . . . 6 4.2. Enabling MPA Mode . . . . . . . . . . . . . . . . . . . . 8 4.3. Lack of Explicit RTR in MPA Request/Reply Exchange . . . . 8 4.4. Limitations on ULP Workaround . . . . . . . . . . . . . . 9 4.4.1. Transport Neutral APIs . . . . . . . . . . . . . . . . 10 4.4.2. Work/Completion Queue Accounting . . . . . . . . . . . 10 4.4.3. Host-based Implementation of MPA Fencing . . . . . . . 11 5. Enhanced MPA Connection Establishment . . . . . . . . . . . . 11 6. Enhanced MPA Request/Reply Frames . . . . . . . . . . . . . . 12 7. Enhanced SCTP Session Control Chunks . . . . . . . . . . . . . 13 8. MPA Error Reporting . . . . . . . . . . . . . . . . . . . . . 15 9. Enhanced RDMA Connection Establishment Data . . . . . . . . . 15 9.1. IRD and ORD Negotiation . . . . . . . . . . . . . . . . . 16 9.2. Peer-to-Peer Connection Negotiation . . . . . . . . . . . 18 9.3. Enhanced Connection Negotiation Flow . . . . . . . . . . . 19 10. Interoperability . . . . . . . . . . . . . . . . . . . . . . . 20 11. IANA Considerations . . . . . . . . . . . . . . . . . . . . . 20 12. Security Considerations . . . . . . . . . . . . . . . . . . . 20 13. Acknowledgements . . . . . . . . . . . . . . . . . . . . . . . 20 14. References . . . . . . . . . . . . . . . . . . . . . . . . . . 21 14.1. Normative References . . . . . . . . . . . . . . . . . . . 21 14.2. Informative References . . . . . . . . . . . . . . . . . . 21 Authors' Addresses . . . . . . . . . . . . . . . . . . . . . . . . 22 Kanevsky, et al. Expires March 13, 2012 [Page 2] Internet-Draft Enhanced RDMA Connection Establishment September 2011 1. Introduction When used over Transmission Control Protocol TCP, the current Remote Direct Data Placement (RDDP) suite of protocols relies on Marker protocol Data Unit PDU Alignment (MPA) [RFC5044] protocol for both connection establishment and for markers for TCP layering. A typical model for establishing an RDMA connection has the following steps: o The passive side Upper Layer Protocol (ULP) listens for connection requests. o The active side ULP submits a connection request using an RDMA endpoint, the desired destination and the parameters to be used for the connection. Those parameters include both RDMA layer characteristics, such as the number of simultaneous RDMA Read Requests to be allowed and application specific data. o The passive side ULP receives a connection request, which includes the identity of the active side and the requested connection characteristics. The passive side ULP uses this information to decide whether to accept the connection, and if it is to be accepted, how to create and/or configure the local RDMA endpoint. o If accepting, the passive side ULP submits its acceptance of the connection request. This local accept operation includes the RDMA endpoint to be used and the connection characteristics (both the RDMA configuration and any application specific private data to be returned). o The active side receives confirmation that the connection has been accepted, what the configured connection characteristics are, and any application supplied private data. Currently, MPA only supports a client-server model for connection establishment, forcing peer-to-peer applications to interact as though they had a client/server relationship. In addition negotiation of some of Remote Direct Memory Access Protocol (RDMAP) [RFC5040] specific parameters are left to ULP negotiation. Providing an optional ULP-independent format for exchanging these parameters would be of benefit to transport neutral Remote Direct Memory Access (RDMA) applications. 1.1. Summary of changes from RFC 5044 This draft enhances [RFC5044] MPA connection setup protocol. First, it adds exchange and negotiation of the parameters necessary to Kanevsky, et al. Expires March 13, 2012 [Page 3] Internet-Draft Enhanced RDMA Connection Establishment September 2011 support RDMA Read Requests. Second, it adds a Ready to Receive (RTR) frame from the Initiator to the Responder as the last message of connection establishment and adds negotiation of an RTR message frame type into MPA request/reply frames. 1.2. Summary of changes from RFC 5043 This draft enhances [RFC5043] by adding new Enhanced Session Control Chunks that extends the currently defined Chunks with the addition of IRD and ORD negotiation. 2. Requirements Language The key words "MUST", "MUST NOT", "REQUIRED", "SHALL", "SHALL NOT", "SHOULD", "SHOULD NOT", "RECOMMENDED", "MAY", and "OPTIONAL" in this document are to be interpreted as described in [RFC2119]. 3. Definitions FULPDU: Framed Upper Layer Protocol PDU. See FPDU of [RFC5044]. CRC: Cyclic Redundancy Check. Completion Queue (CQ): A consumer accessible queue where the RDMA device reports completions of Work Requests. A Consumer is able to reap completions from a CQ without requiring per transaction support from the kernel or other privileged entity. See [RDMAC]. Completion Queue Entry (CQE): Transport and device specific representation of a Work Completion. A Completion Queue holds CQEs. See [RDMAC]. Inbound RDMA Read Request Queue (IRRQ): A queue that is associated with an RDMA Connection that tracks active incoming simultaneous RDMA Read Request Messages. See [RDMAC]. Inbound RDMA Read Queue Depth (IRD): The maximum number of incoming simultaneous RDMA Read Request Messages an RDMA connection can handle. See [RDMAC]. IRD: See Inbound RDMA Read Queue Depth. MPA Fencing: MPA Responder Connection Establishment logic that ensures that no ULP messages will be transferred until the Initiator first message has been received. Kanevsky, et al. Expires March 13, 2012 [Page 4] Internet-Draft Enhanced RDMA Connection Establishment September 2011 ORD: See Outbound RDMA Read Queue Depth. Outbound RDMA Read Queue Depth (ORD): The maximum number of simultaneous RDMA Read Requests that can be issued for the RDMA connection. This should be less than or equal to the peer's IRD. See [RDMAC]. Queue Pair (QP): The traditional name for a local Endpoint in a [VIA] derived local interface. A Queue Pair is the set of Work Queues associated exclusively with a single Endpoint. The Send Queue (SQ), Receive Queue (RQ) and Inbound RDMA Read Queue (IRQ) are considered to be part of the Queue Pair. The potentially shared Completion Queue (CQ) and Shared Receive Queue (SRQ) are not. See [RDMAC]. Ready to Receive (RTR): RTR is the last connection establishment message sent from the Initiator to the Responder indicating that the Initiator is ready to receive messages and that connection establishment is completed. See [IBTA]. Shared Receive Queue(SRQ): A shared pool of Receive Work Requests posted by the Consumer that can be allocated by multiple RDMA endpoints (Queue Pair). See [RDMAC]. Tagged (DDP) Message: - A DDP Message that targets a Tagged Buffer that is explicitly Advertised to the Remote Peer through exchange of an STag (memory handle), offset in the memory region identified by STag, and length [RFC5040]. Untagged (DDP) Message: - A DDP Message that targets an Untagged Buffer associated with a queue specified by Queue Number (QN). [RFC5040]. Work Queue: An element of a [VIA] derived local interface that allows user-space applications to submit Work Requests directly to network hardware. Specific Work Queues include the Send Queue (SQ) for transmit requests, Receive Queue (RQ) for receive requests specific to a single Endpoint and Shared Receive Queues (SRQs) for receive requests that can be allocated by one or more Endpoints. See [RDMAC]. Work Queue Element (WQE): Transport and device specific representation of a Work Request. See [RDMAC]. Work Request: An elementary object used by Consumers to enqueue a requested operation (WQEs) onto a Work Queue. See [RDMAC]. Kanevsky, et al. Expires March 13, 2012 [Page 5] Internet-Draft Enhanced RDMA Connection Establishment September 2011 4. Motivations The goal of this draft is twofold. One is to extend support from the current client-server model for RDMA connection setup to a peer-to- peer model. The second is to add negotiation of RDMA Read queue size for both sides of an RDMA connection. 4.1. Standardization of RDMA Read Parameter Configuration Most RDMA applications are developed using a transport neutral API to access RDMA services based on a "queue pair" paradigm as originally defined by the Virtual Interface Architecture [VIA], refined by the Direct Access Programming Library [DAPL] and most commonly deployed with the OpenFabrics API [OFA]. These transport neutral APIs seek to provide a common set of RDMA services whether the underlying transport is, for example, RDDP over MPA, RDDP over SCTP or InfiniBand. The common model for establishing an RDMA connection has the following steps: o The passive side ULP listens for connection requests. o The active side ULP submits a connection request using an RDMA endpoint ("queue pair"), the desired destination and the parameters to be used for the connection. Those parameters include both RDMA layer characteristics, such as the RDMA Read credits to be allowed and application specific data (typically referred to as "private data"). o The passive side ULP receives a connection request, which includes the identity of the active side and the requested connection characteristics. The passive side ULP uses this information to decide whether to accept the connection, and if it is to be accepted, how to create and/or configure the RDMA endpoint. o If accepting, the passive side ULP submits its acceptance of the connection request. This local accept operation includes the RDMA endpoint to be used and the connection characteristics (both the RDMA configuration and any application specific private data to be returned). o The active side receives confirmation that the connection has been accepted, what the configured connection characteristics are, and any application supplied private data. As currently defined, DDP connection establishment requires the ULP Kanevsky, et al. Expires March 13, 2012 [Page 6] Internet-Draft Enhanced RDMA Connection Establishment September 2011 to encode the RDMA configuration in the application specific private data. This results in undesirable duplication of logic to cover both InfiniBand and RDDP, and to specify the extraction of the RDMA characteristics from the ULP for each specific Upper Layer Protocol. Both RDDP and InfiniBand support an initial private data exchange, therefore a standard definition of the RDMA characteristics within the private data section would enable common connection establishment APIs to format the RDMA characteristics based on the same API information used when establishing either protocol to form the connection. The application would then only have to indicate that it was using this standard format to enable common connection establishment procedures to apply common code to properly parse these fields and configure the RDMA endpoints accordingly. Exchange of parameters necessary to perform RDMA Read operations is a common usage of the initial private data exchange. One of the RDMA operations that is defined in [RDMAC] is an RDMA Read. RDMA Read operations are performed using an untagged message sent from a Queue Pair (QP) on the local endpoint to a QP on the remote endpoint targeting the Inbound RDMA Read Request Queue (QN=1 or IRRQ) associated with the connection. RDMA Read responses transfer data associated with each RDMA Read Request from the remote endpoint to the local endpoint using tagged messages. An inbound RDMA Read Request remains on the IRRQ from the time that it is received until the time that the last tagged message associated with the RDMA request is acknowledged. The IRRQ is associated with a QP but is not a Work Queue. Instead the IRRQ is a standalone queue that is used to manage RDMA read requests associated with a QP. See [RDMAC] section 6 for more information regarding QPs and IRRQ. One of the characteristics that must be configured for a QP is the size of the IRRQ. This parameter is called the Inbound RDMA Read Queue Depth (IRD). Another characteristic of a QP that must be configured a local limit on the number of simultaneous outbound RDMA Read Requests based on the size of the remote endpoint QP's IRRQ. This parameter is call the Outbound RDMA Read Queue Depth (ORD). ORD is used to limit the number of simultaneous RDMA read requests such that the local endpoint does not overrun the remote endpoint's IRRQ depth or IRD. Note that outbound RDMA Reads are submitted to a QP's Send Queue at the local peer, not to a separate outbound RDMA read request queue on the local peer. The local endpoint uses ORD strictly to limit simultaneous read requests so that IRRQ overruns do not occur at the remote endpoint. Determination of the values of the ORD and IRD are left to the ULP by the current RDDP suite of protocols and also by [RDMAC]. Since this negotiation of ORD and IRD is typical, it is desirable to provide a common mechanism described in this draft. Kanevsky, et al. Expires March 13, 2012 [Page 7] Internet-Draft Enhanced RDMA Connection Establishment September 2011 4.2. Enabling MPA Mode MPA defines encoding of DDP Segments in FULPDUs (Framed Upper Layer Protocol PDUs). Generation of FULPDUs requires the ability to periodically insert MPA Markers and to generate the MPA CRC-32c for each frame. Reception may require parsing/removing the markers after using them to identify MPA Frame boundaries, and validation of the MPA-CRC32c. A major design objective for MPA was to ensure that the resulting TCP stream would be a fully compliant TCP stream for any and all TCP- aware middle-boxes. The challenge is that while only some TCP payload streams are a valid stream of MPA FULPDUs, any sequence of bytes is a valid TCP payload stream. The determination that a given stream is in a specific MPA mode cannot be made at the MPA or TCP layer. Therefore enabling of MPA mode is handled by the Upper Level Protocol (ULP). The MPA protocol can be viewed as having two parts. o a specification of generation and reception of MPA FULPDUs. This is unchanged by enhanced RDMA connection establishment. o a pre-MPA exchange of messages to enable a specific MPA mode for the TCP connection. Enhanced RDMA connection establishment extends this protocol with two new features. In typical implementations, generation and reception of MPA FULPDUs is handled by hardware. The exchange of the MPA Request and Reply frames is then handled by host software. As will be explained, this implementation split impedes applications which are not compatible with the client-server assumptions in the current MPA Request/Reply exchange. 4.3. Lack of Explicit RTR in MPA Request/Reply Exchange The exchange of MPA Request and Reply messages to place a TCP connection in MPA mode is specified in [RFC5044]. This protocol provides many benefits to the design of MPA FULPDU hardware: o The ULP is responsible for specifying the exact MPA Mode (Markers enabled or disabled, CRC-32c enabled or suppressed) and the point in the TCP streams (inbound and outbound) where MPA frames will begin. o Before the first MPA frame is transmitted, all pre-MPA mode TCP payload will have been acknowledged by the peer. Therefore it is never necessary to generate a retransmission that mixes pre-MPA Kanevsky, et al. Expires March 13, 2012 [Page 8] Internet-Draft Enhanced RDMA Connection Establishment September 2011 and MPA payload. o Before MPA reception is enabled, all incoming pre-MPA mode TCP payload will have been acknowledged. Therefore the host will never receive a TCP segment that mixes pre-MPA and MPA payload. The limitation of the current MPA Request/Reply exchange is that it does not define a Ready to Receive (RTR) message that the active side would send, so that the passive side can know that the last non-MPA payload (the MPA Reply) had been received. Instead, the role of an RTR message is piggy-backed on the first MPA FULPDU sent by the active side. This is actually a valuable optimization for all applications that fit the classic client/server model. The client only initiates the connection when it has a request to send to the server, and the server has nothing to send until it has received and processed the client request. Even applications where the server sends some configuration data immediately can easily send the same information as application private data in the MPA Reply. So the currently defined exchange works for almost all applications. Many peer-to-peer applications, especially those involving cluster calculations (frequently using MPI [UsingMPI], or [RDS]), have no natural client or server roles ([PPMPI], [OpenMP]). Typically one member of the cluster is arbitrarily selected to initiate the connection when the distributed task is launched, while the other accepts it. At startup time, however, there is no way to predict which node will have the first message to actually send. Establishing the connections immediately, however, is valuable because it reduces latency once results are ready to transmit and it validates connectivity throughout the cluster. The lack of an explicit RTR message in the MPA Request/Reply exchange forces all applications to have a first message from the connection Initiator, whether this matches the application communication model or not. 4.4. Limitations on ULP Workaround The requirement that the RDMA connection Initiator sends the first message does not appear to be onerous on first examination. The natural question is why the application layer would not simply generate a dummy message when there was no other message to submit. There are three factors that make this workaround unsuitable for many peer-to-peer applications. Kanevsky, et al. Expires March 13, 2012 [Page 9] Internet-Draft Enhanced RDMA Connection Establishment September 2011 o Transport Neutral APIs. o Work/Completion Queue Accounting. o Host-based implementation of MPA Fencing. 4.4.1. Transport Neutral APIs Many of these applications access RDMA services using a transport neutral API such as [DAPL] or [OFA]. Only RDDP over TCP [RFC5044] has a first message requirement. Other RDMA transports, including RDDP over SCTP (see [RFC5043]) and InfiniBand (see [IBTA]), do not. Application or middleware communications can be expressed as transport neutral RDMA operations, allowing lower software layers to translate to transport and device specifics. Having a distinct extra message that is required only for one transport undermines the application's goal of being transport neutral. 4.4.2. Work/Completion Queue Accounting RDMA local APIs conventionally use work queues to submit requests (work queue elements or WQEs) and to asynchronously receive completions (in completion queues or CQs). Each work request can generate a completion queue entry (CQE). Completions for successful transmit work requests are frequently suppressed, but the completion queue capacity must account for the possibility that each will complete in error. A completion queue can receive completions from multiple work queues. Completion Queues are defined so as to allow hardware RDMA implementations to generate CQEs directly to a user-space mapped buffer. This enables a user-space RDMA consumer to reap completions without requiring kernel intervention. A hardware RDMA implementation cannot reasonably wait for an available slot in the completion queue. The queue must be sized such that an overflow will not occur. When an overflow does occur it is considered catastrophic and will typically require tearing down all RDMA connections using that CQ. This style of interface is very efficient, but places a burden on the application to properly size each Completion Queue to match the Work Queues that feed it. While the format of both WQEs and CQEs is transport and device dependent, a transport neutral API can deal with WQEs and CQEs as Kanevsky, et al. Expires March 13, 2012 [Page 10] Internet-Draft Enhanced RDMA Connection Establishment September 2011 abstract transport and device neutral objects. Therefore the number of WQEs and CQEs required for an application can be transport and device neutral. The capacity of the work queues and completion queues can be calculated in an abstract transport/device neutral fashion. If a dummy operation approach was used, it would require lower layers to know the usage model, and would disrupt the calculations by inserting a dummy "operation" Work Request and filtering out the matching completion. The lower layer does not know the usage model on which the queue sizes are built, nor does it know how frequently an insertion will be required. 4.4.3. Host-based Implementation of MPA Fencing Many hardware implementations of RDDP using MPA/TCP do not handle the MPA Request/Reply exchange in hardware, rather they are handled by the host processor in software. With such designs it is common for the MPA Fencing to be implemented in the user-space device-specific library (commonly referred to as a 'User Verbs' library or module). When the generation and reception of MPA FULPDUs is already dedicated to hardware, a Work Completion can only be generated by an untagged message since arrival of a message for tagged buffer does not necessarily generates a completion and is done without any interaction with ULP [RFC5040]. 5. Enhanced MPA Connection Establishment Below we provide an overview of Enhanced Connection Setup. The goal is to allow standard negotiation of ORD/IRD setting on both sides of the RDMA connection and/or to negotiate the initial data transfer operation by the Initiator when the existing 'client sends first' rule does not match application requirements. The RDMA connection Initiator sends an MPA Request, as specified in [RFC5044]; the new format defined here allows for: o Standardized negotiation of ORD and IRD. o Negotiation of an RTR message. The RDMA connection Responder processes the MPA Request and generates an MPA Reply, as specified in [RFC5044]; the new format completes the negotiation. The local interface SHOULD require the ULP to explicitly configure on Kanevsky, et al. Expires March 13, 2012 [Page 11] Internet-Draft Enhanced RDMA Connection Establishment September 2011 a per-application or per-connection basis when an explicit RTR message will be required. Piggy-backing the RTR on a Client's first message is a valuable optimization for most connections. The RDMA connection Initiator MUST NOT allow any later FULPDUs to be transmitted before the RTR message. One method to achieve that is to delay notifying the ULP that the RDMA connection has been established until after any required RTR Message has been transmitted. All MPA exchanges are performed via TCP prior to RDMA establishment, and are therefore signaled via TCP and not via RDMA completion. 6. Enhanced MPA Request/Reply Frames Enhanced RDMA connection establishment uses an alternate format for MPA Requests and Replies, as follows: 0 1 2 3 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 0 | | + Key (16 bytes containing "MPA ID Req Frame") + 4 | (4D 50 41 20 49 44 20 52 65 71 20 46 72 61 6D 65) | + Or (16 bytes containing "MPA ID Rep Frame") + 8 | (4D 50 41 20 49 44 20 52 65 70 20 46 72 61 6D 65) | + + 12 | | +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 16 |M|C|R|S| Res | Rev | PD_Length | +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ | | ~ ~ ~ Private Data ~ | | | +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ | | +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ Key: Unchanged from [RFC5044]. M: Unchanged from [RFC5044]. C: Unchanged from [RFC5044]. Kanevsky, et al. Expires March 13, 2012 [Page 12] Internet-Draft Enhanced RDMA Connection Establishment September 2011 R: Unchanged from [RFC5044]. S: One if the Private Data begins with the enhanced RDMA connection establishment data. Zero otherwise. Res: One bit smaller than in [RFC5044], otherwise unchanged. In [RFC5044] 'Res' field, in which the newly defined 'S' bit resides, is reserved for future use. [RFC5044] specifies that 'RES' MUST be set to zero when sending, and MUST NOT be checked on reception, making use of S bit backwards compatible with the original MPA frame format. When the S bit is set to zero, no additional private data is used for enhanced RDMA connection establishment, and therefore the resulting MPA request and reply frames are identical to the unenhanced protocol. Rev: This field contains the revision of MPA. To use any enhanced connection establishment feature this MUST be set to two or higher, If no enhanced connection establishment features are desired it MAY be set to one. A host accepting MPA connections MUST continue to accept MPA Requests with version one even if it supports version two. PD_Length: Unchanged from [RFC5044]. This is the total length of the Private Data field, including the enhanced RDMA connection establishment data if present. Private Data: Unchanged from [RFC5044]. However, if the 'S' flag is set, Private Data begins with enhanced RDMA connection establishment data. 7. Enhanced SCTP Session Control Chunks The type of the SCTP Session Control Chunk is defined by a Function Code (see [RFC4960]). [RFC5043] already defines codes for 'DDP Stream Session Initiate' and 'DDP Stream Session Accept', which are equivalent to a MPA Request Frame and an accepting MPA Reply Frame. Enhanced RDMA connection establishment requires three additional Function codes. All DDP Stream Session Functional Codes are listed below: DDP Stream Session Initiate: 0x001 DDP Stream Session Accept: 0x002 Kanevsky, et al. Expires March 13, 2012 [Page 13] Internet-Draft Enhanced RDMA Connection Establishment September 2011 DDP Stream Session Reject: 0x003 DDP Stream Session Terminate: 0x004 Enhanced DDP Stream Session Initiate: 0x005 Enhanced DDP Stream Session Accept: 0x006 Enhanced DDP Stream Session Reject: 0x007 The Enhanced Reject function code MUST be used to indicate rejection of enhanced DDP stream session for a configuration that would have been accepted for unenhanced DDP Stream Session negotiation. The Enhanced DDP stream session establishment follows the same rules as the standard DDP stream session establishment as defined in [RFC5043]. ULP-supplied Private Data MUST be included for Enhanced DDP Stream Session Initiate, Enhanced DDP Stream Session Accept, and Enhanced DDP Stream Session Reject messages. Private Data length MUST NOT exceed 512 bytes in any message, including enhanced RDMA connection establishment data. Private Data MUST NOT be included in the DDP Stream Session TERM message. Received Extended DDP Stream Session Control messages SHOULD be reported to the ULP. If reported, any supplied Private Data MUST be available for the ULP to examine. For example, a received Extended DDP Stream Session Control message is not reported to ULP if none of the requested RTR message types are supported by receiver. In this case, Provider MAY generate reject reply message indicating which RTR message types it supports. The enhanced DDP stream management MUST use the DDP stream session termination function code to terminate a stream established using enhanced DDP stream session function codes. [RFC5043] already supports either side sending the first DDP Message since the Payload Protocol Identifier (PPID) already distinguishes between Session Establishment and DDP Segments. The enhanced RDMA Connection Establishment provides to the ULP a transport independent way to support peer-to-peer model. The following additional Legal Sequences of DDP Stream Session messages are defined: Kanevsky, et al. Expires March 13, 2012 [Page 14] Internet-Draft Enhanced RDMA Connection Establishment September 2011 o Enhanced Active/Passive Session Accepted: as with section 6.2 of [RFC5043], but with the extended opcodes as defined in this document. o Enhanced Active/Passive Session Rejected: as with section 6.3 of [RFC5043], but with the extended opcodes as defined in this document. o Enhanced Active/Passive Session Non-ULP Rejected: as with section 6.4 of [RFC5043], but with the extended opcodes as defined in this document. 8. MPA Error Reporting The RDMA connection establishment protocol is layered upon [RFC5040] and [RFC5041]. Any enhanced RDMA connection establishment error generates an MPA termination message to a peer. [RFC5040] defines a triplet of protocol layers, error types and error codes for error specification. MPA negotiation for RDMA connection establishment uses the following layer and error type for MPA error reporting: Layer: 0010b - LLP/MPA Error Type: 0x3 - Lower Layer Protocol (LLP) While [RFC5044] defines four error codes, [RFC5043] does not define any. Enhanced RDMA connection establishment extends [RFC5044] error codes by adding two new error codes. Thus, enhanced RDMA connection establishment is backward compatible with both [RFC5043] and [RFC5044]. The following error codes are defined for enhanced RDMA connection establishment negotiation: Error Code Description -------------------------------------------------------- 0x5 Local catastrophic 0x6 Insufficient IRD resources 0x7 No matching RTR option 9. Enhanced RDMA Connection Establishment Data Enhanced RDMA Connection Establishment places the following 32 bits at the beginning of the Private data field of the MPA Request and Reply Frames or the "DDP Stream Session Initiate" and "DDP Stream Session Accept" SCTP Session Control Chunks. ULP specified private data follows this field. The maximum amount of ULP specified private Kanevsky, et al. Expires March 13, 2012 [Page 15] Internet-Draft Enhanced RDMA Connection Establishment September 2011 data is therefore reduced by 4 bytes. Note that this field MUST be sent in network byte order. 0 1 2 3 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 0 |A|B| IRD |C|D| ORD | 4 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ IRD: Inbound RDMA Read Queue Depth. ORD: Outbound RDMA Read Queue Depth. A: Control Flag for connection model. B: Control Flag for zero length FULPDU (Send) RTR message. C: Control Flag for zero length RDMA Write RTR message. D: Control Flag for zero length RDMA Read RTR message. 9.1. IRD and ORD Negotiation IRD and ORD are used for negotiation of Inbound RDMA Read Request Queue depths for both endpoints of the RDMA connection. IRD is used to configure the depth of the Inbound RDMA Read Request Queue (IRRQ) on each endpoint. ORD is used to limit the number of simultaneous outbound RDMA Read Requests allowed at at given point in time in order to avoid IRRQ overruns at the remote endpoint. In order to describe the negotiation of both local endpoint and remote endpoint ORD and IRD values, four terms are defined: Initiator IRD: IRD value sent in the MPA request or "DDP Stream Session Initiate" SCTP Session Control Chunk. This is the value of the Initiator's IRD at the time of the MPA Request generation. The Responder sets its local ORD value to this value or less. Initiator IRD is the maximum number of simultaneous inbound RDMA Read Requests which the Initiator can support for the requested connection. Initiator ORD: ORD value in the MPA request or "DDP Stream Session Initiate" SCTP Session Control Chunk. This is the initial value of the Initiator's ORD at the time of the MPA Request generation and also a request to the Responder to support a Responder IRD of at least this value. Initiator ORD is the maximum number of simultaneous outbound RDMA Read operations that the Initiator Kanevsky, et al. Expires March 13, 2012 [Page 16] Internet-Draft Enhanced RDMA Connection Establishment September 2011 desires the Responder to support for the requested connection. Responder IRD: IRD value returned in the MPA reply or "DDP Stream Session Accept" SCTP Session Control Chunk. This is the actual value that the Responder set for its local IRD. This value is greater than or equal to Initiator ORD for successful negotiations. Responder IRD is the maximum number of simultaneous inbound RDMA Read Requests that the Responder actually can support for the requested connection. Responder ORD: ORD value returned in the MPA reply or "DDP Stream Session Accept" SCTP Session Control Chunk. This is the actual value that the Responder used for ORD and is less than or equal to Initiator IRD for successful negotiations. Responder ORD is the maximum number of simultaneous outbound RDMA Read operations that the Responder will allow for the requested connection. The relationships between these parameters after a successful negotiation is complete are the following: Initiator ORD <= Responder IRD Responder ORD <= Initiator IRD The Responder and Initiator MUST pass the peer's provided IRD and ORD values to the ULP, in addition to using the values as calculated by the preceding rules. Responder ORD SHOULD be set to a value less than or equal to Initiator IRD. If Initiator ORD is insufficient to support the selected connection model, Responder IRD MAY be increased, for example if Initiator ORD is 0 (RDMA Reads will not be used by the ULP) and the Responder supports a zero length RDMA Read RTR message, then Responder IRD can be set to 1. The Responder MUST set its ORD at most to Initiator IRD. The Responder MAY reject the connection request if Initiator IRD is not sufficient for the ULP required ORD and specify the required ORD in the MPA Reject frame Responder ORD. Thus, the TERM message MUST contain Layer 2, Error Type 3, Error Code 6. Upon receiving the MPA Accept frame from the Responder, the Initiator MUST set its IRD at least to Responder ORD and its ORD at most to Responder IRD. If the Initiator does not have sufficient resources for the required IRD, it MUST send a TERM message to the Responder indicating insufficient resources, and terminate the connection due to insufficient resources. Thus, the TERM message MUST contain Layer 2, Error Type 3, Error Code 6. Kanevsky, et al. Expires March 13, 2012 [Page 17] Internet-Draft Enhanced RDMA Connection Establishment September 2011 The Initiator MUST pass the Responder provided IRD and ORD to the ULP for both MPA Accept and Reject messages. The Initiator ULP can decide its course of action. For example, the Initiator ULP may terminate the established connection and renegotiate Responder ORD. An all ones value (0x3FFF) indicates that automatic negotiation of the IRD or ORD is not desired, and that the ULP will be responsible for it. The Responder MUST respond to an Initiator ORD value of 0x3FFF by leaving its local endpoint IRD value unchanged, and setting IRD to 0x3FFF in its reply message. The Initiator MUST leave its local endpoint ORD value unchanged upon receiving a Responder IRD value of 0x3FFF. The Responder MUST respond to an Initiator IRD value of 0x3FFF by leaving its local endpoint ORD value unchanged, and setting ORD to 0x3FFF in its reply message. The Initiator MUST leave its local endpoint IRD value unchanged upon receiving a Responder ORD value of 0x3FFF. 9.2. Peer-to-Peer Connection Negotiation Control Flag A value 1 indicates that a peer-to-peer connection model is being performed, and value 0 indicates a client-server model. Control Flag B value 1 indicates that a zero length FULPDU (Send) RTR message is requested for the Initiator and supported by the Responder, respectively, 0 otherwise. Control Flag C value 1 indicates that a zero length RDMA Write RTR message is requested for the Initiator and supported by the Responder, respectively, 0 otherwise. Control Flag D value 1 indicates that a zero length RDMA Read RTR message is requested for the Initiator and supported by the Responder, respectively, 0 otherwise. The Initiator MUST set Control Flag A to 1 for peer-to-peer model. The Initiator MUST set each Control Flag B, C and D to 1 for each of the options it supports, if Control Flag A is set to 1. The Responder MUST support at least one RTR message option if it supports Enhanced RDMA connection establishment. If Control Flag A is 1 in the MPA request message then the Responder MUST set Control Flag A to 1 in the MPA reply message. For each Initiator supported RTR message option the Responder SHOULD set the corresponding Control Flag if the Responder can support that option in an MPA reply. The Responder is not required to specify all RTR message options it supports. The Responder MUST set at least one RTR message option if it supports more than one Initiator specified RTR message option. The Responder MAY include additional RTR message options it supports, even if not requested by any Initiator specified RTR message options. If the Responder does not support any of the Initiator specified RTR message options then the Responder MUST set at least one RTR message type option it supports. Kanevsky, et al. Expires March 13, 2012 [Page 18] Internet-Draft Enhanced RDMA Connection Establishment September 2011 Upon receiving the MPA accept frame with Control Flag A set to 1, the Initiator MUST generate one of the negotiated RTR messages. If the Initiator is not able to generate any of the Responder supported RTR messages, then it MUST send a TERM message to the Responder indicating failure to negotiate a mutually compatible connection model or RTR option, and terminate the connection. Thus, the TERM message MUST contain Layer 2, Error Type 3, Error Code 7. The ULP can negotiate a ULP level RTR message when a Provider level RTR message cannot be negotiated. The Initiator MUST set Control Flag A to 0 for client-server model. The Responder MUST set Control Flag A to 0 if Control Flag A is 0 in request. If Control Flag A is set to 0 then Control Flags B, C and D MUST also be set to 0. On reception if Control Flag A is set to 0 then Control Flags B, C, and D MUST be ignored. 9.3. Enhanced Connection Negotiation Flow The RTR message type and ORD/IRD negotiation follows the following order: Initiator (MPA Request) --> Set Control Flag A to 1 to indicate peer-to-peer connection model and Initiator IRD, ORD setting on local Endpoint of the connection. Set Control Flags B, C, and D to 1 for each Initiator-supported option of RTR message. Responder (MPA Reply) <-- Match the Initiator Control Flag A value and set ORD/IRD to the Responder local endpoint values based upon the Initiator initial ORD/IRD values and the number of simultaneous RDMA Read Requests required by the ULP. Sets Control Flags B, C, and D to 1 for Responder-supported options of RTR message options for peer-to-peer connection model and sets the Responder IRD/ORD actual values. Initiator (First RDMA Message) --> After the Initiator modifies its ORD/IRD to match the Responder's values as stated above, the Initiator sends the first message of negotiated RTR message option. If no matching RTR message option exists then the Initiator sends a TERM message. The Initiator or Responder MUST generate the TERM message that contains Layer 2, Error Type 3, Error Code 5 when it encounters any error locally for which the special Error Code is not defined in section Section 8 before resetting the connection. Kanevsky, et al. Expires March 13, 2012 [Page 19] Internet-Draft Enhanced RDMA Connection Establishment September 2011 10. Interoperability The Initiator MUST NOT use the enhanced RDMA connection establishment formats or function codes when no enhanced functionality is desired. The Responder MUST continue to accept unenhanced connection requests. There are three Initiator/Responder cases that involve enhanced MPA: both the Initiator and Responder, only the Responder, and only the Initiator. The enhanced MPA frame is defined by field 'S' set to 1. Enhanced MPA Initiator and Responder: If the Responder receives an enhanced MPA message, it MUST respond with an enhanced MPA message. Enhanced MPA Responder only: If the Responder receives an unenhanced MPA message ('S' is set to 0), it MUST respond with an unenhanced MPA message. Enhanced MPA Initiator only: If the Responder receives an enhanced MPA message and it does not support enhanced RDMA connection establishment, it MUST close the TCP connection and exit MPA. From a standard RDMA connection establishment point of view enhanced MPA frame is improperly formatted as stated in [RFC5044]. Thus, both the Initiator and Responder report TCP connection termination to an application locally. In this case the Initiator MAY attempt to establish an RDMA connection using the unenhanced MPA protocol as defined in [RFC5044] if this protocol is compatible with the application, and let ULP deal with ORD and IRD, and peer-to-peer negotiations. 11. IANA Considerations This document has no IANA considerations. 12. Security Considerations The security considerations from RFC 5044 apply and the changes in this document do not introduce new security considerations. 13. Acknowledgements The authors wish to thank Sean Hefty, Dave Minturn, Tom Talpey, David Black and David Harrington for their valuable contributions and reviews of this document. Kanevsky, et al. Expires March 13, 2012 [Page 20] Internet-Draft Enhanced RDMA Connection Establishment September 2011 14. References 14.1. Normative References [RFC2119] Bradner, S., "Key words for use in RFCs to Indicate Requirement Levels", BCP 14, RFC 2119, March 1997. [RFC4960] Stewart, R., "Stream Control Transmission Protocol", RFC 4960, September 2007. [RFC5040] Recio, R., Metzler, B., Culley, P., Hilland, J., and D. Garcia, "A Remote Direct Memory Access Protocol Specification", RFC 5040, October 2007. [RFC5041] Shah, H., Pinkerton, J., Recio, R., and P. Culley, "Direct Data Placement over Reliable Transports", RFC 5041, October 2007. [RFC5043] Bestler, C. and R. Stewart, "Stream Control Transmission Protocol (SCTP) Direct Data Placement (DDP) Adaptation", RFC 5043, October 2007. [RFC5044] Culley, P., Elzur, U., Recio, R., Bailey, S., and J. Carrier, "Marker PDU Aligned Framing for TCP Specification", RFC 5044, October 2007. 14.2. Informative References [DAPL] "Direct Access Programming Library", . [IBTA] "InfiniBand Architecture Specification Release 1.2.1", . [OFA] "OFA verbs & APIs", . [OpenMP] McGraw-Hill, "Parallel Programming in C with MPI and OpenMP", 2003. [PPMPI] Morgan Kaufmann Publishers Inc., "Parallel Programming with MPI", 2008. [RDMAC] "RDMA Protocol Verbs Specification (Version 1.0)", . [RDS] Open Fabrics Association, "Reliable Datagram Socket", Kanevsky, et al. Expires March 13, 2012 [Page 21] Internet-Draft Enhanced RDMA Connection Establishment September 2011 2008, . [UsingMPI] MIT Press, "Using MPI-2: Advanced Features of the Message Passing Interface", 1999. [VIA] Compaq, Intel, Microsoft, "Virtual Interface Architecture Specification", 1997, . Authors' Addresses Arkady Kanevsky (editor) Dell Inc. One Dell Way, MS PS2-47 Round Rock, TX 78682 USA Phone: +1-512-728-0000 Email: arkady.kanevsky@gmail.com Caitlin Bestler (editor) Nexenta Systems 555 E El Camino Real #104 Sunnyvale, CA 94087 USA Phone: +1-949-528-3085 Email: Caitlin.Bestler@nexenta.com Robert Sharp Intel LAD High Performance Message Passing, Mailstop: AN1-WTR1 1501 South Mopac, Suite 400 Austin, TX 78746 USA Phone: +1-512-493-3242 Email: robert.o.sharp@intel.com Kanevsky, et al. Expires March 13, 2012 [Page 22] Internet-Draft Enhanced RDMA Connection Establishment September 2011 Steve Wise Open Grid Computing 4030 Braker Lane STE 130 Austin, TX 78759 USA Phone: +1-512-343-9196 x101 Email: swise@opengridcomputing.com Kanevsky, et al. Expires March 13, 2012 [Page 23]