Storage Maintenance (storm) Working Group Donald Wood Internet Draft Robert Sharp Intended status: Standards Track Kenneth Keels Expires: May 2014 Intel Corporation November 26, 2013 RDMA Protocol Extensions V2 draft-wood-storm-rdmap-ext-v2-00.txt Status of this Memo This Internet-Draft is submitted to IETF in full conformance with the provisions of BCP 78 and BCP 79. Internet-Drafts are working documents of the Internet Engineering Task Force (IETF). Note that other groups may also distribute working documents as Internet-Drafts. The list of current Internet- Drafts is at http://datatracker.ietf.org/drafts/current. Internet-Drafts are draft documents valid for a maximum of six months and may be updated, replaced, or obsoleted by other documents at any time. It is inappropriate to use Internet-Drafts as reference material or to cite them other than as "work in progress." This Internet-Draft will expire on May 26, 2014. Copyright Notice Copyright (c) 2013 IETF Trust and the persons identified as the document authors. All rights reserved. This document is subject to BCP 78 and the IETF Trust's Legal Provisions Relating to IETF Documents (http://trustee.ietf.org/license-info) in effect on the date of publication of this document. Please review these documents carefully, as they describe your rights and restrictions with respect to this document. Code Components extracted from this Wood et al. Expires May 26, 2014 [Page 1] Internet-Draft RDMA Protocol Extensions November 2013 document must include Simplified BSD License text as described in Section 4.e of the Trust Legal Provisions and are provided without warranty as described in the Simplified BSD License. Abstract This document specifies extensions to the IETF Remote Direct Memory Access Protocol and RDMA Protocol Extensions. RDMAP provides read and write services directly to applications and enables data to be transferred directly into Upper Layer Protocol (ULP) Buffers without intermediate data copies. The extensions specified in this document provide the following capabilities and/or improvements to eliminate application visible differences with other RDMA technologies: Send with Immediate Data Operations and a new form of RDMA Read Operation. Table of Contents 1. Introduction...................................................3 1.1. RDMAP Extensions Implementation Requirements..............3 2. Requirements Language..........................................4 3. Glossary.......................................................4 4. Header Format Extensions.......................................5 4.1. RDMAP Control and Invalidate STag Fields..................5 4.2. RDMA Message Definitions..................................9 5. Send with Immediate Data Operations...........................10 5.1. RDMAP Interactions with ULP for Immediate Data...........10 5.2. Ordering and Completions.................................11 5.3. Send with Immediate Data Operations......................11 6. RDMA Read V2..................................................11 6.1. RDMA Read V2 Request Header Format.......................12 6.2. RDMA Read V2 Response Header Format......................12 6.3. Ordering and Completions.................................12 7. Ordering and Completions Table................................13 8. Error Processing..............................................15 8.1. Errors Detected at the Local Peer........................15 8.2. Errors Detected at the Remote Peer.......................15 9. Security Considerations.......................................16 10. IANA Considerations..........................................16 11. References...................................................16 Wood et al. Expires May 26, 2014 [Page 2] Internet-Draft RDMA Protocol Extensions November 2013 11.1. Normative References....................................16 11.2. Informative References..................................17 12. Acknowledgments..............................................17 Appendix A. DDP Segment Formats for RDMA Messages................18 A.1. DDP Segment for RDMA Read V2 Request.....................18 A.2. DDP Segment for RDMA Read V2 Response....................18 A.3. DDP Segment for Send with Immediate Data and Send with Solicited Event and Immediate Data............................19 A.4. DDP Segment for Send with Invalidate and Immediate Data and Send with Invalidate and Solicited Event and Immediate Data...21 1. Introduction The RDMA Protocol [RFC5040] and RDMA Protocol Extensions[RFCYYYY] provides capabilities for zero copy data communications that preserve memory protection semantics, enabling more efficient network protocol implementations. This document specifies the following extensions to the RDMA Protocol (RDMAP): o Send with Immediate Data operations allow the ULP at the sender to provide a small amount of immediate data with an RDMA Message. o RDMA Read with untagged RDMA Read Response. Support for RDMA Read requests and responses without exposing the data sink buffer address(es) on the wire. Other RDMA transport protocols define the functionality added by these extensions leading to differences in RDMA applications and/or Upper Layer Protocols. Removing these differences in the transport protocols simplifies these applications and ULPs and is the main motivation for the extensions specified in this document. The changes in this RFC require the RDMAP version to move to version 2. 1.1. RDMAP Extensions Implementation Requirements The discovery and negotiation of the RDMA Extensions and the RDMAP Extensions is covered in MPA Extensions [RFCZZZZ]. The following paragraphs describe the implementation that advertises RDMAP Extensions and negotiates their use. When the RDMAP version is negotiated to 2, all features described in RDMA Extensions [RFCYYYY] and RDMA Extensions V2 [RFCXXXX] are Wood et al. Expires May 26, 2014 [Page 3] Internet-Draft RDMA Protocol Extensions November 2013 supported. This requirement allows applications to count on all the features being available. These include: * Atomic Operations * Immediate Data and Immediate Data with SE Operations * Send with Immediate Data, Send with Solicited Event and Immediate Data, Send with Invalidate and Immediate Data, and Send with Invalidate and Solicited Event and Immediate Data Operations. * RDMA Read V2 Request and RDMA Read V2 Response Operations. 2. Requirements Language The key words "MUST", "MUST NOT", "REQUIRED", "SHALL", "SHALL NOT", "SHOULD", "SHOULD NOT", "RECOMMENDED", "MAY", and "OPTIONAL" in this document are to be interpreted as described in RFC-2119 [RFC2119]. 3. Glossary This document is an extension of [RFC5040] and key words are defined in the glossary of the referenced document. Immediate Data - a small fixed size portion of data sent from the Data Source to a Data Sink Requester - the sender of an RDMA Operation request. Responder - the receiver of an RDMA Operation request. Wood et al. Expires May 26, 2014 [Page 4] Internet-Draft RDMA Protocol Extensions November 2013 4. Header Format Extensions The control information of RDMA Messages is included in DDP protocol [RFC5041] defined header fields, with the following new formats: . Four new RDMA Messages carry additional RDMAP headers for Send with Immediate Data variants. The Send with Immediate Data, Send with Solicited Event and Immediate Data, Send with Invalidate and Immediate Data, and Send with Invalidate and Solicited Event and Immediate Data messages include 8 bytes of data following the RDMAP header. . Two new RDMA Messages carry additional RDMAP headers for the new RDMA Read Request and RDMA Read Response messages. The new RDMA Read Request message includes a revised definition following the untagged RDMAP header that removes the Data Sink STag and Tagged Offset fields. The new RDMA Read Response is an untagged message. 4.1. RDMAP Control and Invalidate STag Fields Figure 1 depicts the format of the DDP Control and RDMAP Control fields, in the style and convention of [RFC5040]. The DDP Control Field consists of the T,L, Resrv and DV fields [RFC5041]. The RDMAP Control Field consists of the RV, Rsv and Opcode fields [RFC5040]. When the RDMAP version has been negotiated to 2, the RDMAP RV field MUST be set to 2 for all Operations. The RDMA Messages defined by this specification use all 8 bits of the RDMAP Control Field. The first octet reserved for ULP use in the DDP Protocol MUST be used by the RDMAP to carry the RDMAP Control Field. The ordering of the bits in the first octet MUST be as shown in Figure 1. Wood et al. Expires May 26, 2014 [Page 5] Internet-Draft RDMA Protocol Extensions November 2013 0 1 2 3 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ |T|L| Resrv | DV| RV|R| Opcode | | | | | | |s| | | | | | | |v| | +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ | Invalidate STag | +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ Figure 1 DDP Control and RDMAP Control Fields Figure 2 defines the values of RDMA Opcode field that MUST be used for the RDMA Messages defined in this specification. The Opcode field has been extended one additional bit to accommodate the additional opcodes. In RFC 5040, bit 7 was reserved and required to be zero. The expansion of the opcode field size is compatible with RFC 5040 since only the new opcodes will set this bit to one. Figure 2 also defines when the STag, Tagged Offset, and Queue Number fields MUST be provided for the RDMA Messages defined in this specification. All RDMA Messages defined in this specification MUST have: The RDMA Version (RV) field: 10b. Opcode field: See Figure 2. To support RV field set to 10b, an RNIC MUST: * Implement [RFCYYYY] RDMA Protocol Extensions (Atomic, Immediate Data and Immediate Data with SE Operations). * Implement Send with Immediate Data, Send with Solicited Event and Immediate Data, Send with Invalidate and Immediate Data, and Send with Invalidate and Solicited Event and Immediate Data operations. * Implement RDMA Read V2 Request and RDMA Read V2 Response messages. Wood et al. Expires May 26, 2014 [Page 6] Internet-Draft RDMA Protocol Extensions November 2013 For Send with Invalidate and Immediate Data, and Send with Invalidate and Solicited Event and Immediate Data, the second through fifth octets of the DDP RsvdULP field MUST be used by RDMAP to carry the Invalidate STag. Wood et al. Expires May 26, 2014 [Page 7] Internet-Draft RDMA Protocol Extensions November 2013 -------+-----------+-------+------+-------+-----------+-------------- RDMA | Message | Tagged| STag | Queue | Invalidate| Message Opcode | Type | Flag | and | Number| STag | Length | | | TO | | | Communicated | | | | | | between DDP | | | | | | and RDMAP -------+-----------+-------+------+-------+-----------+-------------- -------+-----------+------------------------------------------------- 10001b | RDMA Read | 0 | N/A | 1 | N/A | Yes | Request V2| | | | | -------+-----------+------------------------------------------------- 10010b | RDMA Read | 0 | N/A | 3 | N/A | Yes | Response | | | | | | V2 | | | | | -------+-----------+------------------------------------------------- 10011b | Send with | 0 | N/A | 0 | N/A | Yes | Immediate | | | | | | Data | | | | | -------+-----------+------------------------------------------------- 10100b | Send with | 0 | N/A | 0 | Yes | Yes | Invalidate| | | | | | and | | | | | | Immediate | | | | | | Data | | | | | -------+-----------+------------------------------------------------- 10101b | Send with | 0 | N/A | 0 | N/A | Yes | SE and | | | | | | Immediate | | | | | | Data | | | | | -------+-----------+------------------------------------------------- 10110b | Send with | 0 | N/A | 0 | Yes | Yes | Invalidate| | | | | | and SE and| | | | | | Immediate | | | | | | Data | | | | | -------+-----------+------------------------------------------------- Figure 2 Additional RDMA Usage of DDP Fields Note: N/A means Not Applicable. All other DDP and RDMAP control fields MUST be set as described in [RFC5040] except for the RV field which MUST be 2. Wood et al. Expires May 26, 2014 [Page 8] Internet-Draft RDMA Protocol Extensions November 2013 4.2. RDMA Message Definitions The following figure defines which RDMA Headers MUST be used on each new RDMA Message and which new RDMA Messages are allowed to carry ULP payload: -------+-----------+-------------------+------------------------- RDMA | Message | RDMA Header Used | ULP Message allowed in Message| Type | | the RDMA Message OpCode | | | | | | -------+-----------+-------------------+------------------------- -------+-----------+-------------------+------------------------- 10001b | RDMA Read | RDMA Read Request | No | Request V2| Header V2 | -------+-----------+-------------------+------------------------- 10010b | RDMA Read | None | No | Response | | | V2 | | -------+-----------+-------------------+------------------------- 10011b | Send with | Immediate Data | Yes | Immediate | header | | Data | in the last ULPDU | -------+-----------+------------------------------------------------- 10100b | Send with | Immediate Data | Yes | Invalidate| header | | and | in the last ULPDU | | Immediate | | | Data | | -------+-----------+------------------------------------------------- 10101b | Send with | Immediate Data | Yes | SE and | header | | Immediate | in the last ULPDU | | Data | | -------+-----------+------------------------------------------------- 10110b | Send with | Immediate Data | Yes | Invalidate| header | | and SE and| in the last ULPDU | | Immediate | | | Data | | -------+-----------+------------------------------------------------- Figure 3 RDMA Message Definitions Wood et al. Expires May 26, 2014 [Page 9] Internet-Draft RDMA Protocol Extensions November 2013 This extension uses RDMAP Queue Number 3 for Untagged Buffers for RDMA Read V2 Responses. This queue is used for tracking outstanding RDMA Read V2 Requests. 5. Send with Immediate Data Operations The Send with Immediate Data operations are used to improve ULP processing efficiency by allowing 8 bytes of immediate data to be delivered to the ULP at the Remote Peer. 5.1. RDMAP Interactions with ULP for Immediate Data For Send with Immediate Data operations, the following are the interactions between the RDMAP Layer and the ULP: . At the Data Source: . The ULP passes to the RDMAP Layer the following: . Eight bytes of ULP Immediate Data . When the Immediate Data operation Completes, an indication of the Completion results. . At the Data Sink: . When any of the Send with Immediate Data variants Completes successfully, the RDMAP Layer passes the following information to the ULP Layer: . Eight bytes of Immediate Data . An Event, if the Data Sink is configured to generate an Event. . When any of the Send with Immediate Data variants Completes in error, the Data Sink RDMAP Layer will pass up the corresponding error information to the Data Sink ULP and send a Terminate Message to the Data Source RDMAP Layer. The Data Source RDMAP Layer will then pass up the Terminate Message to the ULP. Wood et al. Expires May 26, 2014 [Page 10] Internet-Draft RDMA Protocol Extensions November 2013 5.2. Ordering and Completions Ordering and completion rules for Immediate Data are the same as those for a Send operation as described in section 5.5 of RFC 5040. 5.3. Send with Immediate Data Operations The Send with Immediate Data, Send with Invalidate and Immediate Data, Send with Solicited Event and Immediate Data, and Send with Solicited Event and Invalidate and Immediate Data messages follow the same model as described in section 5.3 of RFC 5040 with the following differences: . The opcodes for the Send with Immediate Data variants have different values from the RDMA Send (without Immediate Data Messages). . The L bit MUST be set to 1 only in the ULPDU that contains immediate data. . The immediate data MUST be the only data allowed in the ULPDU with the L bit set to 1. . At the data sink, if the operation is completed successfully, the RDMAP Layer MUST pass the eight bytes of immediate data to the ULP Layer. When any Send with Immediate Data variant has a message length of zero, the only ULDPU MUST have the immediate data and the L bit MUST be set to 1. 6. RDMA Read V2 The definition of RDMA Read in RFC 5040 requires exposure of the Data Sink STag and Data Sink Tagged Offset. Applications have to be aware of this exposure since it is different from other RDMA technologies. Dealing with the exposure in a secure manner causes application overhead in the fast path operations specific to RDMAP. Two new RDMA messages are needed to remove the exposure of the Data Sink buffer information. The new RDMA Read message is composed of an RDMA Read V2 Request that removes the SinkSTag and SinkTO parameters. The RDMA Read V2 Response is a simple untagged DDP message targeting queue number 3. RDMA Read V2 Response Messages MUST use the Untagged Buffer model with QN=3. Queue number 3 MUST be used to track outstanding RDMA Read V2 Request messages at the Requester. When the RDMA Read V2 Wood et al. Expires May 26, 2014 [Page 11] Internet-Draft RDMA Protocol Extensions November 2013 Response message is received, the MSN MUST be used to locate the corresponding RDMA Read V2 request in order to place the data and Complete the Operation. 6.1. RDMA Read V2 Request Header Format The RDMA Read V2 Request Message carries an RDMA Read V2 Request Header that describes the Data Source Buffers used by the RDMA Read operation. The RDMA Read Request Header immediately follows the DDP header. The RDMAP layer passes to the DDP layer an RDMAP Control Field. The following figure depicts the RDMA Read V2 Request Header that MUST be used for all RDMA Read V2 Request Messages: 0 1 2 3 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ | RDMA Read Message Size (RDMARDSZ) | +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ | Data Source STag (SrcSTag) | +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ | Data Source Tagged Offset (SrcTO) | + + | | +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ Figure 4 RDMA Read V2 Message Header The RDMA Read Message Size, Data Source STag, and Data Source Tagged Offset parameters are the same as defined in RFC 5040 section 4.4. 6.2. RDMA Read V2 Response Header Format The RDMA Read V2 Response Message does not include an RDMAP header. The RDMAP layer passes to the DDP layer an RDMAP Control Field. The RDMA Read V2 Response Message is fully described by the DDP Headers of the DDP Segments associated with the Message. 6.3. Ordering and Completions Ordering and completion rules for RDMA Read V2 Request are the same as those for a Send operation as described in section 5.2.1 of RFC 5040 with one exception. Wood et al. Expires May 26, 2014 [Page 12] Internet-Draft RDMA Protocol Extensions November 2013 . The Local Peer is no longer required to use multiple RDMA Read Request Messages if the Local Peer wishes to reference multiple data sink buffers. This is because RDMA Read V2 Response is untagged. Ordering and completion rules for the RDMA Read V2 Response are the same as the Send operation described in section 5.3 of RFC 5040 for a Remote Peer with one exception. . The queue number field for the DDP layer MUST be 3. 7. Ordering and Completions Table For reference, the following table is copied from [RFCYYYY] and pared down to show only Send and Read Operations. It summarizes the ordering relationships for from the standpoint of Local Peer issuing the Operations. Note that in the table that follows, Send includes all variants: Send, Send with Invalidate, Send with Solicited Event, and Send with Solicited Event and Invalidate with and without immediate data. Finally note that RDMA Read includes the original RDMA Read and the newly defined RDMA Read V2. ----------+------------+-------------+-------------+------------------- First | Second | Placement | Placement | Ordering Operation | Operation | Guarantee at| Guarantee at| Guarantee at | | Remote Peer | Local Peer | Remote Peer ----------+------------+-------------+-------------+------------------- Immediate | Send | No Placement| Not | Completed in Data | | Guarantee | Applicable | Order | | between Send| | | | Payload and | | | | Immediate | | | | Data | | ----------+------------+-------------+-------------+------------------- Immediate | RDMA | No Placement| RDMA Read | RDMA Read Data | Read | Guarantee | Response | Response | | between | will not be | Message will | | Immediate | Placed until| not be | | Data and | Immediate | generated | | RDMA Read | Data is | until | | Request | Placed at | Immediate Data Wood et al. Expires May 26, 2014 [Page 13] Internet-Draft RDMA Protocol Extensions November 2013 | | | Remote Peer | has been | | | | Completed ----------+------------+-------------+-------------+------------------- Immediate | Immediate | No Placement| Not | Completed in Data or | Data | Guarantee | Applicable | Order Send | | | | ----------+------------+-------------+-------------+------------------- RDMA Read | Immediate | No Placement| Immediate | Not Applicable | Data | Guarantee | Data MAY be | | | between | Placed | | | Immediate | before | | | Data and | RDMA Read | | | RDMA Read | Response is | | | Request | generated | ----------+------------+-------------+-------------+------------------- Atomic | Send | No Placement| Send Payload| Not Applicable | | Guarantee | MAY be | | | between Send| Placed | | | Payload and | before | | | Atomic | Atomic | | | Request | Response is | | | | generated | ----------+------------+-------------+-------------+------------------- Atomic | RDMA | No Placement| No Placement| RDMA Read | Read | Guarantee | Guarantee | Response | | between | between | Message will | | Atomic | Atomic | not be | | Request and | Response | generated | | RDMA Read | and RDMA | until Atomic | | Request | Read | Response Message | | | Response | has been | | | | generated ----------+------------+-------------+-------------+------------------- Send | Atomic | No Placement| Atomic | Atomic Response | | Guarantee | Response | Message will not | | between Send| will not be | be generated until | | Payload and | Placed at | Send has been | | Atomic | the Local | Completed | | Request | Peer Until | | | | Send Payload| | | | is Placed | | | | at the | | | | Remote Peer | ----------+------------+-------------+-------------+------------------- RDMA | Atomic | No Placement| No Placement| Atomic Response Wood et al. Expires May 26, 2014 [Page 14] Internet-Draft RDMA Protocol Extensions November 2013 Read | | Guarantee | Guarantee | Message will | | between | between | not be generated | | Atomic | Atomic | until RDMA | | Request and | Response | Read Response | | RDMA Read | and RDMA | has been | | Request | Read | generated | | | Response | ----------+------------+-------------+-------------+------------------- 8. Error Processing In addition to error processing described in section 7 of [RFC5040], the following rules apply for the new RDMA Messages defined in this specification. 8.1. Errors Detected at the Local Peer The Local Peer MUST send a Terminate Message for each of the following cases: 1. For errors detected while creating a RDMA Read V2 Request, RDMA Read V2 Response, or Immediate Data with SE Message, or other reasons not directly associated with an incoming Message, the Terminate Message and Error code are sent instead of the Message. In this case, the Error Type and Error Code fields are included in the Terminate Message, but the Terminated DDP Header and Terminated RDMA Header fields are set to zero. 2. For errors detected on an incoming RDMA Read V2 Request, RDMA Read V2 Response, or Immediate Data with Solicited Event (after the Message has been Delivered by DDP), the Terminate Message is sent at the earliest possible opportunity, preferably in the next outgoing RDMA Message. In this case, the Error Type, Error Code, and Terminated DDP Header fields are included in the Terminate Message, but the Terminated RDMA Header field is set to zero. 8.2. Errors Detected at the Remote Peer Wood et al. Expires May 26, 2014 [Page 15] Internet-Draft RDMA Protocol Extensions November 2013 9. Security Considerations This document specifies extensions to the RDMA Protocol specification in [RFC5040], and as such the Security Considerations discussed in Section 8 of [RFC5040] apply. 10. IANA Considerations IANA is requested to add the following entries to the "RDMAP Message Operation Codes" registry of "RDDP Registries": 0x11, RDMA Read V2 Request, [RFCXXXX] 0x12, RDMA Read V2 Response, [RFCXXXX] 0x13, Send with Immediate Data, [RFCXXXX] 0x14, Send with Invalidate and Immediate Data, [RFCXXXX] 0x15, Send with Solicited Event and Immediate Data, [RFCXXXX] 0x16, Send with Invalidate and Solicited Event and Immediate Data, [RFCXXXX] RFC Editor: Please replace XXXX in all instances of [RFCXXXX] above with the RFC number of this document and remove this note. RFC Editor: Please replace YYYY in all instances of [RFCYYYY] above with the RFC number of the RDMA Extensions document and remove this note. RFC Editor: Please replace ZZZZ in all instances of [RFCZZZZ] above with the RFC number of the MPA Extensions document and remove this note. 11. References 11.1. Normative References [RFC2119] Bradner, S., "Key words for use in RFCs to Indicate Requirement Levels", BCP 14, RFC 2119, March 1997. Wood et al. Expires May 26, 2014 [Page 16] Internet-Draft RDMA Protocol Extensions November 2013 [RFC5040] Recio, R. et al., "A Remote Direct Memory Access Protocol Specification", RFC 5040, October 2007. [RFC5041] Shah, H. et al., "Direct Data Placement over Reliable Transports", RFC 5041, October 2007. [RFC5226] T. Narten and H. Alvestrand, "Guidelines for Writing an IANA Considerations Section in RFCs", RFC 5226, BCP 26, May 2008. [RFCYYYY] Shah, H. et al., "RDMA Protocol Extensions", RFC YYYY, October 2013. [RFCZZZZ] "MPA Extensions" (not yet written) 11.2. Informative References 12. Acknowledgments The authors would like to acknowledge the following contributors who provided valuable comments and suggestions. This document was prepared using 2-Word-v2.0.template.dot. Wood et al. Expires May 26, 2014 [Page 17] Internet-Draft RDMA Protocol Extensions November 2013 Appendix A. DDP Segment Formats for RDMA Messages This appendix is for information only and is NOT part of the standard. It simply depicts the DDP Segment format for the various RDMA Messages. A.1. DDP Segment for RDMA Read V2 Request The following figure depicts an RDMA Read V2 Request DDP Segment: 0 1 2 3 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ | DDP Control | RDMA Control | +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ | Reserved (Not Used) | +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ | DDP (RDMA Read V2 Request) Queue Number | +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ | DDP (RDMA Read V2 Request) Message Sequence Number | +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ | DDP (RDMA Read V2 Request) Message Offset | +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ | RDMA Read Message Size (RDMARDSZ) | +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ | Data Source STag (SrcSTag) | +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ | Data Source Tagged Offset (SrcTO) | + + | | +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ Figure 5 RDMA Read V2 Request, DDP Segment Format A.2. DDP Segment for RDMA Read V2 Response The following figure depicts an RDMA Read V2 Response, DDP Segment: Wood et al. Expires May 26, 2014 [Page 18] Internet-Draft RDMA Protocol Extensions November 2013 0 1 2 3 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ | DDP Control | RDMA Control | +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ | Reserved (Not Used) | +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ | DDP (RDMA Read V2 Response) Queue Number | +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ | DDP (RDMA Read V2 Response) Message Sequence Number | +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ | DDP (RDMA Read V2 Response) Message Offset | +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ | Source Data | + + | | +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ Figure 6 RDMA Read V2 Response, DDP Segment Format A.3. DDP Segment for Send with Immediate Data and Send with Solicited Event and Immediate Data The following figure depicts a Send with Immediate Data and Send with Solicited and Immediate Data Request, DDP Segment: Wood et al. Expires May 26, 2014 [Page 19] Internet-Draft RDMA Protocol Extensions November 2013 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 0 1 2 3 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ | DDP Control | RDMA Control | +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ | Reserved (Not Used) | +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ | (Send) Queue Number | +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ | (Send) Message Sequence Number | +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ | (Send) Message Offset | +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ | Send ULP Payload | + + | | +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ Figure 7 Send with Immediate Data and Send with Solicited Event and Immediate Data (L bit = 0, payload only), DDP Segment Format 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 0 1 2 3 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ | DDP Control | RDMA Control | +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ | Reserved (Not Used) | +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ | (Send) Queue Number | +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ | (Send) Message Sequence Number | +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ | (Send) Message Offset | +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ | Immediate Data | + + | | +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ Figure 8 Send with Immediate Data and Send with Solicited Event and Immediate Data Operations (L bit = 1, Immediate Data only), DDP Segment Format Wood et al. Expires May 26, 2014 [Page 20] Internet-Draft RDMA Protocol Extensions November 2013 This is the same format that is used in section A.4 of RFC 5040. The differences are: . The opcodes for the immediate data operations are different. . The L bit will be set to 1 only in the ULPDU that contains immediate data. . The only data allowed in the ULPDU with the L bit set to 1 is the immediate data itself. If the send message length is zero, the only ULPDU sent has the immediate data (also, the L bit is set to 1). A.4. DDP Segment for Send with Invalidate and Immediate Data and Send with Invalidate and Solicited Event and Immediate Data The following figure depicts a Send with Invalidate and Immediate Data and Send with Invalidate and Solicited and Immediate Data Request, DDP Segment: 0 1 2 3 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ | DDP Control | RDMA Control | +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ | Invalidate STag | +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ | (Send) Queue Number | +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ | (Send) Message Sequence Number | +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ | (Send) Message Offset | +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ | Send ULP Payload | + + | | +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ Figure 9 Send with Invalidate and Immediate Data and Send with Invalidate and Solicited Event and Immediate Data (L bit = 0, payload only), DDP Segment Format Wood et al. Expires May 26, 2014 [Page 21] Internet-Draft RDMA Protocol Extensions November 2013 0 1 2 3 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ | DDP Control | RDMA Control | +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ | Invalidate STag | +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ | (Send) Queue Number | +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ | (Send) Message Sequence Number | +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ | (Send) Message Offset | +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ | Immediate Data | + + +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ Figure 10 Send with Invalidate and Immediate Data and Send with Invalidate and Solicited Event and Immediate Data (L bit = 1, Immediate Data only), DDP Segment Format This is the same format that is used in section A.5 of RFC 5040. The differences are: . The opcodes for the immediate data operations are different. . The L bit will be set to 1 only in the ULPDU that contains immediate data. . The only data allowed in the ULPDU with the L bit set to 1 is the immediate data itself. If the send message length is zero, the only ULPDU sent has the immediate data (also, the L bit is set to 1). Wood et al. Expires May 26, 2014 [Page 22] Internet-Draft RDMA Protocol Extensions November 2013 Authors' Addresses Donald Wood Intel Corporation 1300 South Mopac Expressway, Mailstop: AN4-4B Austin, TX 78746 Phone: 1-512-362-1440 Email: donald.e.wood@intel.com Robert Sharp Intel Corporation 1300 South Mopac Expressway, Mailstop: AN4-4B Austin, TX 78746 Phone: 1-512-362-1407 Email: robert.o.sharp@intel.com Kenneth Keels Intel Corporation 1300 South Mopac Expressway, Mailstop: AN4-4B Austin, TX 78746 Phone: 1-512-362-1419 Email: kenneth.g.keels@intel.com Wood et al. Expires May 26, 2014 [Page 23]