Internet DRAFT - draft-bailey-roi-ddp-rdma-arch

draft-bailey-roi-ddp-rdma-arch









                                               S. Bailey    (Sandburst)
Internet-draft Expires: July 2002

            The Architecture of Direct Data Placement (DDP)
                 And Remote Direct Memory Access (RDMA)
                         On Internet Protocols
                   draft-bailey-roi-ddp-rdma-arch-00


Status of this Memo

     This document is an Internet-Draft and is in full conformance with
     all provisions of Section 10 of RFC2026.

     Internet-Drafts are working documents of the Internet Engineering
     Task Force (IETF), its areas, and its working groups.  Note that
     other groups may also distribute working documents as Internet-
     Drafts.

     Internet-Drafts are draft documents valid for a maximum of six
     months and may be updated, replaced, or obsoleted by other
     documents at any time.  It is inappropriate to use Internet-Drafts
     as reference material or to cite them other than as "work in
     progress."

     The list of current Internet-Drafts can be accessed at
     http://www.ietf.org/ietf/1id-abstracts.txt

     The list of Internet-Draft Shadow Directories can be accessed at
     http://www.ietf.org/shadow.html.

Copyright Notice

     Copyright (C) The Internet Society (2001). All Rights Reserved.


Abstract


     This document defines an abstract architecture for Direct Data
     Placement (DDP) and Remote Direct Memory Access (RDMA) protocols to
     run on Internet Protocol-suite transport protocols.  This
     architecture does not necessarily reflect the proper way to
     implement such protocols, but is, rather, a descriptive tool for
     defining and understanding the protocols.






Bailey                      Expires July 2002                   [Page 1]

Internet-Draft           DDP & RDMA Architecture         4 February 2002


Table Of Contents

     1.   Introduction . . . . . . . . . . . . . . . . . . . . . . .   2
     2.   Direct Data Placement (DDP) Architecture . . . . . . . . .   2
     2.1. Transport Operations . . . . . . . . . . . . . . . . . . .   4
     2.2. DDP Operations . . . . . . . . . . . . . . . . . . . . . .   5
     2.3. Transport Characterstics In DDP  . . . . . . . . . . . . .   8
     3.   Remote Direct Memory Access (RDMA) Protocol Architecture .   9
     3.1. RDMA Operations  . . . . . . . . . . . . . . . . . . . . .  10
     3.2. Transport Characterstics In RDMA . . . . . . . . . . . . .  12
     4.   Security Considerations  . . . . . . . . . . . . . . . . .  13
     5.   IANA Considerations  . . . . . . . . . . . . . . . . . . .  13
          Author's Address . . . . . . . . . . . . . . . . . . . . .  13
          Full Copyright Statement . . . . . . . . . . . . . . . . .  14



1.  Introduction

     This document defines an abstract architecture for Direct Data
     Placement (DDP) and Remote Direct Memory Access (RDMA) protocols to
     run on Internet Protocol-suite transport protocols.  This
     architecture does not necessarily reflect the proper way to
     implement such protocols, but is, rather, a descriptive tool for
     defining and understanding the protocols.

     The first section describes the architecture of DDP protocols,
     including assumptions of the transports on which DDP is built.  The
     second section describes the architecture of RDMA protocols layered
     on top of DDP.

2.  Direct Data Placement (DDP) Architecture

     The central idea of general-purpose DDP is that a data sender will
     supplement the data it sends with placement information that allows
     the receiver's network interface (NI) to place the data directly at
     its final destination without any copying.  DDP can be used to
     steer received data to its final destination for any ULP without
     requiring ULP-specific behavior in the NI for each different ULP.
     Data sent with DDP information is said to be `DDP-decorated'.

     The central component of the DDP architecture is the `buffer',
     which is an object with beginning and ending addresses, and a
     method (set()) to set the value of an octet at an address.  In many
     cases, a buffer corresponds directly to a portion of host memory.
     However, DDP does not depend on this---a buffer could be a disk
     file, or anything else that can be viewed as an addressable
     collection of octets.  Abstractly, a buffer provides the interface:



Bailey                      Expires July 2002                   [Page 2]

Internet-Draft           DDP & RDMA Architecture         4 February 2002


          typedef struct {
            const address_t start;
            const address_t end;
            void            set(address_t a, uint8_t v);
          } buffer_t;


     The protocol layering and in-line data flow of DDP is:

                        Client Protocol
                      (e.g. ULP or RDMA)
                             |  ^
        undecorated messages |  | undecorated messages
      DDP-decorated messages |  | DDP-decorated message reception
                             v  |          indications
                             DDP
                              ^
                              | transport messages
                              v
                          Transport
                       (e.g. SCTP, DCP)
                              ^
                              | IP datagrams
                              v
                            . . .


     In addition to in-line data flow, the client protocol registers
     buffers with DDP, and DDP performs buffer update (set()) operations
     as a result of receiving DDP-decorated messages.

     Undecorated messages correspond directly to messages of the
     underlying transport, but must still be distinguished from DDP-
     decorated messages in some way.

     DDP-decorated messages may be split into multiple, smaller DDP-
     decorated messages each in a separate transport message.  However,
     if the transport is unreliable or unordered, DDP-decorated messages
     split across transport messages may or may not provide useful
     behavior, in the same way as splitting regular, undecorated
     messages across unreliable or unordered transport messages may or
     may not provide useful behavior.  In other words, the same
     considerations apply to building client protocols on different
     types of transports with or without the use of DDP.

     A DDP-decorated message split across transport messages looks like:





Bailey                      Expires July 2002                   [Page 3]

Internet-Draft           DDP & RDMA Architecture         4 February 2002


     DDP-decorated message:      Transport messages:

       stag=s, offset=o,          message 1:
       notify=y, id=i               |type=ddp  |
       message=                     |stag=s    |
         |aabbccddee|-------.       |offset=o  |
         ~   ...    ~----.   \      |notify=n  |
         |vvwwxxyyzz|-.   \   \     |id=?      |
                      |    \   `--->|aabbccddee|
                      |     \       ~    ...   ~
                      |      +----->|iijjkkllmm|
                      |      |
                      +      |    message 2:
                       \     |      |type=ddp  |
                        \    |      |stag=s    |
                         \   +      |offset=o+n|
                          \   \     |notify=y  |
                           \   \    |id=i      |
                            \   `-->|nnooppqqrr|
                             \      ~    ...   ~
                              `---->|vvwwxxyyzz|


     Although this picture suggests that DDP decoration information is
     carried in-line with the message payload, components of the DDP
     decoration may also be in transport-specific fields, or derived
     from transport-specific control information if the transport
     permits.

2.1.  Transport Operations

     For the purposes of this architecture, the transport provides:

          void      xpt_send(socket_t s, message_t m);
          message_t xpt_recv(socket_t s);
          msize_t   xpt_max_msize(socket_t s);


     socket_t

          a transport address, including IP addresses, ports and other
          transport-specific identifiers.

     message_t

          a string of octets.

     msize_t (unsigned integer)



Bailey                      Expires July 2002                   [Page 4]

Internet-Draft           DDP & RDMA Architecture         4 February 2002


          a message size.

     xpt_send(socket_t s, message_t m)

          send a transport message.

     xpt_recv(socket_t s)

          receive a transport message.

     xpt_max_msize(socket_t s)

          get the current maximum transport message size.  Corresponds,
          roughly, to the current path Maximum Transfer Unit (PMTU),
          adjusted by underlying protocol overheads.

     Real implementations of xpt_send() and xpt_recv() typically return
     error indications, but that is not relevant to this architecture.

2.2.  DDP Operations

     The DDP layer provides:

          void       ddp_send(socket_t s, message_t m);
          void       ddp_send_ddp(socket_t s, message_t m, ddp_addr_t d,
                                  ddp_notify_t n);
          ddp_recv_t ddp_recv(socket_t s);
          bdesc_t    ddp_register(socket_t s, buffer_t b);
          void       ddp_deregister(bhand_t bh);
          msizes_t   ddp_max_msizes(socket_t s);


     ddp_addr_t

          the buffer address portion of a DDP-decoration:

               typedef struct {
                 stag_t stag;
                 address_t offset;
               } ddp_addr_t;


     stag_t (unsigned integer)

          a steering tag.  A stag_t identifies the destination buffer
          for DDP-decorated messages.  stag_ts are generated when the
          buffer is registered, communicated to the sender by some
          client protocol convention and inserted in DDP-decorated



Bailey                      Expires July 2002                   [Page 5]

Internet-Draft           DDP & RDMA Architecture         4 February 2002


          messages.  stag_t values in this DDP architecture are assumed
          to be completely opaque to the client protocol, and
          implementation-dependent.  However, particular
          implementations, such as DDP on a multicast transport (see
          below), may provide the buffer holder some control in
          selecting stag_ts.

     ddp_notify_t

          the notification portion of a DDP-decoration:

               typedef struct {
                 bool notify;
                 ddp_msg_id_t i;
               } ddp_notify_t;


     ddp_msg_id_t (unsigned integer)

          a DDP-decorated message identifier.  msg_id_ts are chosen by
          the DDP-decorated message receiver (buffer holder),
          communicated to the sender by some client protocol convention
          and inserted in DDP-decorated messages.  Whether a message
          reception indication is requested for a DDP-decorated message
          is a matter of client protocol convention.  Unlike stag_ts,
          the structure of msg_id_ts is opaque to DDP, and therefore,
          completely in the hands of the client protocol.

     bdesc_t

          a description of a registered buffer:

               typedef struct {
                 bhand_t bh;
                 ddp_addr_t a;
               } bdesc_t;

          `a.offset' is the starting offset of the registered buffer,
          which may have no relationship to the `start' or `end'
          addresses of that buffer.  However, particular implemenations,
          such as DDP on a multicast transport (see below), may allow
          some client protocol control over the starting offset.

     bhand_t

          an opaque buffer handle used to unregister a buffer.

     ddp_recv_t



Bailey                      Expires July 2002                   [Page 6]

Internet-Draft           DDP & RDMA Architecture         4 February 2002


          an undecorated message, a DDP-decorated message reception
          indication, or a DDP-decorated message reception error:

               typedef union {
                 message_t m;
                 ddp_msg_id_t i;
                 ddp_err_t e;
               } ddp_recv_t;


     ddp_err_t

          indicates an error while receiving a DDP-decorated message,
          typically `offset' out of bounds, or `stag' is not registered
          to the socket.

     msizes_t

          The maximum undecorated and DDP-decorated messages that fit in
          a single transport message:

               typedef struct {
                 msize_t max_undec;
                 msize_t max_dec;
               } msizes_t;


     ddp_send(socket_t s, message_t m)

          send an undecorated message.

     ddp_send_ddp(socket_t s, message_t m, ddp_addr_t d, ddp_notify_t n)

          send a DDP-decorated message.

     ddp_recv(socket_t s)

          get the next received undecorated message, DDP-decorated
          message reception indication, or DDP-decorated message error.

     ddp_register(socket_t s, buffer_t b)

          register a buffer for DDP on a socket.  The same buffer may be
          registered multiple times on the same or different sockets.
          Different buffers may also refer to portions of the same
          underlying addressable object (buffer aliasing).

     ddp_deregister(bhand_t bh)



Bailey                      Expires July 2002                   [Page 7]

Internet-Draft           DDP & RDMA Architecture         4 February 2002


          unregister a buffer from a socket.

     ddp_max_msizes(socket_t s)

          get the current maximum undecorated and DDP-decorated message
          sizes that will fit in a single transport message.

2.3.  Transport Characterstics In DDP

     Certain characteristics of the transport on which DDP is mapped
     determine the nature of the service provided to client protocols.
     Specifically, transports are:

     o    reliable or unreliable,

     o    ordered or unordered,

     o    single source or multisource,

     o    single destination or multidestination (multicast or anycast).

     Some transports support several combinations of these
     characteristics.  For example, SCTP is reliable, single source,
     single destination (point-to-point) and supports both ordered and
     unordered modes.

     In general, these transport characteristics equally affect
     transport and DDP-decorated message delivery.  However, there are
     several issues specific to DDP-decorated messages.

     A key component of DDP, is how operations on the receiving side:

     o    set()s,

     o    undecorated messages, and

     o    DDP-decorated message reception indications

     are ordered among themselves, and how they relate to corresponding
     operations on the sending side.  These relationships depend upon
     the characteristics of the underlying transport in a way which is
     defined by the DDP protocol.  For example, if the transport is
     unreliable and unordered, the DDP protocol might specify that the
     client protocol is subject to the consequences of transport
     messages being lost or duplicated, rather requiring different
     characteristics be presented to the client protocol.

     Multidestination data delivery is the other transport



Bailey                      Expires July 2002                   [Page 8]

Internet-Draft           DDP & RDMA Architecture         4 February 2002


     characteristic which may require specific consideration in a DDP
     protocol.  As mentioned above, the basic DDP model assumes that
     buffer address values returned by ddp_register() are opaque to the
     client protocol, and can be implementation dependent.  The most
     natural way to map DDP to a multidestination transport is to
     require all receivers produce the same buffer address when
     registering a multidestination destination buffer.  Restriction of
     the DDP model to accomodate multiple destinations involves
     engineering tradeoffs comparable to those of providing non-DDP
     multidestination transport capability.

3.  Remote Direct Memory Access (RDMA) Protocol Architecture

     Remote Direct Memory Access (RDMA) extends the capabilities of DDP
     with the ability to read from buffers registered to a socket (RDMA
     Read).  This allows a client protocol to perform arbitrary,
     bidirectional data movement without involving the remote client
     protocol.  When RDMA is implemented in the NI, arbitrary data
     movement can be performed without involving the remote host CPU at
     all.

     In addition, RDMA protocols usually specify a transport-independent
     undecorated message service (Send) with characteristics which are
     both very efficient to implement in an NI, and convenient for
     client protocols.

     The RDMA architecture is patterned after the traditional model for
     device programming, where the client requests an operation using
     Send-like actions (programmed I/O), the server performs the
     necessary data transfers for the operation (DMA reads and writes),
     and notifies the client of completion.  The programmed I/O+DMA
     model efficiently supports a high degree of concurrency and
     flexibility for both the client and server, even when operations
     have a wide range of intrinsic latencies.

     RDMA is implemented as a client protocol on top of DDP:















Bailey                      Expires July 2002                   [Page 9]

Internet-Draft           DDP & RDMA Architecture         4 February 2002


                        Client Protocol
                             |  ^
                       Sends |  | Sends
          RDMA Read Requests |  | RDMA Read Completion indications
                 RDMA Writes v  | RDMA Write Completion indications
                             RDMA
                             |  ^
        undecorated messages |  | undecorated messages
      DDP-decorated messages |  | DDP-decorated message reception
                             v  |          indications
                             DDP
                              ^
                              | transport messages
                              v
                            . . .

     In addition to in-line data flow, read (get()) and update (set())
     operations are performed on buffers registered with RDMA as a
     result of RDMA Read Requests and RDMA Writes, respectively.

     An RDMA `buffer' extends a DDP buffer with a get() operation that
     retrieves the value of the octet at address `a':

          typedef struct {
            const address_t start;
            const address_t end;
            void            set(address_t a, uint8_t v);
            uint8_t         get(address_t a);
          } buffer_t;



3.1.  RDMA Operations

     The RDMA layer provides:

          void        rdma_send(socket_t s, message_t m);
          void        rdma_write(socket_t s, message_t m, ddp_addr_t d,
                                 rdma_notify_t n);
          void        rdma_read(socket_t s, ddp_addr_t s, ddp_addr_t d);
          rdma_recv_t rdma_recv(socket_t s);
          bdesc_t     rdma_register(socket_t s, buffer_t b, bmode_t mode);
          void        rdma_deregister(bhand_t bh);
          msizes_t    rdma_max_msizes(socket_t s);


     Although, for clarity, these data transfer interfaces are
     synchronous, rdma_read() and possibly rdma_send() (in the presence



Bailey                      Expires July 2002                  [Page 10]

Internet-Draft           DDP & RDMA Architecture         4 February 2002


     of Send flow control), can require an arbitrary amount of time to
     complete.  To express the full concurrency and interleaving of RDMA
     data transfer, these interfaces are also defined to be
     multithreaded.  For example, a client protocol may perform an
     rdma_send(), while an rdma_read() operation is in progress.

     rdma_notify_t

          RDMA Write notification information:

               typedef struct {
                 bool notify;
                 rdma_write_id_t i;
               } rdma_notify_t;

          identical in function to ddp_notify_t, except that the type
          rdma_write_id_t may not be equivalent to ddp_msg_id_t.

     rdma_write_id_t (unsigned integer)

          an RDMA Write identifier.

     rdma_recv_t

          a Send message, an RDMA Write completion identifier, or an
          RDMA error:

               typedef union {
                 message_t m;
                 rdma_write_id_t i;
                 rdma_err_t e;
               } rdma_recv_t;


     rdma_err_t

          an RDMA protocol error indication.  RDMA errors include buffer
          addressing errors corresponding to ddp_err_ts, and buffer
          protection violations (e.g. RDMA Writing a buffer only
          registered for reading).

     bmode_t

          buffer registration mode (permissions).  Any combination of
          permitting RDMA Read (BMODE_READ) and RDMA Write (BMODE_WRITE)
          operations.

     rdma_send(socket_t s, message_t m)



Bailey                      Expires July 2002                  [Page 11]

Internet-Draft           DDP & RDMA Architecture         4 February 2002


          Send a message.

     rdma_write(socket_t s, message_t m, ddp_addr_t d, rdma_notify_t n)

          RDMA Write to remote buffer address d.

     rdma_read(socket_t s, ddp_addr_t s, ddp_addr_t d)

          RDMA Read from remote buffer address s to local buffer address
          d.

     rdma_recv(socket_t s);

          get the next received Send message, RDMA Write completion
          identifier, or RDMA error.

     rdma_register(socket_t s, buffer_t b, bmode_t mode)

          register a buffer for RDMA on a socket (for read access, write
          access or both).  As with DDP, the same buffer may be
          registered multiple times on the same or different sockets,
          and different buffers may refer to portions of the same
          underlying addressable object.

     rdma_deregister(bhand_t bh)

          unregister a buffer from a socket.

     rdma_max_msizes(socket_t s)

          get the current maximum Send (max_undec) and RDMA Read or
          Write (max_dec) operations that will fit in a single transport
          message.  The values returned by rdma_max_msizes() are closely
          related to the values returned by ddp_max_msizes(), but may
          not be equal.

3.2.  Transport Characterstics In RDMA

     As with DDP, RDMA can be used on transports with a variety of
     different characteristics that manifest themselves directly in the
     service provided by RDMA.

     Like DDP, an RDMA protocol must specify how:

     o    set()s,

     o    get()s,




Bailey                      Expires July 2002                  [Page 12]

Internet-Draft           DDP & RDMA Architecture         4 February 2002


     o    Send messages, and

     o    RDMA Read completions

     are ordered among themselves and how they relate to corresponding
     operations on the remote peer(s).  These relationships are likely
     to be a function of the underlying transport characteristics.

     There are some additional characteristics of RDMA which may
     translate poorly to unreliable or multipoint transports due to
     attendent complexities in managing endpoint state:

     o    Send flow control

     o    RDMA Read

     These difficulties can be overcome by placing restrictions on the
     service provided by RDMA.  However, many RDMA clients, especially
     those that separate data transfer and application logic concerns,
     are likely to depend upon capabilities only provided by RDMA on a
     point-to-point, reliable transport.


4.  Security Considerations

     Security considerations are not addressed in this document.  Any
     security considerations resulting from the use of DDP or RDMA must
     be addressed in the relevant standards.

5.  IANA Considerations

     IANA considerations are not addressed in by this document.  Any
     IANA considerations resulting from the use of DDP or DMA must be
     addressed in the relevant standards.

Author's Address


     Stephen Bailey
     Sandburst Corporation
     600 Federal Street
     Andover, MA  01810
     USA

     Email: steph@sandburst.com






Bailey                      Expires July 2002                  [Page 13]

Internet-Draft           DDP & RDMA Architecture         4 February 2002


Full Copyright Statement

     Copyright (C) The Internet Society (2001). All Rights Reserved.

     This document and translations of it may be copied and furnished to
     others, and derivative works that comment on or otherwise explain
     it or assist in its implementation may be prepared, copied,
     published and distributed, in whole or in part, without restriction
     of any kind, provided that the above copyright notice and this
     paragraph are included on all such copies and derivative works.
     However, this document itself may not be modified in any way, such
     as by removing the copyright notice or references to the Internet
     Society or other Internet organizations, except as needed for the
     purpose of developing Internet standards in which case the
     procedures for copyrights defined in the Internet Standards process
     must be followed, or as required to translate it into languages
     other than English.

     The limited permissions granted above are perpetual and will not be
     revoked by the Internet Society or its successors or assigns.

     This document and the information contained herein is provided on
     an "AS IS" basis and THE INTERNET SOCIETY AND THE INTERNET
     ENGINEERING TASK FORCE DISCLAIMS ALL WARRANTIES, EXPRESS OR
     IMPLIED, INCLUDING BUT NOT LIMITED TO ANY WARRANTY THAT THE USE OF
     THE INFORMATION HEREIN WILL NOT INFRINGE ANY RIGHTS OR ANY IMPLIED
     WARRANTIES OF MERCHANTABILITY OR FITNESS FOR A PARTICULAR PURPOSE.
























Bailey                      Expires July 2002                  [Page 14]