Internet DRAFT - draft-haagens-ips-iscsireqs


INTERNET DRAFT                                          Randy Haagens
                                                  Hewlett-Packard Co.
Expires January 2001                                        July 2000

                   iSCSI (Internet SCSI) Requirements

Status of this Memo

     This document is an Internet-Draft and is in full conformance with
     all provisions of Section 10 of RFC2026.  Internet-Drafts are work-
     ing documents of the Internet Engineering Task Force (IETF), its
     areas, and its working groups. Note that other groups may also dis-
     tribute working documents as Internet-Drafts.  Internet-Drafts are
     draft documents valid for a maximum of six months and may be
     updated, replaced, or obsoleted by other documents at any time. It
     is inappropriate to use Internet-Drafts as reference material or to
     cite them other than as "work in progress."

     The list of current Internet-Drafts can be accessed at The list of Internet-
     Draft Shadow Directories can be accessed at


     Comments should be sent to the ips mailing list (
     or to the author(s).


     This document explains the motivation behind an efficient transport
     of SCSI commands on top of TCP/IP and describes scenarios where
     such a transport will be used. The document also enumerates and
     discusses requirements for supporting SCSI on top of IP.


     We propose to define a mapping of SCSI protocol to TCP/IP so that
     SCSI storage controllers (principally disk and tape arrays and
     libraries) can be attached to IP networks, notably Gigabit Ethernet
     (GbE) and 10 Gigabit Ethernet (10 GbE).

Randy Haagens                                                   [Page 1]
Internet-Draft              TCP RDMA option                 July 7, 2000


     We seek timely adoption of a protocol mapping for block storage
     over IP networks.  Accordingly, we have chosen to work with the
     existing SCSI architecture and commands and also the existing
     TCP/IP transport layer.  Both these protocols are widely-deployed
     and well-understood.  Using them means a minimum of new invention,
     the most rapid possible adoption, and the greatest compatibility
     with Internet architecture, protocols, and equipment.

     The iSCSI protocol is a mapping of SCSI to TCP, and constitutes a
     "SCSI transport" as defined by the SCSI SAM-2 document [SAM2, p. 3,
     "Transport Protocols"].

1 Applicability

     Traditionally, volume/block-oriented storage controllers (e.g.,
     disk array controllers, tape library controllers) have supported
     the SCSI-3 protocol, and have been attached to computers through
     the SCSI parallel bus or through Fibre Channel.  File-oriented
     storage controllers have supported the NFS and/or CIFS protocols,
     and have been attached directly to IP networks such as Ethernet.

     The IP/Ethernet infrastructure offers compelling advantages for
     volume/block-oriented storage attachment compared to current

          * Increasing performance and reduced cost driven by Internet
          economics and "IP convergence"

          * Seamless conversion from local to wide area using IP routers

          * Emerging availability of "IP datatone" service from car-
          riers, in preference to ATM or SONET or T-1, T-3 services

          * Protocols and middleware for management, security and QoS

          * Economies arising from the need to install and operate only
          single type of network

     The following applications for iSCSI are contemplated:

          * Local storage access, consolidation, clustering and pooling
          (as in the data center)

          * Remote disk access (as for a storage utility)

Randy Haagens                                                   [Page 2]
Internet-Draft              TCP RDMA option                 July 7, 2000

          * Local and remote synchronous and asynchronous mirroring
          between storage controllers

          * Local and remote backup and restore

          * Evolution with SCSI to support of emerging object-oriented
          storage model

     And the following connection topologies are contemplated:

          * Point-to-point direct connections

          * Dedicated storage LAN, consisting of one or more LAN seg-

          * Shared LAN, carrying a mix of traditional LAN traffic plus
          storage traffic

          * LAN-to-WAN extension using IP routers or carrier-provided
          "IP Datatone"

          * Private networks and the public Internet

     The iSCSI standard will permit SCSI volume/block-oriented devices
     to be attached directly to IP networks such as Ethernet.  The
     SCSI-3 command sets (defined by the ANSI NCITS T10 committee) will
     be mapped to TCP.  iSCSI is this mapping, and is analogous to (but
     not the same as) SCSI-FCP (aka "FCP"), which is the mapping of SCSI
     to Fibre Channel.

     Local-area storage networks will be built using Ethernet LAN
     switches.  These networks may be dedicated to storage, or shared
     with traditional Ethernet uses, as determined by cost, performance,
     administration, and security considerations.  In the local area,
     TCP's adaptive retransmission timers will provide for automatic and
     rapid error detection and recovery, compared to alternative techno-

     IP LAN-WAN routers will be used to extend the IP storage network to
     the wide area, permitting remote disk access (as for a storage
     utility), synchronous and asynchronous remote mirroring, and remote
     backup and restore (as for tape vaulting).  In the WAN, TCP end-
     to-end will avoid the need for specialized equipment for protocol
     conversion, ensure data reliability, cope with network congestion,
     and automatically adapt retransmission strategies to WAN delays.

     The full realization of iSCSI will involve the following elements:
     (1) Completion of  Requirements (this document) and Specification

Randy Haagens                                                   [Page 3]
Internet-Draft              TCP RDMA option                 July 7, 2000

     documents; (2) Development of Ethernet storage NICs and related
     driver and protocol software; (3) Development of compatible storage
     controllers; and (4) The likely development of translating gateways
     to provide connectivity between the Ethernet storage network and
     the Fibre Channel and/or parallel-bus SCSI domains.

     Products will initially be offered for Gigabit Ethernet attachment,
     with rapid migration to 10 GbE.  For performance competitive with
     alternative SCSI transports, it will be necessary to implement the
     performance path of the full protocol stack in hardware.  These new
     storage NICs will perform full-stack processing of a complete SCSI
     task, analogous to today's SCSI and Fibre Channel HBAs.  They typi-
     cally also will support all host protocols that use TCP, including
     NFS, CIFS and HTTP.

     A key goal is not to require modifications to the current IP and
     Ethernet infrastructure to support storage traffic over TCP.
     Nevertheless, the performance and security requirements of storage
     will create opportunities for improvement in security protocols and
     QoS implementations.  The addition of storage traffic to local- and
     wide-area internets (and even to the public Internet) may introduce
     increased requirements for traffic monitoring and engineering in
     those environments.

     It is contemplated that many organizations initially will choose to
     operate storage networks based on iSCSI that are independent of
     (isolated from) their current data networks except for secure rout-
     ing of storage management traffic.  These organizations will bene-
     fit from the high performance/cost of IP equipment and a unified
     management architecture, compared to alternative means of building
     storage networks.  As security and QoS evolve, it will become more
     reasonable to build combined networks with shared infrastructure;
     nevertheless, it is likely that sophisticated users will choose to
     keep their storage subnetworks isolated, for the best control of
     security and QoS.

     The proposed charter of the IETF IP SCSI Working Group (IPSWG)
     describes the broad goal of mapping SCSI to IP.  Within that broad
     charter, many transport alternatives may be considered.  Our ini-
     tial work focuses on TCP, and this Requirements document is res-
     tricted to that domain of interest.  At the current time, we do not
     seek a more generic requirements statement that would justify the
     choice of TCP (or another protocol) as transport, since the merits
     of using TCP are readily evident to the working group participants.

Randy Haagens                                                   [Page 4]
Internet-Draft              TCP RDMA option                 July 7, 2000

2 Definitions

     Certain definitions are offered here, with references to the origi-
     nal document where applicable, in order to clarify the discussion
     of requirements.  Throughout the text, use of defined terms is
     emphasized by producing them in bold face type.  Definitions
     without references are the work of the authors and reviewers of
     this document.

     Logical Unit (LU): A target-resident entity that implements a dev-
     ice model and executes SCSI commands sent by an application client
     [SAM-2, section 3.1.50, p. 7].

     Logical Unit Number (LUN): A 64-bit identifier for a logical unit
     [SAM-2, section 3.1.52, p. 7].

     SCSI Device:  A device that is connected to a service delivery sub-
     system and supports an SCSI application protocol [SAM-2, section
     3.1.78, p. 9].

     Service Delivery Port (SDP): A device-resident interface used by
     the application client, device server, or task manager to enter and
     retrieve requests and responses from the service delivery subsys-
     tem.  Synonymous with port (SAM-2 section 3.1.61) [SAM-2, section
     3.1.89, p. 9].

     Target: An SCSI device that receives SCSI command and directs such
     commands to one or more logical units for execution [SAM-2 section
     3.1.97, p. 10].

     Task: An object within the logical unit representing the work asso-
     ciated with a command or a group of linked commands [SAM-2, section
     3.1.98, p. 10].

     Transaction: A cooperative interaction between two objects, involv-
     ing the exchange of information or the execution of some service by
     one object on behalf of the other [SAM-2, section 3.1.109, p. 10].
     [A transaction seems to be a smaller unit than a task.]

3 Requirements

     In the attached, actual requirements statements are flagged with
     [R].  Related discussion is flagged with [D].

     The requirements are somewhat arbitrarily grouped into categories.
     This is for convenience only.  No semantic meaning is to be implied
     from the category names.

Randy Haagens                                                   [Page 5]
Internet-Draft              TCP RDMA option                 July 7, 2000

3.1 General

     [R] Support block storage IO over IP networks.

          [D] Our initial approach uses SCSI for the block storage pro-
          tocol, and TCP/IP for the network transport.

     [R] Minimize optional features; but when allowed, (1) Allow for
     option negotiation at session establishment (login); (2) Provide
     for signaling an error (reject) when an unsupported feature is

3.2 Performance/Cost2

     In general, iSCSI must allow implementations to equal or improve on
     the current state of the art for SCSI interconnects.

     [R] Low delay communication.

          [D] Conventional storage access is of a stop-and-wait or
          remote procedure call type.  Applications typically employ
          very little pipelining of their storage accesses, and so
          storage access delay directly impacts performance.  The delay
          imposed by current storage interconnects, including protocol
          processing, is generally in the range of 100 microseconds.
          The use of caching in storage controllers means that many
          storage accesses complete almost instantly, and so the delay
          of the interconnect can have a high relative impact on overall

     [R] High bandwidth, bandwidth aggregation.

          [D] The bandwidth (transfer rate, MB/sec) supported by storage
          controllers is rapidly increasing, due to several factors: (1)
          Increase in disk spindle and controller performance; (2) Use
          of ever-larger caches, and improved caching algorithms; (3)
          Increased scale of storage controllers (number of supported
          spindles, speed of interconnects).  Not only must the iSCSI
          provide for full utilization of available link bandwidth, it
          also must exploit parallelism (multiple connections) at the
          device interfaces and within the interconnect fabric.

     [R] Low CPU utilization, equal to or better than current technol-

          [D] For competitive performance, the iSCSI protocol must allow
          three key implementation choices to be realized: (1) iSCSI

Randy Haagens                                                   [Page 6]
Internet-Draft              TCP RDMA option                 July 7, 2000

          must make it possible to build I/O adapters that handle an
          entire SCSI task, as alternative SCSI transport implementa-
          tions do.  (2) The protocol must permit "zero-copy" memory
          architectures, where the I/O adapter reads or writes host
          memory exactly once per disk transaction. (3) The protocol
          must not impose complex operations on the host software, which
          would increase host instruction path length relative to alter-

     [R] Cost competitive with alternative storage network technologies.

3.3 SCSI

     [R] Collaboration with ANSI NCITS T10 (SCSI)

          [D] iSCSI is a new SCSI "transport" [SAM2].  Being the inter-
          section of SCSI and TCP, iSCSI has potential impact on T10 as
          well as on IETF.  However, a stated requirement (below) is
          that iSCSI shall have no impact on T10 architecture or command
          sets.  Collaboration with T10 will be necessary to achieve
          this requirement.

          [D] Collaboration with T10 concerns three phases of T10
          activity: (1) Past.  For T10 work completed in the past, and
          well-document in T10 standards publication, we will seek
          assistance in properly interpreting those standards; (2)
          Present.  For T10 work that is ongoing, or recently completed
          (but not widely published), we will seek review of our work by
          individuals active in T10, and/or the participation of those
          individuals in the IETF process; (3) Future.  For compatibil-
          ity with future T10 work, it is essential that iSCSI be a leg-
          itimate and recognized "SCSI transport", no less so than the
          several other SCSI transports.  SCSI command standards must
          evolve within the context of all existing SCSI transports.

          [D] Storage attachment to IP networks will engender an unpre-
          cedented potential for device sharing.  This alone may impact
          future T10 work.

     [R] Supported SCSI Device types.  iSCSI shall support all SCSI dev-
     ice types.  Our primary focus is on supporting "larger" devices:
     host computers and storage controllers (disk arrays, tape library

          [D] Supported SCSI Devices will typically have adequate memory
          to implement the TCP transport and required iSCSI session
          state, and a cost structure that can support VLSI for full-

Randy Haagens                                                   [Page 7]
Internet-Draft              TCP RDMA option                 July 7, 2000

          stack protocol acceleration. Generally, a controller will be
          interposed between the iSCSI (typically Ethernet) connections
          and the drive interface (typically parallel SCSI or Fibre
          Channel).  In the longer term, it will become feasible, due to
          the march of technology, to support iSCSI economically in disk
          spindle and tape mechanism controllers.

     [R] Support SCSI SAM-2 architecture model.

          [D] It would be helpful to produce a document discussing iSCSI
          with reference to SAM-2.  No promises.

     [R] Reliable Transport.  The iSCSI mapping provides the SCSI-3 com-
     mand layer with a reliable transport, equal to or greater in relia-
     bility than the parallel SCSI bus, and providing in-order delivery,
     as suggested by SAM-2.

          [D] See [SAM-2, p. 17.] "The function of the service delivery
          subsystem is to transport an error-free copy of the request or
          response between the sender and the receiver..." [SAM-2, p.
          22] "The manner in which ordering constraints are established
          is implementation-specific.  An implementation may choose to
          delegate this the service delivery port.
          In some cases, in-order delivery may be an intrinsic property
          of the transport subsystem or a requirement established by the
          SCSI protocol standard.  For convenience, the SCSI architec-
          ture model assumes in-order delivery to be a property of the
          service delivery subsystem.  This assumption is made to sim-
          plify the description of behavior and does not constitute a

     [R] Support for SCSI Task Queuing.

          [D] SAM-2 defines task queuing, and so strictly speaking, we
          don't need to call this out specifically.  However, task queu-
          ing is not widely implemented today; and it will increase in
          importance with WAN IP networks, given speed-of-light delays.
          We are particularly interested in supporting task queuing of
          pipelined remote backup and asynchronous disk mirroring

          [D] Just because iSCSI supports task queuing doesn't mean that
          the end SCSI node is required to do so also.  Task queuing is
          an optional feature of SCSI.

     [R] Supports all SCSI-3 command sets [SPC-2, SBC, etc.].  There
     will be no requirement by T10 to modify the SCSI command documents.
     No modifications are required of the SCSI command layer implementa-
     tion, except possibly to lengthen task timers to accommodate wide-

Randy Haagens                                                   [Page 8]
Internet-Draft              TCP RDMA option                 July 7, 2000

     area delays due to speed-of-light and switching.

          [D] Note the restriction to SCSI-3 command sets.  There are
          potential problems with gateways between iSCSI and SCSI-2
          parallel bus devices.  It may not be feasible to transport
          SCSI-2 commands over iSCSI.  Gateways that wish to support
          older SCSI-2 devices may have to proxy for those devices,
          using SCSI-3 commands.

     [R] Forward compatibility with future revisions of SCSI architec-
     ture and protocol.  Attention to clean layering of protocols.

          [D] This is a difficult requirement to achieve in practice,
          since we cannot predict how SCSI will evolve.  However, care-
          ful attention to protocol layering principles will help ensure
          this result.

     [R] Gateways to parallel SCSI [SPI-X] and to SCSI-FCP[FCP, FCP-2].
     It will be possible to construct "translating" gateways so that
     iSCSI hosts can talk to SCSI-X devices; so that SCSI-X devices can
     talk to each other over a iSCSI network; and so that SCSI-X hosts
     can talk to iSCSI devices (where SCSI-X refers to parallel SCSI,
     SCSI-FCP, or SCSI over any other transport).

          [D] This requirement is implied by support for SAM-2, but is
          worthy of emphasis.

          [D] These are true application protocol gateways, and not just
          bridge/routers.  The different standards have only the SCSI-3
          command set layer in common.  These gateways are not mere
          packet forwarders.  We need to look into their remote proxy

          [D] Adequate liaison must be established with related stan-
          dards bodies, principally ANSI T10 (SCSI).

3.4 iSCSI Session Layer

     [R] SCSI command, data, and response transactions occur in a TCP
     connection that is determined by the initiator, in advance of
     starting the SCSI task.

          [D] This requirement allows the initiator to assign the data
          transfer phase of a task to a given data transfer engine, at
          initiation of the task.

     [R?] TCP connection allegiance.  SCSI commands, data and status

Randy Haagens                                                   [Page 9]
Internet-Draft              TCP RDMA option                 July 7, 2000

     information for a given task shall flow within the same single TCP

          [D] This is a stronger statement than the one above, and is
          left here as a potential requirement, mostly so that it will
          be clear that the discussion topics below pertain to the
          notion of channel allegiance.

          [D] SAM-2 seems to require this channel allegiance: "A task
          involving one initiator-target pair shall not specify a third
          SCSI device to participate in transmitting and receiving the
          remote procedure model elements for that task.  Thus, an SMU
          initiator [e.g., a host computer] shall not create a task
          using one service delivery port with the expectation that the
          data transfer or status return for that task would occur via a
          different service delivery port" [SAM-2, section  4.10.7,
          p.33].  Of course, interpretation of this clause depends on
          the definition of service delivery port.  If a service
          delivery port is a TCP connection, then channel allegiance is
          pretty clearly required.  But if a service delivery port is an
          iSCSI session or an abstract target device, then the interpre-
          tation of this clause is less clear.

          [D] We have found a number of other possible virtues in chan-
          nel allegiance: (1) It supports multiple instances of the TCP
          protocol engine being controlled by a single iSCSI session
          layer; (2) Failure of a TCP connection will affect only a sub-
          set of the extant tasks (those that use the failed connec-
          tion); (3) All TCP connections are used in exactly the same
          manner; (4) There is no need to have more than one IP port
          defined for the iSCSI protocol, which is firewall-friendly.

     [R] Command striping (load balancing) across multiple host and dev-
     ice interfaces.  It shall be possible to utilize multiple con-
     current paths between hosts and devices for the purpose of load

          [D] Load balancing refers to concurrent tasks from a single
          initiator.  There is no ordering constraint among these tasks.
          We aim to distribute these tasks (commands and their related
          data and status) across multiple host ports, links, switch
          ports and device ports, in order to achieve aggregate perfor-
          mance equal to a multiple of single link performance.

     [R] Command ordering for tape backup and asynchronous remote mir-
     roring.  It must be possible to pipeline commands to a device, and
     to have them executed in order by that device, as prescribed by

Randy Haagens                                                  [Page 10]
Internet-Draft              TCP RDMA option                 July 7, 2000

          [D] Ordering can be maintained by allowing each command to
          complete before issuing the next.  But that means there is no
          pipelining.  For tape backup in the local area, this may be
          adequate, as the tape controller buffer can be made suffi-
          ciently large to cover the lower duty cycle of data transfers,
          and LAN speeds are fast enough to burst-fill the buffer.  But
          in the wide area, a method of pipelining commands and
          responses is needed if the slower WAN link is to be filled
          continuously with data.

          [D] This brings up an issue, if commands are sent in different
          TCP connections.  Although a single TCP connection delivers an
          ordered byte stream, there is no ordering constraint between
          TCP connections.  So command striping across TCP connections
          will result in the commands possibly being executed out of
          order, unless the commands themselves are numbered, and can be
          put back into order.  SCSI does not provide a means for put-
          ting commands back in order, but requires that functionality
          of the "transport".

          [D] We contemplate bonding multiple TCP connections into an
          iSCSI session for the purpose of ordered command striping.  A
          command reference number (CRN) will allow iSCSI to receive
          commands in order from the initiator SCSI command layer, and
          deliver them in order to its peer command layer in the target.
          Note that this mechanism can be employed at all times, because
          delivering commands in order never hurts, even if the SCSI
          layer imposes no ordering constraints among them.  This is the
          safest route, in fact, as it upholds the SAM-2 expectation of
          in-order delivery.  We expect the ability to support a session
          consisting of multiple channels to be optional.

     [R] Recovery at the session layer.  The session layer specification
     shall explicitly address recovery at the session layer (from a
     failed TCP connection, for example).

          [D] TCP will recover from data loss due to bit errors or
          congestion.  But what if a TCP connection fails (hangs)?  The
          specification needs to address this issue.

          [D] Another case that we should consider is loss of session
          state at either the target or the initiator, for example, when
          a target is power cycled.  Should it be possible to restore
          the session in this case, or will we have to report service
          delivery failure to the SCSI layer, for recovery at that
          level?  In the case of a recovered session, we're concerned
          about "ghost IOs" that may inappropriately linger from a pre-
          vious session.

Randy Haagens                                                  [Page 11]
Internet-Draft              TCP RDMA option                 July 7, 2000

3.5 Transport, Network and Link

     [R] Works with existing installed Ethernet and IP WAN infrastruc-
     ture.  iSCSI should not require any modification to Ethernet hubs,
     switches or WAN routers to achieve minimum acceptable performance,
     QoS and security.

          [D] Using existing and off-the-shelf technology will allow
          iSCSI to fully leverage the cost, performance and rapid
          improvement of widely-deployed IP LAN and WAN technologies.
          Therefore, iSCSI cannot require the installation of special,
          non-standard features in the underlying technology.  However,
          it may be desirable to apply certain optimizations that will
          enhance storage protocol performance, or the performance of
          other protocols in the presence of the storage protocol.

     [R] Joint operation (coexistence) with other IP protocols.  iSCSI
     shall not preclude concurrent operation with any of the protocols
     in the IP protocol suite, and shall be a good Internet citizen.

          [D] Many organizations will choose to operate iSCSI storage
          networks as separate networks from their traditional data net-
          works, by a router only for management traffic.  This approach
          delivers the most manageable environment from a performance
          and security perspective, and is analogous to today's separate
          Fibre Channel storage networks, except for the obvious bene-
          fits that derive from using LAN technologies.  On the other
          hand, some organizations will favor using fewer networks, and
          mixing storage with other types of traffic.  This practice
          will be more prevalent in the wide-area, where dedicated
          storage links exact a high price.  For these reasons, graceful
          co-existence is required.  Over time, improved support for the
          QoS and security features inherent in IP and Ethernet proto-
          cols will make it more and more reasonable to combine storage
          with other types of network traffic.

          [D] When storage is transported over the wider Internet, it
          must be done in a way that respects TCP's bandwidth management
          and congestion avoidance algorithms.  This is one of the rea-
          sons for selecting TCP as the transport.  We feel that TCP
          itself is a good Internet citizen, and our best chance for

     [R] Uses TCP/IP.  iSCSI is a protocol mapping from SCSI to TCP.

          [D] While we don't preclude consideration of alternative tran-
          sports, we have focused our attention on TCP. Given wide-area
          functions in a storage controller, and the resulting need for

Randy Haagens                                                  [Page 12]
Internet-Draft              TCP RDMA option                 July 7, 2000

          TCP support, inclusion of an alternative local-area transport
          may imply an increment of cost, not a cost savings; and it
          certainly represents an increment of complexity.

     [R] Link Independent.  iSCSI is defined for all IP networks, and is
     link-independent.  All IP-compatible LAN and WAN links are sup-
     ported.  Specifically, there are no dependencies on Ethernet.

          [D] We may nevertheless want to benefit from certain link
          capabilities like Ethernet port aggregation and PPP multi-
          link.  But the spec should not depend on these capabilities
          for its viability.

     [R] LAN, MAN and WAN -capable.  SCSI Devices that implement iSCSI
     will be capable of communicating with similarly-equipped devices
     and host computers over any IP network, whether local, metropoli-
     tan, or wide-area in scale.

          [D] iSCSI is used not only for local area disk block access
          and tape operations.  It also is used for remote disk access
          (as for a storage utility), remote disk mirroring, and remote
          backup and restore (as for tape vaulting).  Using TCP in the
          iSCSI end nodes means that the protocol is scalable from the
          local to the wide area.

     [R] Handles high bandwidth x delay fabrics.

          [D] This requirement must be clarified further, as an exten-
          sion of the WAN requirement.  Consider that the TCP pipe at 10
          Gbps x 200 msec holds 250 megabytes.  Will TCP sequence counts
          be up to this, or will they wrap too frequently?

     [R] Recovery of data stream processing immediately after TCP seg-
     ment drop.

          [D] In a conventional TCP implementation, loss of a TCP seg-
          ment means that stream processing must stop until that segment
          is recovered, which takes a network round trip to accomplish.
          Following the example above, we would be obliged to catch 250
          MB of data into an anonymous buffer before we could resume
          stream processing; later, this data would need to be moved to
          its proper location.  We seek some means of putting data
          directly where it belongs, and avoiding extra data movement in
          the case of segment drop.

          [D] Two possibilities are known at this time: (1) A Remote DMA
          feature added to TCP headers (in the options field) would
          allow the data portion of subsequent TCP segments to be placed

Randy Haagens                                                  [Page 13]
Internet-Draft              TCP RDMA option                 July 7, 2000

          directly, even though the iSCSI protocol headers have not been
          parsed; (2) A means of recovering iSCSI framing is the TCP
          stream would allow iSCSI protocol processing to continue, and
          the data to be put in its proper location.

     [R?] Framing.  Some method of framing iSCSI protocol units within
     the TCP stream may be required.

          [D] We are unresolved as to whether this is a requirement.
          The more basic requirement, described above, is to be able to
          recover the processing of the data stream immediately after a
          segment drop.  Framing is one way to recover processing.

          [D] The conventional way to locate higher-level protocol
          headers in the TCP stream is simply by parsing from the begin-
          ning of the stream, and never making a mistake.  Is this suf-
          ficient?  Or, should we use some other means such as byte
          stuffing or use of the push bit?  Related, how do we ensure
          that data actually is transmitted, and doesn't languish in a
          TCP buffer somewhere?

          [D] As an example of the problem: suppose a TCP segment is
          lost due to congestion, and it happens to contain an iSCSI
          header.  At that point, stream synchronization will be lost,
          as we cannot find the next iSCSI header.  Following the exam-
          ple above, we're obliged to catch 250 MB of data before we can
          resume iSCSI operation.  If we could find the next iSCSI
          header, we could implement an optimization (non-traditional
          for TCP implementations) that would require us only to catch a
          single iSCSI message's-worth of data.  Subsequent iSCSI mes-
          sages could be decoded, and the data put where it belongs
          (even though command ordering constraints would preclude act-
          ing upon the data until the missing SCSI command is received
          and inspected for ordering constraints).

          [D] Several methods have been discussed for providing framing
          by TCP: (1) A flag could be added in the TCP options that
          indicates that this segment begins a next-level Protocol Data
          Unit (PDU); (1a) Method 1 could be combined with a remote DMA
          mechanism for TCP; (2) The TCP transmitter function could be
          modified so that it emits a TCP segment for every next-level
          PDU, effectively turning TCP into a reliable, sequenced,
          datagram protocol.  Protocols such as iSCSI would then need to
          limit their PDUs to less than the maximum TCP segment size
          (which is dictated by link considerations), if IP fragmenta-
          tion is to be avoided.

          [D] Other methods could work above TCP.  (1) Byte stuffing is

Randy Haagens                                                  [Page 14]
Internet-Draft              TCP RDMA option                 July 7, 2000

          an old technique for framing within byte streams; its main
          disadvantage is that every byte must be processed by the fram-
          ing mechanism, which would make software implementation
          impractical; (2) A special marker header could be placed
          periodically in the TCP stream.  These headers would be found
          by doing arithmetic on TCP sequence numbers.  They contain
          information about the exact location of iSCSI PDUs.

     [R?] Error detection.  Stronger CRC.

          [D] The TCP checksum is rather weak as error detection goes.
          It is supported by the link layer check codes (CRC-32 for Eth-
          ernet).  Is that sufficient?  We don't have strong protection
          from re-assembly errors.  Routers modify the frame and recom-
          pute the CRC.  Even switches recompute CRCs when adding VLAN
          tags, although good implementations do the CRC recomputation
          incrementally.  The TCP checksum is our only end-to-end pro-
          tection.  If the TCP checksum is not sufficient, do we intro-
          duce some kind of check on the SCSI data buffers by the iSCSI
          layer?  Possibilities: byte count, CRC.  Whatever we do, it
          must be possible to compute these check codes on the fly, as
          data is transferred from NIC to memory, without making a
          second pass over the data once it is in memory.

          [D] We are considering using the IPsec messsage digest func-
          tion for this purpose.  It's already defined, and it could be
          used as a check code (only) using well-known keys; hence,
          without introducing the key distribution problem.  Using IPsec
          in conjunction with TCP would not require a modification to
          TCP.  A concern about using the IPsec message digest function
          is that it may be more difficult to compute at high speed than
          a simpler CRC.

          [D] But is TCP truly an end-to-end protocol?  The notion of an
          end-to-end error check is that it and the data it protects
          pass through the network unchanged, but possibly subject to
          errors while on a link or in a memory.  At the receiving end
          node, checking the CRC verifies the correct receipt of data.
          In some cases, such as the use of a SOCKS proxy server or
          perhaps a NAT, the connection is not end-to-end, but is the
          concatenation of two end-to-end connections.  In these cases,
          the iSCSI PDU (message) may be a better candidate for CRC pro-

          [D] When considering a CRC at the iSCSI layer, we will give
          consideration to separate CRCs for iSCSI headers and data, and
          to the need to intersperse CRCs within long data messages.

Randy Haagens                                                  [Page 15]
Internet-Draft              TCP RDMA option                 July 7, 2000

     [R] Selective TCP retransmission.

          [D] Given the long delays in the WAN, using TCP selective
          retransmission must be supported by iSCSI, in order to minim-
          ize the bandwidth impact of retransmission.

     [R] Firewall friendly.  The protocol's use of IP addressing and TCP
     port numbers should be firewall friendly.

          [D] This probably means that all connection requests should be
          addressed a specific, well-known TCP port.  That way,
          firewalls can filter based on source and destination IP
          addresses, and destination (target) port number.  The source
          (initiator) port number also should be well-known for the ini-
          tial TCP connection.  Additional TCP connections would require
          different source port numbers (for uniqueness), but could be
          opened after a security dialogue on the control channel.

     [R] Possible to move data directly from end-to-end, without having
     retransmission buffers in the middle.

          [D] This is an important implementation detail.  In an iSCSI
          system, each of the end nodes (for example host computer and
          storage controller) has ample memory; but the intervening
          nodes (NIC, switches) do not.  We contemplate a WAN-scale
          retransmission requirementof 25 MB (1 Gbps) or 250 MB (10
          Gbps, see earlier footnote).  Therefore, it must not be neces-
          sary for intervening nodes to buffer data.

     [R] Conservative in use of TCP and session-layer connections.  The
     number required should not scale directly with the number of sup-
     ported LUs.

          [D]  TCP connection and iSCSI session state is fairly expen-
          sive in terms of memory consumed both on- and off-chip (we
          contemplate VLSI implementation).  At a minimum, we seek to
          support only the number of connections required to achieve
          required bandwidth and delay characteristics between hosts and
          storage controllers.

     [R] Compatible with both IPv4 and IPv6.

          [D] We need to add a literal format for IPv6 addresses in tar-
          get domain names.

Randy Haagens                                                  [Page 16]
Internet-Draft              TCP RDMA option                 July 7, 2000

3.6 Naming

     [R] Naming.  Whenever possible, iSCSI shall support the naming
     architecture of SAM-2.  Deviations and uncertainties will be made
     explicit, and comment/resolution invited.

          [D] It may be necessary to provide a unique naming scheme for
          SCSI LUs.  Fibre Channel does so using WWNs.  There's some
          indication that the T10 Security work will complicate this
          problem through LUN renumbering.  The manner of determining a
          unique, worldwide, unchanging LU name must be determined.  We
          will attempt to make use of SPC-2 provisions for LU Identif-
          iers (Vital product data page 83h [SPC-2, p. 203] ).

          [D] We need to resolve whether the notion of "target" is
          relevant to iSCSI.  Does an iSCSI session connect to a target?
          Can it subsequently address multiple targets and LUs or just a
          bunch of LUs?

          [D] We need to provide an understanding of just what a Service
          Delivery Port (SDP) is in the iSCSI context.  Is it an IP end-
          point?  A session endpoint?  A virtual device (target) that a
          session can be connected to?  SAM-2 seems to equate an SDP
          with a target address, "...the application clients in each
          initiator have the ability to discover that logical units in
          the SMU target are accessible via multiple Target Identifiers
          (service delivery ports)..." [SAM-2, pp. 12-13]

     [R] URLs.  It shall be possible to name SCSI devices and possibly
     LUs using a URL syntax.  These names shall be global (uniform) and
     suitable for passing as handles between SCSI application clients.

     [R] Domain names.  The Domain Name Service (DNS) shall be used to
     resolve the <hostname> portion of the url to one, or multiple IP
     addresses.  When a hostname resolves to multiple addresses, these
     addresses shall be equivalent for functional (possibly not perfor-
     mance) purposes.

          [D] This means that the addresses can be used interchangeably
          as long as we don't care about performance.  For example, the
          same set of SCSI targets and/or LUs (tbd) must be accessible
          from each of these addresses.

     [R] Deal with the complications of the new SCSI security architec-
     ture [99-245r8].

          [D] Pay attention to the proxy naming architecture defined by
          the new security model.  In this new model, SCSI Logical Unit

Randy Haagens                                                  [Page 17]
Internet-Draft              TCP RDMA option                 July 7, 2000

          Numbers (LUNs) can be mapped in a manner that gives each host
          (more correctly, each AccessID) a unique LU map.  Thus, a
          given LU within a target may be addressed by different LUNs.

     [R] Support SCSI 3rd-party operations.

          [D] The key issue here relates to the naming architecture for
          SCSI LUs.  We need to determine a method of passing a name or
          handle between parties

3.7 Security

     [R] Authentication.  At a minimum, iSCSI parties shall participate
     in a simple principals authentication protocol.  This protocol
     shall involve a minimum of encryption and no special hardware for

     [R] Bootstrapping.  It shall be possible to negotiate higher levels
     of security than the minimum, technique to be defined.

     [R] Data encryption.  Data encryption shall be optional, but when
     implemented, shall be done in a manner prescribed by iSCSI, by
     reference to other standards.

     [R] Compatible with IP protocol suite security protocols for the
     present and future.

          [D] We anticipate incorporating IPsec (host-to-host) and
          SSL/TSL (TCP connection) security into the iSCSI protocol by
          reference, and as options.  Adherence to good layering will
          ensure (as much as possible) that future security developments
          at the IP and TCP layers can be utilized by iSCSI.

     [R] Permits use of firewall for security screening.

          [D] It's important to allow a firewall to be used to offload
          authentication from the end node.  This is a possible means of
          defending against Denial of Service (DoS) assaults, from a
          less-trusted area of the network.  We assume that the
          firewall(s) have much greater processing power for dismissing
          bogus connection requests than do the end nodes.

3.8 Topology Discovery

          [D] OK, we said we'd leave this for later.  But why not open
          the discussion?

Randy Haagens                                                  [Page 18]
Internet-Draft              TCP RDMA option                 July 7, 2000

     [R] iSCSI shall have no impact on the use of conventional IP net-
     work discovery techniques.

          [D] IP discovery techniques are well-evolved.  Various network
          management platforms have ways of discovering IP addresses,
          such a mining router caches.  We assume that these techniques
          will be used, and will find all of the IP end points that con-
          tain iSCSI nodes.

     [R] iSCSI shall provide some means of determining that a discovered
     IP end point in fact is an iSCSI node.

          [D] This requirement is just a placeholder.  Generally in IP
          discovery, there is some way of determining the type of the
          discovered device.  Possibly this is due to the presence of
          the SNMP protocol and specific MIB variables.  In this case,
          SNMP is the bootstrap protocol.  Alternatively, one could
          probe various TCP port numbers to determine if there exists a
          higher-level protocol at each port (the port number would tell
          you which protocol).  To be determined.  But in any case, some
          means is needed to determine that an iSCSI entity is present
          at an IP end point.

     [R] When a device supports multiple IP end points, some means of
     determining the IP connection topology is needed.

          [D] A device may support multiple end points, yet it may not
          be reasonable to bind any combination of the end points
          together into an iSCSI session.  For example, a port con-
          troller (aka channel group) card may have four ports that can
          be bound together.  The storage controller may support four of
          these port controllers, yet not allow the binding together
          into a session of TCP connections made on different port con-

          [D] A really simple solution to this problem would be to
          define a means of describing port topology, and provide for
          reading that description either from a MIB or directly from
          the iSCSI layer (with a command).

     [R] SCSI protocol-dependent techniques shall be use for further
     discovery beyond the iSCSI layer.

          [D] Discovery is a complex process.  But SCSI provides
          specific hooks for doing the work, and all we need to do is
          transport the commands associated with this process.  Gen-
          erally the SCSI discovery process involves using the Report
          LUNs command to determine which LUs are addressable at a given

Randy Haagens                                                  [Page 19]
Internet-Draft              TCP RDMA option                 July 7, 2000

          service delivery port.  Subsequently, the true identity of
          each LU (ie, name) is discovered by reading Vital product data
          page 83h.  By comparing LU IDs, the discovery process can find
          that a given LU is accessible through multiple paths.

          [D] We need only verify that this SCSI mechanism is suffi-
          cient.  Hopefully, we will not need to augment SCSI at the
          iSCSI layer.

3.9 Management

     [R] IP-based management protocols.  It shall be possible (but not
     required) to use IP-based management protocols such as SNMP and RMI
     in conjunction with iSCSI.  However, the present effort will not
     define the management architecture for iSCSI networks.

     [R] SCSI management protocols.  It shall be possible to use SCSI
     commands for management (eg, SCSI Enclosure Services, SES commands)
     to manage iSCSI devices.

3.10 Interoperability

     [R] It must be possible for hosts and devices that implement only
     those features specified in the RFC to interoperate.

     [R] Software implementation is possible using conventional TCP/IP
     protocol stack.

          [D] Although some low-performance products may contemplate an
          all-software implementation, we expect the majority of iSCSI
          products to employ hardware protocol acceleration.  This
          requirement really is here to solve two problems (1) Proof of
          interoperability, by compatibility with extant TCP implementa-
          tions; (2) Prototyping, where the iSCSI protocol is first
          implemented in software using these conventional stacks.
          These prototypes will likely become the early reference imple-

4 References

     [SAM-2] ANSI NCITS.  Weber, Ralph O., editor.  SCSI Architecture
     Model -2 (SAM-2).  T10 Project 1157-D.  rev 13, 22 Mar 2000.

     [SPC-2] ANSI NCITS.  Weber, Ralph O., editor.  SCSI Primary Com-
     mands - 2 (SPC-2).  T10 Project 1236-D.  rev 18, 21 May 2000.

Randy Haagens                                                  [Page 20]
Internet-Draft              TCP RDMA option                 July 7, 2000

     [CAM-3] ANSI NCITS.  Dallas, William D., editor.  Information Tech-
     nology - Common Access Method - 3 (CAM-3)).  X3T10 Project 990D.
     rev 3, 16 Mar 1998.

     [99-245r8] Hafner, Jim.  A Detailed Proposal for Access Controls.
     T10/99-245 revision 8, 26 Apr 2000.

     [SPI-X] ANSI NCITS.  SCSI Parallel Interface - X.

     [FCP] ANSI NCITS.  SCSI-3 Fibre Channel Protocol [ANSI X3.269:1996]

     [FCP-2] ANSI NCITS.  SCSI-3 Fibre Channel Protocol - 2 [T10/1144-D]

5 Author

     Randy Haagens
     Roseville, R5U-P5/R5
     Hewlett-Packard Company
     8000 Foothills Blvd. MS 5668
     Roseville, CA 95747-5668

     Phone:+1 916 785 4578

Expires January 2001

Randy Haagens                                                  [Page 21]