DTN Research Group V. Cerf INTERNET-DRAFT Worldcom/Jet Propulsion Laboratory S. Burleigh March 2003 A. Hooke Expires September 2003 L. Torgerson NASA/Jet Propulsion Laboratory R. Durst K. Scott The MITRE Corporation K. Fall Intel Corporation H. Weiss SPARTA, Inc. Delay-Tolerant Network Architecture Status of this Memo This document is an Internet-Draft and is in full conformance with all provisions of Section 10 of RFC2026. Internet-Drafts are working documents of the Internet Engineering Task Force (IETF), its areas, and its working groups. Note that other groups may also distribute working documents as Internet-Drafts. Internet-Drafts are draft documents valid for a maximum of six months and may be updated, replaced, or obsoleted by other documents at any time. It is inappropriate to use Internet-Drafts as reference material or to cite them other than as "work in progress." The list of current Internet-Drafts can be accessed at http://www.ietf.org/ietf/1id-abstracts.txt. The list of Internet-Draft Shadow Directories can be accessed at http://www.ietf.org/shadow.html. This document was produced within the IRTF's Delay Tolerant Networking Research Group (DTNRG). See http://www.dtnrg.org Abstract This document describes an architecture for delay-tolerant networks, and is a generalization of the architecture designed for the Interplanetary Internet: a communication system to provide Internet- like services across interplanetary distances in support of deep space exploration. This generalization addresses networks with operational and performance characteristics make conventional networking approaches either unworkable or impractical. We define a message-based overlay that exists above the transport layer of the networks on which it is hosted. The document presents an architectural overview followed by discussions of services, topology, routing, security, reliability and state management. Internet Draft draft-irtf-dtnrg-arch-02.txt March 2003 Table of Contents Status of this Memo................................................1 Abstract...........................................................1 Table of Contents..................................................2 1 Introduction................................................4 2 Why an Architecture for Delay-Tolerant Networking?..........4 3 DTN Architectural Description...............................5 3.1 The DTN Architecture is Based on Virtual Message Switching..............................................5 3.2 DTN Classes of Service Mimic Postal Operation..........5 3.3 DTN Postal-Style Delivery Options......................6 3.4 Nodes..................................................7 3.5 Regions and Gateways...................................8 3.6 Tuples.................................................9 3.7 Late Binding...........................................9 3.8 Routing...............................................10 3.9 Bundle Fragmentation and Reassembly...................12 3.10 Bundle Layer Reliability and Custodianship............13 3.11 Time Synchronization..................................13 3.12 Congestion and Flow Control at the Bundle Layer.......14 3.13 Security..............................................15 4 State Management Considerations............................16 4.1 Demultiplexing and Binding State......................16 4.2 Bundle Retransmission State...........................17 4.3 Bundle Routing State..................................17 4.4 Security-Related State................................17 5 Bundle Header Information..................................18 6 Application Structuring Issues.............................19 7 Convergence Layer Considerations for Use of Underlying Protocols..................................................20 8 Summary....................................................20 9 Informative References.....................................20 10 Security Considerations....................................21 11 Authors' Addresses.........................................22 12 Intellectual Property Notices..............................23 13 Copyright Notices..........................................23 Cerf, et al. Expires September 2003 [Page 2] Internet Draft draft-irtf-dtnrg-arch-02.txt March 2003 Acknowledgments John Wroclawski, David Mills, Greg Miller, James P. G. Sterbenz, Joe Touch, Steven Low, Lloyd Wood, Robert Braden, Deborah Estrin, Stephen Farrell and Craig Partridge all contributed useful thoughts and criticisms to previous versions of this document. We are grateful for their time and participation. This work was performed in part under DOD Contract DAA-B07-00-CC201, DARPA AO H912; JPL Task Plan No. 80-5045, DARPA AO H870; and NASA Contract NAS7-1407. Release Notes draft-irtf-ipnrg-arch-00.txt, May 2001: Original Issue. draft-irtf-ipnrg-arch-01.txt, August 2002: -Restructured document to generalize architecture for delay-tolerant networking. -Refined DTN classes of service and delivery options. Added a "reply-to" address to have response information such as error messages or end-to-end acks directed to a source-specified third party. -Further defined the topological elements of delay tolerant networks. -Elaborated routing and reliability considerations. -Initial model for securing the delay tolerant network infrastructure. -Expanded discussion of flow and congestion control. -Added section discussing state information at the bundle layer. -Updated bundle header information and end-to-end data transfer. draft-irtf-dtnrg-arch-02.txt, March 2003: -Revised model for delay tolerant network infrastructure security. -Introduced fragmentation and reassembly to the architecture. -Removed significant amounts of rationale and redundant text. -Moved bundle transfer example(s) to separate draft(s). Cerf, et al. Expires September 2003 [Page 3] Internet Draft draft-irtf-dtnrg-arch-02.txt March 2003 1 Introduction This document describes an architecture for Delay-Tolerant interoperable networking. The architecture embraces the concepts of occasionally-connected networks that suffer from frequent partitions and that may be comprised of more than one divergent set of protocol families. The basis for this architecture lies with that of the Interplanetary Internet, which focused primarily on the issue of deep space communication in high-delay environments. We expect the current DTN architecture described here to be utilized in various high-delay and unusual environments; the case of deep space is one of these, and is still being pursued as a specialization of this architecture (http://www.ipnsig.org). Other networks to which we believe this architecture may apply include sensor-based networks using scheduled intermittent connectivity, classes of terrestrial wireless networks that cannot ordinarily maintain end-to-end connectivity, and more "conventional" internets with long delays. A DTN tutorial [FW03], aimed at introducing DTN and the types of networks for which it is designed is available to introduce new readers to the fundamental concepts and motivation. The particular approach we employ is that of an end-to-end message- based overlay that exists above the transport layers of the networks on which it is hosted. The overlay employs intermediate storage at message switches, and includes a hop-by-hop transfer of reliable delivery responsibility known as "custody transfer." It also includes a flexible naming format and a security model aimed at protecting infrastructure from unauthorized use. 2 Why an Architecture for Delay-Tolerant Networking? The reason for pursuing an architecture for delay tolerant networking stems from several factors. These factors are summarized below; much more detail on their rationale can be explored in [SB03,KF03,DFS02]. The existing Internet Protocols do not work well for some environments. Some environments violate these inherent assumptions in the TCP/IP approach: - that an end-to-end path between source and destination exists for the duration of the communication session; - (for reliable communication) that the maximum round-trip time over that path is not excessive and not highly variable from packet to packet; and - that the end-to-end loss is relatively small - that all routers and end stations support the TCP/IP protocols - that applications need not worry about communication performance In light of these issues, the DTN architecture is conceived based on a number of design principles that are summarized here (and further discussed in [KF03], as mentioned above): - use messages (not streams or packets) as the communication abstraction - encourage applications to minimize round-trip exchanges Cerf, et al. Expires September 2003 [Page 4] Internet Draft draft-irtf-dtnrg-arch-02.txt March 2003 - guide application design to cope with application restart after failure while network transactions remain pending - use storage within the network to support store-and-forward operation over potentially long timescales (i.e. to support operation in environments where no end-to-end path may ever exist) - provide security mechanisms that protect the infrastructure from unauthorized use; if appropriate, provide access to these mechanisms by applications for use in their own application-level security protocols - provide a coarse-grained class of service and delivery options based on services provided by the US (and other) postal systems 3 DTN Architectural Description The previous section presented the design principles that guide the definition of the DTN architecture. This section presents a description of the design decisions that result from those design principles. 3.1 The DTN Architecture is Based on Virtual Message Switching A DTN transmits application-layer "bundles" that contain whatever the requesting application wishes to send. Bundles are sent by and delivered to applications in an atomic fashion, although they may be split up during transmission. Bundles are also called "messages" in this document. Message senders and recipients are identified by (variable-length) source and destination names called tuples (described below). Messages may also contain an optional reply-to tuple used when special diagnostic operations are requested to direct diagnostic output to an entity other than the sender. Generally, messages are transferred in an overlay above the transport layer called the "bundle layer." A message-switched abstraction provides the network with a priori knowledge of the size and performance requirements of requested data transfers. When there is a significant amount of queuing that can occur prior to transmission over an outbound route (as is the case in the DTN version of store-and-forward) the advantage provided by knowing this information may be significant for making scheduling decisions. An alternative abstraction (i.e. of stream based delivery) would make such scheduling very difficult. Although packets provide some of the same benefits as messages, larger aggregates provide a way for the network to apply scheduling and buffer management to entire units of data that are useful to applications. 3.2 DTN Classes of Service Mimic Postal Operation The (U.S. or similar) Postal Service provides a strong metaphor for the services that a Delay-Tolerant Network offers. Traffic is generally not interactive. It may be one-way in nature. There are generally no strong guarantees of timely delivery, yet there are some forms of class of service and security. Cerf, et al. Expires September 2003 [Page 5] Internet Draft draft-irtf-dtnrg-arch-02.txt March 2003 The DTN Architecture, like the Postal Service, offers *relative* measures of priority (low, medium, high). It may offer basic notifications, for example: "the intended recipient has accepted delivery of the message," "the route taken by this message is as follows..." and "the message has been transmitted." An essential element of Postal Service operation for networking is that messages have a place to wait in queue until an outbound communication opportunity is available. This highlights the following assumptions: 1. that storage is available and well-distributed throughout the network 2. that storage is sufficiently persistent and robust to store messages until forwarding can occur, and 3. (implicitly) that this 'store-and-forward' model is a better choice than attempting to effect continuous connectivity or other alternatives For a network to effectively support the DTN architecture, these assumptions must be considered and must be found to hold. We have currently defined three specific classes of service in the DTN architecture: - Bulk - In bulk class, bundles are shipped on a "best effort" basis. No bulk class bundle will be shipped until all complete bundles of other classes bound for the same next hop destination have been shipped. - Normal - Normal class bundles that are in queue and bound for the same next hop destination are shipped prior to any complete bulk class bundles that are in queue. - Expedited - Expedited bundles, in general, are shipped prior to bundles of other classes. However, the bundle layer *may* implement a queue service discipline that prevents starvation of the normal class, which may result in some normal bundles being shipped before expedited bundles bound for the same next hop destination as the normal class bundles. Applications specify their requested class of service. This request, coupled with policy applied at message switches, affects the overall likelihood and timeliness of message delivery. 3.3 DTN Postal-Style Delivery Options Applications may request any of the following five delivery options: - Custodial Delivery - a bundle will be transmitted by the bundle layer using reliable transport protocols (if available), and the point of reliable delivery responsibility (i.e. retransmission buffer) will advance through the network from one custodian to another until the bundle reaches its destination. The bundle Cerf, et al. Expires September 2003 [Page 6] Internet Draft draft-irtf-dtnrg-arch-02.txt March 2003 layer depends on the underlying transport protocols of the networks that it operates over to provide the primary means of reliable transfer from one bundle layer to the next. However, when custodial delivery is requested, the bundle layer additionally provides a coarse-grained timeout and retransmission mechanism and an accompanying (bundle-layer) hop-by-hop acknowledgment mechanism. When a bundle application does *not* request custodial delivery, this bundle layer timeout and retransmission mechanism is not employed, and successful bundle layer delivery depends solely on the reliability mechanisms of the underlying transport layers - Return Receipt - a return-receipt bundle is issued by the receiving bundle layer implementation when the (arriving) subject bundle is consumed *by the destination application* (not simply the destination bundle layer). The receipt is issued to the entity specified in the source tuple of the subject bundle or the source's designated alternate (reply-to field), which would typically be located on a different host. The return receipt uses the same custodial delivery option setting used in the subject bundle. (Return receipts are never issued with the return receipt delivery option requested, to avoid "return receipt storms.") - Forwarding Indication - sent by a bundle router when the last fragment of a bundle has been forwarded. The indication is sent to the reply-to destination if specified, and to the source of the subject bundle otherwise - Custody Transfer Indication - same as forwarding indication, but sent when a custodial transfer has successfully completed - Secured Delivery - indicates the application has provided authentication material along with its message send request. To operate under general circumstances, applications should be prepared to supply authentication credentials and request secured delivery. Local policy determines whether any bundles may be sent lacking the security option, and regions beyond the originating region may require security even if the originating region does not. 3.4 Nodes A DTN node (or "node" in this document) is an engine for sending and receiving DTN messages (bundles). DTN nodes may act as sources, destinations, or forwarders of bundles. Each node is uniquely identified by at least one tuple containing a region name and an entity name; a node that is reachable within multiple regions will have at least one identifying tuple for each region in which it is reachable. The name for a DTN node itself, as opposed to an application *using* that node, is specified in a region-specific manner using all or part of the entity identifier. This is necessary for generating custody acknowledgments to the bundle layer itself rather than to a specific application. Cerf, et al. Expires September 2003 [Page 7] Internet Draft draft-irtf-dtnrg-arch-02.txt March 2003 3.5 Regions and Gateways The DTN architecture defines a "network of internets" Comprised of "DTN regions." Assignment of DTN nodes to particular regions is an administrative decision, and may be influenced by differences in protocol families, connection dynamics, or administrative policies. Regions may also be delimited based upon other criteria, such as trust boundaries [NEWARCH]. Each DTN region has a unique name that is known (or knowable) among all regions of the DTN. Thus, a DTN- wide repository for region names is required. All inter-region communication takes place via DTN *gateways*, which provide the interconnection points between regions. These correspond to "waypoints" in [META], and to the gateways described in the original ARPANET/Internet designs [CK74]. DTN gateways differ from ARPANET gateways because they operate above the transport layer and are focused on message switching rather than packet switching. However, both DTN gateways and ARPANET gateways provide interoperability between the protocols specific to one region and the protocols specific to another. Regions are key concepts in the DTN architecture. DTN bundles that originate outside the destination region are first transmitted toward one of the DTN gateways that connect the source region with one or more other regions. Routing outside the destination region is based solely on the destination region's name, and not on the completely formed destination name (see below). The following list identifies the requirements for DTN regions: - Each DTN region shall have an identifier space that is shared by all DTN nodes within the region. A region must specify the naming conventions to be employed within that region for entity identification. - Each node that is a member of the region shall have a unique identifier drawn from that identifier space. (Note that for some types of regions, a "node" may be made up of a collection of computational elements, possibly geographically distributed. A single unique identifier may collectively refer to them. Further, the unique identifier requirement only applies to nodes that are intended to *receive* data from other DTN nodes.) - To be considered a member of a region, each prospective member of the region shall have the ability to reach every other member of the region without depending on any DTN nodes outside the region using some protocol(s) known or knowable to each node. (Although a DTN node may not necessarily be *directly* reachable. This may require forwarding and/or store-and-forward operation by other DTN nodes inside the same region.) Cerf, et al. Expires September 2003 [Page 8] Internet Draft draft-irtf-dtnrg-arch-02.txt March 2003 3.6 Tuples The region name described above (plus some forwarding state) is necessary and sufficient to route a bundle of data to its destination region, but not to deliver it to the specific endpoint(s) for which it is intended. The DTN architecture uses a tuple comprised of the region name and a region-specific entity name to identify a particular set of entities in the DTN. The entity name is *opaque* outside the region of definition. An entity might be a host, a protocol, an application, some aggregate of these, or something else depending on the nature of the addressing and naming structures used in the containing region. In the Internet, for example, an entity name could be as general as a URI or URL. In any case, some form of binding mechanism (local to the containing region) is required to associate communication endpoints (and their local addressing or naming ID) with a DTN tuple. The details of the binding mechanism are region-specific and not discussed here. However, such a mechanism must provide a way for a requesting application to bind to a prefix of a fully specified destination tuple. Regions are named by applications using syntax identical to that used in the domain name system (DNS). (That is, hierarchical tree structure, dot-separated text node names, left to right progresses from leaf to root, sibling nodes must have different names.) The bundle layer may translate a region name to a bundle-layer-specific region address for transmission. The scope of the region name space (and region address space, if used) spans an entire DTN. Discovery of valid DTN region names is the responsibility of bundle- layer routing. Knowledge of appropriate entity identifiers is out of the scope of this document, but corresponds directly to the Internet problem of knowing port numbers a-priori. 3.7 Late Binding The opacity of entity names outside their local region enforces another key concept in the DTN Architecture: that of late binding. Late binding refers to the fact that the entity name of a tuple is not interpreted (e.g., used to form an address for delivery within the region) outside its local region. This avoids having a universal name-to-address binding space (and its associated database and synchronization issues). This approach preserves a significant amount of autonomy within each region. The entity names used in bundles might be built on DNS names, or URLs, but they might also be "expressions of interest" or forms of database-like queries as in a directed diffusion-routed network [IGE00] or intentional naming [WSBL99]. Additionally, late binding avoids the delay-related issue that the binding information might be highly ephemeral relative to the transit time of a bundle. We assume that the internal details of a Cerf, et al. Expires September 2003 [Page 9] Internet Draft draft-irtf-dtnrg-arch-02.txt March 2003 destination region will be sufficiently stable for the duration of a transmission of a message within that region, or that delay-tolerant mechanisms will be employed *within* the region to compensate. (This is, by definition, a local issue within the region and may be accommodated in whatever manner is most practical for that region.) 3.8 Routing The bundle layer provides routing among DTN nodes, including DTN gateways. There may be many DTN gateways that interconnect adjacent regions, and there may be multiple bundle routing "hops" within a region (particularly if intra-regional connectivity is intermittent). The distinction between a router and a gateway relates to the late binding of names, as described above. In particular, a region's gateways are the first in an inter-region store-and-forward chain to utilize the region-local entity identifier portion of tuples for forwarding decisions. The DTN nodes are responsible for using whatever reliability mechanisms exist in the underlying (transport- and-below) layers, and augmenting those mechanisms with bundle-layer mechanisms to implement the required reliability. 3.8.1 Types of Routes DTNs may be required to function in the presence of any or all of the following types of connectivity. Routes are comprised of a sequence of "contacts" that indicate the duration, endpoints, and forwarding capacity of a link in the topology graph. They are generally assumed to be describing edges on a *directed* graph, as communication capabilities cannot be assumed to be symmetric. Persistent Contacts Persistent contacts are edges with a neighboring DTN node that are always available, with no connection establishment required. In the IP world, a Digital Subscriber Line (DSL) or other "always-on" connection is an example. OnDemand Contacts OnDemand contacts are contacts that require some action in order to instantiate, but then otherwise function as persistent contacts until terminated. Dial-up connections are an example of OnDemand contacts (at least, from the viewpoint of the dialer; they may be viewed as an Opportunistic Contact û below û from the viewpoint of the dial-up service provider). Intermittent - Scheduled Contacts Scheduled contacts are those where there is an agreement to establish a link between two points at a particular time, for a particular duration. An example of a scheduled route is a link that uses a low- earth orbiting satellite. A schedule of view times and durations can Cerf, et al. Expires September 2003 [Page 10] Internet Draft draft-irtf-dtnrg-arch-02.txt March 2003 be constructed when next-hop neighbors are accessible via the satellite. Intermittent - Opportunistic Contacts Opportunistic contacts are those that are not scheduled, but rather present themselves unexpectedly. Such contacts could be with an aircraft flying overhead and beaconing, advertising its availability for communication. Another type of opportunistic contacts might be via infrared communication link between a personal digital assistant (PDA) and a kiosk in an airport concourse offering bundle service as the PDA's owner walks by. If the PDA's owner authorizes it, the PDA could communicate the owner's planned path and a desire for contacts with subsequent kiosks in the concourse, resulting in a set of probable communication opportunities for which bundle transfers can be scheduled. Intermittent - Predicted Contacts Predicted contacts are those that are based on no fixed schedule, but rather a history of opportunistic contacts that suggests that contact with an intermittently-connected neighbor will probably occur within a certain period of time and will probably last for some inferred duration. Given a great enough certainty that the contact will occur, a DTN node may allocate bundles to that predicted contact period that would be allocated to different contacts otherwise. In the previous example, the probable contacts in the airport concourse would result in the establishment of a set of predicted contacts of a given duration (perhaps based on the PDA owner's walking speed and the kiosk's coverage area). The PDA could decide how to use those contacts for transmitting waiting bundles, as well as perhaps to request bundles that were awaiting delivery at any of a number of store-and-forward points to which the user had access. The algorithms for establishing the predicted time and duration of a contact, the degree of uncertainty in those estimates, the time at which to abandon the wait for a predicted contact, and the guidelines for allocating bundles to such contacts are all open research areas. 3.8.2 Bundle Storage for Store-and-forward operation While IP networks are based on "store-and-forward" operation, there is an assumption that the "storing" will not persist for more than a modest amount of queuing and transmission delay. In contrast, the DTN architecture does not expect that an outbound link will be available when a bundle is received, and expects to store that bundle for some time until a link does become available. We anticipate that most DTN nodes will use some form of persistent storage for this -- disk, flash memory, etc. All DTN forwarding nodes ("DTN routers"), even those that do not provide custodial operations as described in section 3.3, must be able to queue bundles until outbound contacts are available. Each Cerf, et al. Expires September 2003 [Page 11] Internet Draft draft-irtf-dtnrg-arch-02.txt March 2003 DTN node that is also willing to provide custody transfer operations should provision for longer-term storage of bundles, committing to store bundles for which it takes custody for the entire remainder of their lifetimes, if necessary. It is entirely possible that a custodian will need to "take a break" from further custodianship while it completes pending custodial operations and recovers persistent storage. Two mechanisms support this: First, the node can simply forward incoming bundles without taking custody. The fact that a node is a potential custodian is no guarantee that it will take custody of any given bundle that is routed to it. Second, the node can revise its advertisement of custodial capability in routing updates. This is a longer-term solution, but has the attractive property that DTN nodes searching for a custodian do not route a bundle out of its way vainly in search of custodianship at the node in question. Also, see section 3.12 on DTN flow and congestion control. The availability of long-term storage for bundles allows the next-hop forwarding decision to potentially be revoked. In particular, if a future contact is chosen to carry a bundle and some other superior contact becomes known, the bundle could be re-assigned. Details of this re-assignment operation are local to a particular bundle router and influenced by the times at which contact schedules become known. 3.9 Bundle Fragmentation and Reassembly There are two forms of fragmentation/reassembly in Bundling: Any DTN router may ûproactively- choose to divide a block of data into multiple self-identifying smaller blocks and transmit each such block as a bundle. In this case the *final destination(s)* are responsible for extracting the smaller blocks from incoming bundles and reassembling them into the original large block. This form of fragmentation is analogous to IP fragmentation. A bundle router may ûreactively- choose to fragment a bundle on receipt. This situation arises when a portion of a bundle may have been received successfully. In this case, the receiving node modifies the incoming bundle to indicate it is a fragment, and forwards it normally. The previous-hop sender may learn that only a portion of the bundle was delivered to the next hop, and optimistically continue sending the remaining portion of the original bundle when subsequent contacts become available. Reactive fragmentation is specifically designed to handle cases in which a router is faced with forwarding a bundle for which no single contact provides sufficient data transfer volume. Cerf, et al. Expires September 2003 [Page 12] Internet Draft draft-irtf-dtnrg-arch-02.txt March 2003 3.10 Bundle Layer Reliability and Custodianship The bundle layer provides an end-to-end reliable message delivery service that employs in-network retransmission between (possibly non- network-layer-adjacent) DTN nodes. The bundle layer makes use of the reliability mechanisms available from the underlying transport layers of each region, and a single bundle-layer hop *may* span an entire region. The bundle layer supports end-to-end reliability derived solely from the custody transfer mechanism, but also provides a true end-to-end acknowledgement for application use. Applications wishing to utilize this indication for their own end-to-end bundle retransmission mechanisms are free to do so. The bundle layer provides three types of delivery services, with various levels of reliability-enhancing mechanisms: end-to-end acknowledgment of a bundle, custodial transmission (with in-network retransmission if required), and unacknowledged bundle transmission. Custody transfer allows the source to delegate retransmission responsibility and recover its retransmission-related resources relatively soon after sending the bundle (on the order of a round- trip time to the first bundle hop). While not every node in a DTN must be capable of providing custodial services, all DTN routers (that span networks that may be frequently disconnected) are expected to be able to be custodians. This expectation supports custodial operation along the primary path without forcing custodial bundles to make routing diversions in order to locate a custodian. 3.11 Time Synchronization The DTN architecture depends on time synchronization (supported by external, region-local protocols) for two primary purposes: routing with scheduled or predicted contacts and bundle "time to live" computations. Routing based on schedules and dependent upon coordination of shared assets (such as directional antennas) creates a requirement for time synchronization to achieve contact rendezvous. Time to live computations are achieved by including a source time stamp and an explicit time to live field (in time units after the time in the source time stamp). Its sole use is for purging data from the network, so the synchronization requirements posed here are not strict. This approach allows a source time stamp to be used for a number of purposes, such as unique identification of individual messages from a particular source. DTN nodes must ensure that timestamps in bundles they send never decrease. Applications specify an expiration time (actually, a time to live in seconds) for bundles they send. If not supplied, or if the user- supplied value is larger than local policy permits, the bundle layer will provide a value. Note that this value is treated as an actual Cerf, et al. Expires September 2003 [Page 13] Internet Draft draft-irtf-dtnrg-arch-02.txt March 2003 "time to live" -- it is added to the time that a bundle was submitted by the application to determine the time at which the bundle will be purged from the network. Appropriate values depend on the network and the data, and could conceivably vary widely (e.g. from milliseconds to weeks). 3.12 Congestion and Flow Control at the Bundle Layer The subject of congestion control and flow control at the bundle layer is one on which the authors of this document have not yet reached consensus. We have unresolved concerns about the efficiency and efficacy of congestion and flow control schemes implemented across long and/or highly variable delay environments. One view of congestion control is as follows: "Congestion control is concerned with allocating the resources in a network such that the network can operate at an acceptable performance level when the demand exceeds or is near the capacity of the network resources. These resources include bandwidths of links, buffer space (memory), and processing capacity at intermediate nodes. Congestion occurs when the demand is greater than the available resources." [RJ90] For the purposes of this document, we define "flow control" as a means of assuring that the rate at which a sending node transmits data to a receiving node does not exceed the maximum rate at which the receiving node is prepared to receive data from that sender. (Note that this is a generalized notion of flow control, rather than one that applies only to end-to-end communication. In particular, the concept of flow control between the two ends of a single link may be indispensable in such DTN regions as the "interplanetary backbone.") We define "congestion control" as a means of assuring that the aggregate rate at which all traffic sources inject data into a network does not exceed the maximum aggregate rate at which the network can deliver data to destination nodes. If flow control is propagated backward from destination nodes to routers and eventually back to traffic sources, then flow control can be at least a partial solution to the problem of congestion as well. DTN flow control decisions must be made within the bundling layer itself based on information about resources (in this case, primarily persistent storage) available within the bundle node. However, the bundle layer *might* be able to delegate the implementation of those decisions to the underlying transport protocols, as follows. We have not yet considered multipoint communication at or below the bundle layer, so each individual flow control problem involves just two nodes. Because inter-region traffic must pass through inter- region gateways, any two nodes between which flow control is an issue must inhabit a common DTN region and be using a common transport protocol below the bundle layer (otherwise they could not be communicating and there would be no flow to control). Therefore, it may be possible to accomplish DTN flow control by invoking whatever Cerf, et al. Expires September 2003 [Page 14] Internet Draft draft-irtf-dtnrg-arch-02.txt March 2003 flow control mechanisms are already provided within the region by the common transport protocol, if such mechanisms exist. Alternatively, a new, supplementary flow control protocol could be developed at the bundling layer; this would have the advantage of reducing a DTN's reliance on capabilities provided by the underlying protocols. At this time it's still unclear which approach is preferable, and some combination of the two may eventually be declared to be the best choice. In either case, the problem of flow control between nodes separated by very large signal propagation times remains to be solved: TCP's flow control and congestion control facilities could be leveraged within regions characterized by small round-trip times, but at this time no comparable protocol exists for very long delay paths. It remains as an exercise for us to demonstrate that a hop-by-hop flow control scheme in long and/or highly variable delay environments can effect end-to-end congestion control in a fair and efficient manner. 3.13 Security Resource scarcity in delay-tolerant networks dictates that some form of access control to the network itself is required in many circumstances. It is not acceptable for an unauthorized user to be able to easily flood the network with traffic, possibly preventing the network's mission from being accomplished. Implicit in this statement is a requirement for some form of admission control and/or in-network authentication that is sensitive to the class of service that a user has requested, and the means to verify that the user is authorized to make that request. In a low delay environment, this would be tedious for performance reasons. In a high/variable delay, and possibly low data rate environment, it is potentially much worse: remote access control lists are difficult to update, query/response keying protocols are not resource-efficient, and routers or end nodes might be compromised for significant periods of time before being noticed. To implement the security model, each message includes an immutable "postage stamp" (a type of capability) containing a verifiable identity of the sender (or role), an approval (and approving authority) of the requested class of service (CoS) associated with the message, and other conventional cryptographic material to verify accuracy of the message content. Routers check credentials at each DTN hop, and discard traffic as early as possible if authentication fails. This approach has the associated benefit of making denial-of- service attacks considerably harder to mount as compared with conventional Internet routers. The current approach uses public key cryptography as a starting point for keying. Routers and principals are issued public/private key pairs, and a principal sending a message must obtain a signed copy of its public key from a certificate authority known to DTN routers. Cerf, et al. Expires September 2003 [Page 15] Internet Draft draft-irtf-dtnrg-arch-02.txt March 2003 (All routers in a DTN are assumed to be pre-equipped with copies of one or more certificate authority public keys and their own public/private key pairs). A user then presents her signed public key along with a message to be carried signed using her private key. At the first DTN router, the signed public key is used to validate the sender and requested CoS against an access control list stored in the router. Accepted messages are then re-signed in the key of the router for transit. Using this approach, only first-hop routers need cache per-user certificates, and then only for adjacent users. Non-edge "core" routers can rely on the authentication of upstream routers to verify the authenticity of messages. We believe this approach will help to improve the scalability of key management for these networks, as it will limit the number of cached public key certificates to a function of the number of adjacent routers rather than the number of end-users. This should provide both the obvious advantage of space savings, but also an improvement to system management as router keys are expected to be changed less frequently than end-user keys. As DTN routers are likely to be deployed in remote areas, re-keying operations may be comparatively burdensome system management tasks, so limiting the number and frequency of certificate updates should provide additional savings. The current approach is partially susceptible to compromised routers. If an otherwise-legitimate router is compromised, it would be able to utilize network resources at an arbitrary CoS setting and send traffic purportedly originating from any user who's identity is known to the router. However, if the message signature is carried end-to- end (an option for DTN security), a legitimate user could repudiate the origin of any traffic generated in this manner. Thus, we believe a reasonable trade-off is to admit the possibility that a compromised router could launch a denial-of-service attack in order to gain the scalability benefits of not checking end-user credentials at every hop. 4 State Management Considerations An important aspect of any networking architecture is its management of state information. This section describes the state information that is managed at the bundle layer and discusses the conditions under which that state information is established and how it is removed. 4.1 Demultiplexing and Binding State In long/variable delay environments, an asynchronous application interface seems to be the only sensible approach. Such interfaces involve callback registration actions that create state information. This information is typically created by explicit request of the application, and is removed by a separate explicit request, and is thus "hard" state. In most cases, there must be provision for retaining this state information across application termination/restart operations, and across operating system termination/restart operations because a client/server message round- Cerf, et al. Expires September 2003 [Page 16] Internet Draft draft-irtf-dtnrg-arch-02.txt March 2003 trip time may exceed the requesting application's execution time (or hosting system's uptime). In addition, the tuples for which an application wishes to receive data must be associated with the protocol endpoint identifier information needed to reach the application itself in the region it exists. This operation (analogous to the socket bind() operation) may result in a client/server type of interaction within a region because regional gateways must extract and resolve incoming bundle entity names to information required to perform delivery within the destination region (compare, for example with other name/addressing mapping services such as dynamic DNS [RFC2136]). 4.2 Bundle Retransmission State State information to support bundle retransmission is created at a DTN node when a bundle is received from a DTN user application requesting custodial transmission) or when a bundle is received from a peer DTN node and the receiving node intends to assume custody of the bundle. The bundle's time-to-live field (possibly mitigated by local policy) determines when this state is purged from the system in the event that it is not purged explicitly due to acknowledgment. 4.3 Bundle Routing State Forwarding tables, whether statically configured or maintained by routing protocols, introduce state information that must be managed in a manner that is dependent upon the specific routing mechanisms that are employed. While routing protocols have not yet been developed for DTN networks, in order to provide forwarding a DTN router will be required to have available a forwarding table containing region names and next-hops. In addition, it seems highly useful to also include a method for routers to learn about potential custodians. This information would enlarge the overall forwarding state, but probably not significantly. We have not yet seriously considered the routing protocols or metrics that we will use, so we do not have an estimate for the size of each routing entry, whether it be for inter-region or intra-region routing. This remains work to be done. 4.4 Security-Related State The infrastructure protection model described in this architecture requires maintenance of state in all DTN nodes. All nodes are required to store their own public/private key pairs and their network access credentials, signed by an appropriate approving authority. Additionally, all nodes are required to cache public keys for one or more certificate authorities and. Further, in most cases, DTN nodes will cache the public keys (and possibly the credentials) of their next-hop (in bundle-space) neighbors. All keys will have expiration times, and nodes are responsible for acquiring and distributing newly signed copies of their public keys and credentials Cerf, et al. Expires September 2003 [Page 17] Internet Draft draft-irtf-dtnrg-arch-02.txt March 2003 prior to the expiration of the old set (in order to avoid a disruption in network service). Some DTNs may implement security boundaries at various points in the network, at which point end-user credentials are checked in addition to checking router credentials. User application public keys will typically be cached at these points in the network. 5 Bundle Header Information The bundle layer must carry some information end-to-end. The full details of the fields present in a bundle header are specified in [BundleSpec]. This section documents the meta-data information that must be carried end-to-end, and notes which of those data elements may be supplied by the application using the bundle service. * Version Identifier: an 8-bit bundle protocol * Destination entity ID: a variable-length field containing the destination tuple, as described above. It is supplied by the local application when sending using the bundle service. * Source entity ID: this is the identifier of the source bundle application instance, and is a tuple. It is supplied by the local bundle service, since a particular host may have multiple names and one may be chosen based on routing decisions or other criteria opaque to the application (as in multihomed hosts). The source entity ID may be returned to the application to support return receipt processing. * Reply-to entity ID (optional): a source may anticipate not being able to accept replies, and may use the reply-to entity ID to specify the destination for return receipts and delivery records. * Current custodian ID (optional): Entity ID of the current custodian. It is necessary to identify the upstream node that currently has custody of a bundle, in order to acknowledge correct receipt or custody transfer of a bundle or bundle fragments. * Class of service flags: - Flags: custody, return receipt, delivery record - CoS Selector: bulk, normal, expedited - Security: presence of authentication and/or encryption * Send timestamp: the time that a bundle was presented by the sending application to the bundle layer for transmission. * Time to live: an offset, in seconds, from the send timestamp that indicates when the bundle shall be purged from the DTN. * Authentication information (optional): authentication data used to prove that this bundle should be forwarded in the network. * Fragmentation information (optional): for a bundle fragment, indicates the place in the original bundle this fragment belongs. Some bundles (or events) cause a status indication to be generated by the bundle layer. Bundle layer indications are sent as bundles with Cerf, et al. Expires September 2003 [Page 18] Internet Draft draft-irtf-dtnrg-arch-02.txt March 2003 the user data portion replaced by a Status Report, consisting of the following information: * Source entity ID of the subject bundle: a copy of the source tuple from the subject bundle. * Send timestamp of the subject bundle: used to disambiguate status reports for different bundles from the same source entity * Status flags, indicating whether or not a bundle was: . received correctly by the sender of the Status Report . Custodially transferred to the sender of the Status Report . Forwarded by the sender of the Status Report * Time of Receipt (optional): the time at which the sender of the Status Report received the bundle. * Time of Forward (optional): the time at which the sender of the Status Report forwarded the bundle 6 Application Structuring Issues DTN bundle delivery is intended to operate in a *delay-tolerant* fashion over a broad range of network types. That does not mean that there *must* be delays in the network, but that there *may* be very significant delays. The protocols providing the services exposed by the bundle layer are delay tolerant, so to take best advantage of them, applications using them should be, too. Message-oriented communication differs from conversational communication. In general, applications should attempt to include enough information in a bundle so that it may be treated as an independent unit of work by the remote entity (a form of "application data unit" [CT90] or "transaction", although transactions carry connotations of multi-phase commitment that we do not intend here). The goal is to minimize conversation between applications that are separated by a network that presents long and possibly highly variable delays, and to constrain any conversation that does occur to be asynchronous in nature. A single file transfer request bundle, for example, might include authentication information, file location information, and requested file operation. Comparing this style of operation to a classic FTP transfer, one sees that the bundled model can complete in one round trip time, whereas an FTP file "put" operation can take as many as eight round trips to get to a point where file data can flow [DFS02]. Delay-tolerant applications must consider additional factors beyond the conversational implications of long delay paths. Application instances may terminate (voluntarily or not) between the time that a bundle is sent and the time its response is received. If an application has anticipated this possibility, it is possible to re- instantiate the application instance with state information saved in persistent storage. This is an implementation issue, but also an application design consideration. Some consideration of delay-tolerant application design can result in applications that work reasonably well in low-delay environments, and Cerf, et al. Expires September 2003 [Page 19] Internet Draft draft-irtf-dtnrg-arch-02.txt March 2003 that do not suffer extraordinarily in high or highly-variable delay environments. 7 Convergence Layer Considerations for Use of Underlying Protocols Implementation experience with the DTN architecture has revealed an important architectural construct and interface for DTN routers. Not all transport protocols in different protocol families provide the same exact functionality, so some additional adaptation or augmentation on a per-stack basis may be require. This adaptation is accomplished by a set of convergence layers placed between the bundle layer and underlying transport protocols. The convergence layers manage the protocol-specific details of interfacing with a particular transport service and present a consistent interface to the bundle layer. The complexity of a convergence layer may vary substantially depending on the type of protocol stack it adapts. For example, a TCP/IP convergence layer for use in the Internet might only have to add message boundaries to TCP streams, whereas a convergence layer for some network where no reliable transport protocol exists may have a considerable amount of work to do, at least if custody transfer is to be supported. 8 Summary The DTN architecture addresses many of the problems of networks that must operate in environments subject to poor performance and non- continuous end-to-end connectivity. It is based on asynchronous messaging, and uses as a model of service classes those offered by the postal mail system. It accommodates many different forms of connectivity, including scheduled, predicted, and opportunistically connected links. It introduces a novel approach to end-to-end reliability across frequently partitioned and unreliable networks. It also proposes a scheme for securing the network infrastructure against unauthorized access. It is our belief that this architecture is applicable to many different types emerging environments. In subsequent documents, we intend to specify profiles of this architecture that address specific environments in detail. 9 Informative References [SB03] S. Burleigh et al, "Delay-Tolerant Networking û An Approach to Interplanetary Internet," To appear in IEEE Communications Magazine approximately May 2003. [CT90] D. Clark, D. Tennenhouse, "Architectural Considerations for a new generation of protocols," Proc. SIGCOMM 1990. [FW03] F. Warthman, "Delay-Tolerant Networks (DTNs): A Tutorial", Wartham Associates, Available from http://www.dtnrg.org Cerf, et al. Expires September 2003 [Page 20] Internet Draft draft-irtf-dtnrg-arch-02.txt March 2003 [KF03] K. Fall, "A Delay-Tolerant Network Architecture for Challenged Internets," Intel Research, IRB-TR-03-003. Available from http://www.intel-research.net/publications.asp [IGE00] C. Intanagonwiwat, R. Govindan, D. Estrin, "Directed Diffusion: A scalable and robust communication paradigm for sensor networks", MobiCOM 2000, Aug 2000. [WSBL99] William Adjie-Winoto, Elliot Schwartz, Hari Balakrishnan, Jeremy Lilley, The design and implementation of an intentional naming system, Proc. 17th ACM SOSP, Kiawah Island, SC, Dec. 1999. [NEWARCH] NewArch Project: Future-Generation Internet Architecture. http://www.isi.edu/newarch [META] Wroclawski, John T., "The Metanet," Workshop on Research Directions for the Next Generation Internet, May 12-14, 1997, Vienna, VA. http://www.cra.org/Policy/NGI/papers/wroklawWP. [CK74] V. Cerf, R. Kahn, "A Protocol for Packet Network Intercommunication", IEEE Trans. on Comm., COM-22(5), May 1974 [DFS02] R. Durst, P. Feighery, K. Scott, "Why not use the Standard Internet Suite for the Interplanetary Internet?", MITRE White Paper, http://www.ipnsig.org/reports/TCP_IP.pdf [RJ90] R. Jain, "Congestion Control in Computer Networks: Issues and Trends," IEEE Network Magazine, May 1990. [RFC2136] P. Vixie, ed., "Dynamic Updates in the Domain Name System (DNS UPDATE)," Internet Request for Comments, RFC2136, Apr. 1997 [BundleSpec] S. Burleigh, et al, "Bundle Protocol Specification" work in progress, (draft-irtf-dtnrg-bundle-spec-00.txt), March 2003. Available from http://www.dtnrg.org. 10 Security Considerations Security is an integral concern of the Delay Tolerant Network Architecture. Section 3.13 of this document presents an approach for securing the DTN architecture. These capabilities may also be useful in providing facilities to DTN applications to construct their own end-to-end security protocols. Cerf, et al. Expires September 2003 [Page 21] Internet Draft draft-irtf-dtnrg-arch-02.txt March 2003 11 Authors' Addresses Dr. Vinton G. Cerf MCI WorldCom 22001 Loudoun County Parkway Building F2, Room 4115, ATTN: Vint Cerf Ashburn, VA 20147 Telephone +1 (703) 886-1690 FAX +1 (703) 886-0047 Email vcerf@mci.net Scott C. Burleigh Jet Propulsion Laboratory 4800 Oak Grove Drive M/S: 179-206 Pasadena, CA 91109-8099 Telephone +1 (818) 393-3353 FAX +1 (818) 354-1075 Email Scott.Burleigh@jpl.nasa.gov Robert C. Durst The MITRE Corporation 1820 Dolley Madison Blvd. M/S W650 McLean, VA 22102 Telephone +1 (703) 883-7535 FAX +1 (703) 883-7142 Email durst@mitre.org Dr. Kevin Fall Intel Research, Berkeley 2150 Shattuck Ave., #1300 Berkeley, CA 94704 Telephone +1 (510) 495-3014 FAX +1 (510) 495-3049 Email kfall@intel-research.net Adrian J. Hooke Jet Propulsion Laboratory 4800 Oak Grove Drive M/S: 303-400 Pasadena, CA 91109-8099 Telephone +1 (818) 354-3063 FAX +1 (818) 393-3575 Email Adrian.Hooke@jpl.nasa.gov Dr. Keith L. Scott The MITRE Corporation 1820 Dolley Madison Blvd. M/S W650 McLean, VA 22102 Telephone +1 (703) 883-6547 Cerf, et al. Expires September 2003 [Page 22] Internet Draft draft-irtf-dtnrg-arch-02.txt March 2003 FAX +1 (703) 883-7142 Email kscott@mitre.org Leigh Torgerson Jet Propulsion Laboratory 4800 Oak Grove Drive M/S: T1710- Pasadena, CA 91109-8099 Telephone +1 (818) 393-0695 FAX +1 (818) 354-9068 Email Leigh.Torgerson@jpl.nasa.gov Howard S. Weiss SPARTA, Inc. 9861 Broken Land Parkway Columbia, MD 21046 Telephone +1 (410) 381-9400 x201 FAX +1 (410) 381-5559 Email hsw@sparta.com Please refer comments to dtn-interest@mailman.dtnrg.org 12 Intellectual Property Notices The IETF takes no position regarding the validity or scope of any intellectual property or other rights that might be claimed to pertain to the implementation or use of the technology described in this document or the extent to which any license under such rights might or might not be available; neither does it represent that it has made any effort to identify any such rights. Information on the IETF's procedures with respect to rights in standards-track and standards-related documentation can be found in BCP-11. Copies of claims of rights made available for publication and any assurances of licenses to be made available, or the result of an attempt made to obtain a general license or permission for the use of such proprietary rights by implementors or users of this specification can be obtained from the IETF Secretariat. The IETF invites any interested party to bring to its attention any copyrights, patents or patent applications, or other proprietary rights which may cover technology that may be required to practice this standard. Please address the information to the IETF Executive Director. 13 Copyright Notices Copyright (C) The Internet Society (2003). All Rights Reserved. This document and translations of it may be copied and furnished to others, and derivative works that comment on or otherwise explain it or assist in its implmentation may be prepared, copied, published and distributed, in whole or in Cerf, et al. Expires September 2003 [Page 23] Internet Draft draft-irtf-dtnrg-arch-02.txt March 2003 part, without restriction of any kind, provided that the above copyright notice and this paragraph are included on all such copies and derivative works. However, this document itself may not be modified in any way, such as by removing the copyright notice or references to the Internet Society or other Internet organizations, except as needed for the purpose of developing Internet standards in which case the procedures for copyrights defined in the Internet Standards process must be followed, or as required to translate it into languages other than English The limited permissions granted above are perpetual and will not be revoked by the Internet Society or its successors or assigns. This document and the information contained herein is provided on an "AS IS" basis and THE INTERNET SOCIETY AND THE INTERNET ENGINEERING TASK FORCE DISCLAIMS ALL WARRANTIES, EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO ANY WARRANTY THAT THE USE OF THE INFORMATION HEREIN WILL NOT INFRINGE ANY RIGHTS OR ANY IMPLIED WARRANTIES OF MERCHANTABILITY OR FITNESS FOR A PARTICULAR PURPOSE. Cerf, et al. Expires September 2003 [Page 24]