Network Working Group D.J. Blackketter Internet-Draft WebTV Networks, Inc. Expires: August 11, 2000 February 11, 2000 The Unidirectional Hypertext Transfer Protocol draft-blackketter-uhttp-00.txt Status of this Memo This document is an Internet-Draft and is NOT offered in accordance with Section 10 of RFC2026, and the author does not provide the IETF with any rights other than to publish as an Internet-Draft. Internet-Drafts are working documents of the Internet Engineering Task Force (IETF), its areas, and its working groups. Note that other groups may also distribute working documents as Internet-Drafts. Internet-Drafts are draft documents valid for a maximum of six months and may be updated, replaced, or obsoleted by other documents at any time. It is inappropriate to use Internet-Drafts as reference material or to cite them other than as "work in progress." The list of current Internet-Drafts can be accessed at http://www.ietf.org/ietf/1id-abstracts.txt The list of Internet-Draft Shadow Directories can be accessed at http://www.ietf.org/shadow.html. This Internet-Draft will expire on August 11, 2000. Abstract This document describes the Unidirectional Hypertext Transfer Protocol, or UHTTP. UHTTP is a simple, robust, one-way data transfer protocol that is designed to efficiently deliver resource data in a one-way broadcast-only environment. This transfer protocol is appropriate for delivery of HTML and other content resources using IP multicast over television vertical blanking interval (IP/VBI) and other unidirectional transport systems. Blackketter Expires August 11, 2000 [Page 1] Internet-Draft UHTTP February 2000 Table of Contents 1. Introduction . . . . . . . . . . . . . . . . . . . . . . . . 3 2. Data Transfer Features Enabled by the UHTTP Protocol . . . . 4 2.1 Robust Delivery: Gathering data over multiple transmissions 4 2.2 Meta-information in the form of HTTP-style headers . . . . . 4 3. UHTTP Header Format . . . . . . . . . . . . . . . . . . . . 5 3.1 Basic UHTTP Header Format . . . . . . . . . . . . . . . . . 5 3.2 UHTTP Extension Header Format . . . . . . . . . . . . . . . 7 3.2.1 HTTPHeaderMap Extension Header . . . . . . . . . . . . . . . 8 4. Forward Error Correction Mechanism . . . . . . . . . . . . . 11 5. HTTP-style headers used in UHTTP . . . . . . . . . . . . . . 13 5.1 Supported HTTP-style headers . . . . . . . . . . . . . . . . 13 6. Packaging more than one resource . . . . . . . . . . . . . . 15 7. Security Considerations . . . . . . . . . . . . . . . . . . 17 References . . . . . . . . . . . . . . . . . . . . . . . . . 18 Author's Address . . . . . . . . . . . . . . . . . . . . . . 18 A. Acknowledgements . . . . . . . . . . . . . . . . . . . . . . 19 Blackketter Expires August 11, 2000 [Page 2] Internet-Draft UHTTP February 2000 1. Introduction The Unidirectional Hypertext Transfer Protocol, or UHTTP, is a simple, robust data transfer protocol that is designed to efficiently deliver resource data in a one-way broadcast-only environment. This transfer protocol is appropriate for delivery of HTML and other content resources using IP multicast over television vertical blanking interval (IP/VBI) and other unidirectional transport systems. The UHTTP protocol is used in the Advanced Television Enhancement Forum specification[6] to deliver television-related content resources which were broadcast along with a television signal. Resources sent using the UHTTP protocol are divided into a set of data segments encapsulated in UDP packets. Typically, these packets are delivered via multicast IP, but this is not required. Each packet contains enough header information to begin capturing the data at any time during the broadcast, even midway through the transfer. This header contains a unique identifier (in the form of an UUID[1]) that uniquely identifies the transfer, and additional information that enables the receiver to place the data following the header in the appropriate location within the transfer. Additional header information indicates to the receiver how long to continue listening for additional data. UHTTP includes the ability to gather segments over multiple retransmissions to correct for missing packets. It is also possible to group resources together for all-or-none delivery within a UHTTP transfer. The protocol also includes a simple forward error correcting mechanism which provides for the ability to restore missing data in the event of limited packet loss. Blackketter Expires August 11, 2000 [Page 3] Internet-Draft UHTTP February 2000 2. Data Transfer Features Enabled by the UHTTP Protocol 2.1 Robust Delivery: Gathering data over multiple transmissions Data can be resent via UHTTP using the same globally unique TransferID. The data is delivered as individual segments, each of which is encapsulated within a UDP message, potentially delivered via IP multicast. Information in the header allows a receiving application to receive segments out of order or multiple times. If the transfer data is sent repeatedly, the receiving service can fill in missing ranges using these retransmissions. This provides robust (though not necessarily reliable) data delivery. Additionally, forward error correction (FEC), using an exclusive-or-based algorithm, provides for recovery of some missing segments in the face of segment loss without retransmission. 2.2 Meta-information in the form of HTTP-style headers The protocol provides for the inclusion of HTTP-style headers preceding the resource data. These headers may include information describing the content type of the resource and content location in the form of a URL. It may also be used to describe groups of resources as a multipart construction. Other meta-information, including date stamping and expiration dates, may be used to provide additional information about the resource content. Blackketter Expires August 11, 2000 [Page 4] Internet-Draft UHTTP February 2000 3. UHTTP Header Format 3.1 Basic UHTTP Header Format This section describes the format of the message packets that carry UHTTP data. It describes the information needed to create the messages using the protocol on the broadcast side and to turn those messages back into resources on the receiving side. The UHTTP header is at the start of every UHTTP IP/UDP datagram. All values are network byte order. The UHTTP header has the following format: 0 1 2 3 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ | version |X|H|C| PcktsInXORBlk | retransmit expiration | +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ | | +- -+ | | +- transfer ID -+ | | +- -+ | | +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ | resource size | +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ | segment start offset | +=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=| | payload or extension header | : .... : The fields are described below. Version : 5 bits Describes the version of the protocol. The protocol described here is version 0. Extension Header (X) : 1 bit When set, this bit indicates that one or more extension header fields are present and immediately follow the main UHTTP header. HTTP Headers Precede (H) : 1 bit Blackketter Expires August 11, 2000 [Page 5] Internet-Draft UHTTP February 2000 A bit flag that, when set to 1, indicates that HTTP-style headers precede the resource data. These HTTP-style headers are considered part of the data when calculating the ResourceSize and SegStartByte fields, as well as for forward error correction. This bit must be set in all packets associated with a UHTTP transfer when HTTP-style headers precede the data. When set to zero, no HTTP-style headers precede the resource data. CRC Follows (C) : 1 bit When the CRCFollows bit is set to 1, a 32 bit CRC is calculated and can be used used to detect possible corruption in the data delivered via UHTTP. Using the MPEG-2 CRC algorithm, the CRC is calculated on the complete data, including HTTP-style headers, if any. It is then appended to the end of the data in the last logical packet. This CRC field is considered part of the data for the purposes of calculating the resource length and calculating the forward error correction. The bit must be set in all packets associated with a UHTTP transfer when a CRC is used. Packets In XOR Block (PcktsInXORBlk) : 1 byte Describes the number of packets in a forward error correction block, including the forward error correction packet. Set to zero when no forward error correction is used. See Section 4 for details on the forward error correction mechanism. Retransmit Expiration : 2 bytes Time in seconds over which the resource may be retransmitted. This indicates how long the receiving software should wait to try to recover missing packets that follow in retransmissions of the same resource. This allows a resource to be carouseled, or sent repeatedly to increase the chances of delivery without missing segments. The RetransmissionExpiration field should be updated to remain accurate during retransmissions, including the current transmission. TransferID : 16 bytes Globally unique identifier (UUID[1]) for the UHTTP transfer. This ID allows receiving software to identify which segments correspond to a given transfer, and determine when retransmission occurs. Resource Size : 4 bytes Size of the complete resource data itself (excluding segment headers, XOR segments and padding for exclusive-or correction). Blackketter Expires August 11, 2000 [Page 6] Internet-Draft UHTTP February 2000 This length does include the length of the HTTP-style headers, if any, as well as the 4-byte CRC, if the CRCFollows bit is set to 1. Segment Start Offset : 4 bytes Start offset for the first byte in the transfer for this data segment. When XOR data is used to replace missing packets, SegStartByte includes the XOR data as well as the resource data, and optional HTTP-style headers and CRC. This allows for determining where all packets fit regardless of delivery order. The exclusive-or correction packet looks like any other UHTTP packet. Its data payload is simply the exclusive-or of a number of packets that precede it in order in the data. The number of packets in an XOR block is specified in the PacketsInXORBlock field described above. The UDP packet data length for the enclosing UDP packet is used to determine the length of the segment. It is permissible to send a packet that contains UHTTP header (and optional extension headers), but without any data. If no data is included, then the SegStartByte field is ignored. 3.2 UHTTP Extension Header Format If the ExtensionHeader flag is set in a UHTTP packet header, additional optional header fields are present. These fields appear directly after main UHTTP header. Extension headers are optional on a packet-by-packet basis, and may appear on none, some or all of the UHTTP packets transmitted, depending on the ExtensionHeaderType. This specification defines a single extension header type, HTTPHeaderMap (defined below). Any extension headers with an unknown type should be ignored by receivers. When the Extension Header (H) is set to a value of one, one or more extension headers may follow the UHTTP header and precede the data segment in that packet. Extension headers have the following format: 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ |F| extension header type | extension header length | +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ | extension header data | | .... | ExtensionHeaderFollows (F) : 1 bit When 1, this field indicates that another extension header Blackketter Expires August 11, 2000 [Page 7] Internet-Draft UHTTP February 2000 follows this one. When 0, the UHTTP data payload follows this extension header. ExtensionHeaderType : 15 bits Identifies the extension header type. (1 = HTTPHeaderMap, all other values reserved). ExtensionHeaderDataSize : 2 bytes Describes the length of the complete Extension Header data in bytes. Zero indicates that there is no ExtensionHeaderData following. ExtensionHeaderData The variable length data for this extension header. The length of the ExtensionHeaderData field is indicated by the ExtensionHeaderDataSize. If the ExtensionHeaderFollows bit is set, then another ExtensionHeader follows this header. If the bit is cleared, then the UHTTP data payload follows the ExtensionHeaderData (if any) immediately. 3.2.1 HTTPHeaderMap Extension Header One ExtensionHeaderType is defined for this specification. When ExtensionHeaderType is set to a value of 1, then the ExtensionHeaderData field contains an HTTPHeaderMap. A HTTPHeaderMap extension header may optionally be included whenever the UHTTP transfer contains HTTP-style header information (as indicated by the HTTPHeadersPrecede bit in the main UHTTP header). If HTTPHeaderMap extension headers are used, they should be included in every packet in a UHTTP transfer that contains header, body or forward error correction (FEC) data. Blackketter Expires August 11, 2000 [Page 8] Internet-Draft UHTTP February 2000 The HTTPHeaderMap consists one or more sets of the following fields: 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ | HTTP header start | +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ | HTTP header size | +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ | HTTP body size | +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ HTTPHeaderStart : 4 bytes This field indicates an offset into the UHTTP data, in bytes, where a HTTP-style header is found. The offset is calculated from the beginning of the corrected UHTTP data, and does not include the FEC data when the FEC mechanism is used. HTTPHeaderSize : 4 bytes This field indicates the length of the HTTP-style header, in bytes, including the HTTP-style header fields, the terminating pair of carriage-return/newline character sequences, and any preceding multipart boundary lines. HTTPBodySize : 4 bytes This field indicates the length of the data body, in bytes, associated with the HTTP header described in this map entry. When the UHTTP transfer consists of a single (i.e. non-multipart) resource, a single 12 bytes set of HTTPHeaderMap fields is present in the HTTPHeaderMap. The HTTPHeaderStart, in this case, will be set to zero and the HTTPHeaderSize will be set to the sum of the length of the HTTP-style header fields and all separating carriage-return/newline characters. The HTTPBodySize field will contain the size, in bytes, of the body data related to that header field. When a UHTTP transfer contains multiple resources (as specified by a multipart content-type), multiple sets of HTTPHeaderMap groups may be included in the HTTPHeaderMap data, each indicating the offset and size of the HTTP-style headers for each resource, (including any multipart boundary lines, HTTP-style header fields and separating carriage-return/newline characters), as well as the size of the body relating to each header. Blackketter Expires August 11, 2000 [Page 9] Internet-Draft UHTTP February 2000 When including HTTPHeaderMap data, senders must at a minimum include HTTPHeaderMap entries for each HTTP-style header that is partially or completely included in a given packet. Additionally, when forward error correction is used in UHTTP transfers that contain HTTPHeaderMaps extension headers, senders must include HTTPHeaderMap entries as extension headers in FEC-data packets for all HTTP-style header sections that may be corrected by the FEC packet. Senders are free to include additional HTTPHeaderMap entries in any packet beyond the minimum. Blackketter Expires August 11, 2000 [Page 10] Internet-Draft UHTTP February 2000 4. Forward Error Correction Mechanism In addition to the ability to retransmit data and have missing segments filled in during retransmissions, this transfer protocol can also include extra data packets that can be used for simple missing packet error correction. When the PacketsInXORBlock header field is set to zero, there is no exclusive-or forward error correction. When non-zero, forward error correction packets may be sent to allow for correction of missing or corrupted packets by the receiver. It is permissible to send packets with no data payload (but with UHTTP headers and optional extension headers). In this case, the packet is ignored in the calculation of forward error correction. In blocks of PacketsInXORBlock equal size data segments, the last data segment in the block contains the exclusive-or of the preceding segments (PacketsInXORBlock - 1). Each byte of the data in this "XOR segment" is the exclusive-or of the corresponding byte in each of the other segments in that data block. If the data is thought of as laid out separated into consecutive segments, then after every PacketsInXORBlock - 1 segments another segment is inserted that looks exactly like resource data and has its own position offset into the transfer like resource data. The data in that segment is the exclusive-or of the previous packets in that block. Data in the XOR segment is the exclusive-or of data segment contents only, including the HTTP-style header fields but not including the UHTTP header that is also in the packet. If forward error correction is used, the data payload of all packets must be the same size. The packet containing the data at the end of the file (including the optional CRC) must be zero filled. Packets between this packet and the last XOR packet need not be sent since the receiver knows their contents are all zeros, as it was provided the overall length in the UHTTP resource size field. If they are sent they must be zero filled after the segment header; if not sent, they are assumed to contain a payload of all zero bytes for the purposes of forward error correction calculations. The last XOR packet has the value SegStartByte calculated to be just as if zero-filled extra packets were sent, but there is no requirement to send those empty packets. To correct for a single missing packet in a block, the receiver should calculate the exclusive-or the data payload of the packets that arrived with the XOR data segment for that block. An important point is that segments can be sent in any order since each segment including the XOR segment indicate where in order they belong. By sending segments (including the XOR packets) out of order, there may be protection against burst errors that lose successive packets. When re-transmitting a UHTTP transfer, a different order of segments Blackketter Expires August 11, 2000 [Page 11] Internet-Draft UHTTP February 2000 can be used in each retransmission to avoid different types of burst errors. This protocol allows the headend (broadcast side) tools to decide how to order sending packets, thereby providing a great deal of flexibility. The receiving side does not need to know the transmission order (consistent with the fact that in general it cannot know the transmission order for IP multicast delivery). Blackketter Expires August 11, 2000 [Page 12] Internet-Draft UHTTP February 2000 5. HTTP-style headers used in UHTTP The UHTTP transfer protocol can be used to deliver resources via a broadcast medium, which can simultaneously deliver resources, including web-related content, to large numbers of users simultaneously. HTTP-style header fields can be included in the UHTTP data to provide meta-information about the content, including identifying URIs, expiration dates and content type and encoding. The use of HTTP-style headers is optional in UHTTP, but is required for resources intended to be interpreted as web content. HTTP-style headers, as specified by HTTP 1.1[5], precede the resource content just as in HTTP transfers when resources are sent as a response to a HTTP GET or POST command. The HTTP-style headers may provide additional information to the browser, for example, the expiration time for the resource. The HTTP-style headers precede the body of the resource data, and are treated as part of the data payload. The protocol header and its version imply the equivalent HTTP response line (e.g. "HTTP/1.1 200 OK"). Relevant header fields are listed below and should be interpreted per the HTTP 1.1[5] specification. 5.1 Supported HTTP-style headers Header fields required for every resource when using HTTP-sytle headers: Content-Length: Content-Location: Recommended HTTP-style header fields: Content-Type: Blackketter Expires August 11, 2000 [Page 13] Internet-Draft UHTTP February 2000 Optional HTTP-style header fields: Content-Base: Content-Encoding: Content-Language: Content-Style-Type: Date: Expires: Last-Modified: UHTTP transfers that utilize HTTP-style headers MUST contain the required headers listed above ("Content-location" and "Content-length"), to ensure that an appropriate URI is specified for the resource and to ensure that the resource data is delivered in whole. No other header fields are strictly required, but may provide useful information to the receiver in determining the disposition of the content. Receivers will decode the headers and data and store them in a local cache system. Different receiver platforms will have different cache sizes for storing local resources, which may or may not correspond to traditional web browser caches. Resources transmitted with "Content-location" header fields which contain "http:" URLs identify resources which are also available via an HTTP request specified by that URL. The use of "Content-Location:" headers with local identifier[3], or "lid:" URLs is intended to mirror resource delivery to a local cache without requiring that the data be available on the Internet. Receiving platforms should take into consideration that the same resources will likely be sent repeatedly to provide resources for users who tune in late. HTTP-style header fields can be examined to determine if the resource is already present, and so can be ignored. The "Date:", "Expires:", and "Last-Modified:" headers can be used to determine the lifetime of a resource in a given receiver's cache. Blackketter Expires August 11, 2000 [Page 14] Internet-Draft UHTTP February 2000 6. Packaging more than one resource It is possible to send multiple resources in a single UHTTP transfer by packaging them into a single multipart resource for all-or-nothing transfer. In this case, the HTTP-style "Content-Type:" header field would have a value of "multipart/related"[2]. When this type is used, the HTTP-style header is ended as usual and is followed by the usual boundary structure for "multipart/related" separating multiple related resources that each use the HTTP-style header formats. This is a mechanism to package multiple related resources together in a single all-or-nothing transfer. The HTTP-style headers for individual sub-parts describe only that sub-part, and are interpreted as per the HTTP 1.1 specification[5]. In this case, it may be convenient to specify a "Content-Base:" for the entire package and then specify relative URLs for each of the "Content-Location:" headers for subsequent subparts. The "multipart/related" content type should be used as per RFC 2387[2]. An example using HTTP scheme URLs: Content-Base: http://www.blahblah.com/ Content-Length: 3495 Content-Type: Multipart/Related; boundary=example98203804805 --example98203804805 Content-Location: resource1.html Content-Length: 495 Content-Type: text/html ...Resource data for resource1.html... ...... --example98203804805 Content-Location: image1.jpg Content-Length: 1495 Content-Type: image/jpeg ...Resource data for image1.jpg... --example98203804805 Blackketter Expires August 11, 2000 [Page 15] Internet-Draft UHTTP February 2000 A similar example using "lid:" style URLs and relative URLs: Content-Base: lid://unique2345@blahblah.com/ Content-Length: 3495 Content-Type: Multipart/Related; boundary=example98203804805 --example98203804805 Content-Location: resource1.html Content-Length: 495 Content-Type: text/html ...Resource data for resource1.html... ...... --example98203804805 Content-Location: image.jpg Content-Length: 1495 Content-Type: image/jpeg ...Resource data for image1.jpg... --example98203804805 Blackketter Expires August 11, 2000 [Page 16] Internet-Draft UHTTP February 2000 7. Security Considerations There are a number of security issues associated with the use of UHTTP to deliver content on public broadcast networks, many of which are shared with electronic mail systems. When HTTP-style headers are used to identify the resources in a UHTTP transfer, it is possible for the sender to misrepresent the content with URIs which do not match the transmitted content. The security issues associated with this mislabeling, as well as the security issues associated with the use of HTML content which is broadcast are the same as those identified in section 11.1 of RFC 2387[2]. Additionally, there are security issues associated with the caching of data transmitted via UHTTP which are the same as those identified in section 11.2 of RFC 2387[2]. It should be noted that many broadcast systems employ conditional access systems (satellite and cable television, for example) which provides a level of security for the broadcast channel leading up to the receiver. Unfortunately, it may be possible to insert UHTTP data earlier in the broadcast chain at points which are less secure (on videotape to be played into the system, for example), so applications using UHTTP may which cannot ensure end-to-end security of the broadcast system should employ additional security mechanisms. Blackketter Expires August 11, 2000 [Page 17] Internet-Draft UHTTP February 2000 References [1] Leach, P. J. and R. Salz, "UUIDs and GUIDs", Internet Draft draft-leach-uuids-guids-01, February 1998. [2] Levinson, E., "The MIME Multipart/Related Content-type", RFC 2387, August 1998. [3] Blackketter, D., "The Local Identifier URL Scheme", Internet Draft draft-blackketter-lid-00, August 1999. [4] Technical committee / subcommittee: JTC 1 / SC 29, "ISO/IEC 13818-1:1996, GENERIC CODING OF MOVING PICTURES AND ASSOCIATED AUDIO INFORMATION - PART 1: SYSTEMS, Annex A: CRC Decoder Model", ISO/IEC 13818-1:1996, January 1996. [5] Fielding, R., "Hypertext Transfer Protocol -- HTTP/1.1", RFC 2068, January 1997. [6] ATVEF, "Advanced Television Enhancement Forum Specification, draft 1.1r26", February 1999. Author's Address Dean J. Blackketter WebTV Networks, Inc. 1295 Charleston Road Mountain View, CA 94043 US Phone: +1 650 614 5521 EMail: dean@corp.webtv.net URI: http://www.webtv.net/ Blackketter Expires August 11, 2000 [Page 18] Internet-Draft UHTTP February 2000 Appendix A. Acknowledgements The author gratefully acknowledges the contributions of the ATVEF Technical Working Group, and in particular: Lee Acton, Jonathan Boltax, Wayne Carr, Michael Dolan, CJ Frederickson, Iain Hackett, Cheryl Kadis, David Mott, Scott Watson, Mark Vickers, and Dan Zigmond. Blackketter Expires August 11, 2000 [Page 19]