Internet Engineering Task Force A. Ford Internet-Draft Roke Manor Research Intended status: Experimental M. Handley Expires: January 7, 2010 University College London July 6, 2009 HTTP Extensions for Simultaneous Download from Multiple Mirrors draft-ford-http-multi-server-00 Status of this Memo This Internet-Draft is submitted to IETF in full conformance with the provisions of BCP 78 and BCP 79. Internet-Drafts are working documents of the Internet Engineering Task Force (IETF), its areas, and its working groups. Note that other groups may also distribute working documents as Internet- Drafts. Internet-Drafts are draft documents valid for a maximum of six months and may be updated, replaced, or obsoleted by other documents at any time. It is inappropriate to use Internet-Drafts as reference material or to cite them other than as "work in progress." The list of current Internet-Drafts can be accessed at http://www.ietf.org/ietf/1id-abstracts.txt. The list of Internet-Draft Shadow Directories can be accessed at http://www.ietf.org/shadow.html. This Internet-Draft will expire on January 7, 2010. Copyright Notice Copyright (c) 2009 IETF Trust and the persons identified as the document authors. All rights reserved. This document is subject to BCP 78 and the IETF Trust's Legal Provisions Relating to IETF Documents in effect on the date of publication of this document (http://trustee.ietf.org/license-info). Please review these documents carefully, as they describe your rights and restrictions with respect to this document. Abstract This document describes an extension to HTTP by which servers can automatically inform clients of mirrors of resources. Clients can Ford & Handley Expires January 7, 2010 [Page 1] Internet-Draft Multi-server HTTP July 2009 then simultaneously request segments of the resource from different servers, enhancing both network and server utilisation, download speeds, and thus user experience. Table of Contents 1. Introduction and Motivation . . . . . . . . . . . . . . . . . 3 1.1. Operation Overview . . . . . . . . . . . . . . . . . . . . 3 1.2. Requirements Language . . . . . . . . . . . . . . . . . . 4 2. HTTP Extension Headers . . . . . . . . . . . . . . . . . . . . 4 2.1. Overview . . . . . . . . . . . . . . . . . . . . . . . . . 4 2.2. Requests . . . . . . . . . . . . . . . . . . . . . . . . . 5 2.2.1. X-Multiserver-Version . . . . . . . . . . . . . . . . 5 2.2.2. X-If-Checksum-Match . . . . . . . . . . . . . . . . . 5 2.3. Responses . . . . . . . . . . . . . . . . . . . . . . . . 6 2.3.1. X-Multiserver-Version . . . . . . . . . . . . . . . . 6 2.3.2. X-Checksum . . . . . . . . . . . . . . . . . . . . . . 6 2.3.3. X-Mirrors . . . . . . . . . . . . . . . . . . . . . . 6 3. Example Operation . . . . . . . . . . . . . . . . . . . . . . 7 4. Full Interaction Specification . . . . . . . . . . . . . . . . 8 4.1. Error Handling . . . . . . . . . . . . . . . . . . . . . . 8 4.1.1. Checksum Failure . . . . . . . . . . . . . . . . . . . 8 5. Alternative Approaches . . . . . . . . . . . . . . . . . . . . 9 6. Heuristics and Optimizations . . . . . . . . . . . . . . . . . 9 7. Managing Server Load . . . . . . . . . . . . . . . . . . . . . 10 8. Security Considerations . . . . . . . . . . . . . . . . . . . 10 9. Acknowledgements . . . . . . . . . . . . . . . . . . . . . . . 10 10. IANA Considerations . . . . . . . . . . . . . . . . . . . . . 11 11. References . . . . . . . . . . . . . . . . . . . . . . . . . . 11 11.1. Normative References . . . . . . . . . . . . . . . . . . . 11 11.2. Informative References . . . . . . . . . . . . . . . . . . 11 Authors' Addresses . . . . . . . . . . . . . . . . . . . . . . . . 11 Ford & Handley Expires January 7, 2010 [Page 2] Internet-Draft Multi-server HTTP July 2009 1. Introduction and Motivation Mirrored HTTP servers are regularly used for software downloads, whereby copies of data to be downloaded are duplicated on many servers distributed around the Internet. Users are encouraged to manually choose a nearby mirror from which to download. This is intended to increase both throughput and resilience, and reduce load on individual servers. Manual mirror choice rarely works well; users do not wish to make a choice, but if they are not forced to, then the default server takes a disproportionate share of the load. Even when they are forced to choose, they rarely have enough information to choose the server that will provide the best performance. Some popular sites automate this process using DNS load balancing, both to approximately balance load between servers, and to direct clients to nearby servers with the hope that this improves throughput. Indeed, DNS load balancing can balance long-term server load fairly effectively, but it is less effective at delivering the best throughput to users when the bottleneck is not the server but the network. This document specifies an alternative mechanism by which the benefit of mirrors can be automatically and more efficiently realised. These benefits are achieved using a number of extensions to HTTP which allow the discovery of mirrors, the verification of the integrity of files on each mirror, and the simultaneous downloading of chunks from multiple mirrors. The use of this mechanism allows greater efficiency in resource utilisation in the Internet as a whole, balances server utilization, even on short timescales, and enhances user experience through faster downloads. 1.1. Operation Overview In an HTTP request a client will first declare that it conforms to multi-server HTTP extensions. The capable server that wishes to take advantage of mirroring for the file requested will respond in its headers with a list of mirrors for the given file, and it will also return a checksum for the file. Multi-server HTTP extensions require the client and server support persistent connections. A server responding with a mirror list uses chunked encoding for the response to permit the client to request ranges of bytes from the file from different servers. It is normally up to the client to decide on the ranges requested from each mirror. However to avoid wasted time when talking to the initial server, this server starts to return data immediately, and it is up to the server to determine this initial chunk size. Ford & Handley Expires January 7, 2010 [Page 3] Internet-Draft Multi-server HTTP July 2009 The server will make an estimate of a sensible initial chunk size, and immediately respond with this amount of the file (a Content- Range: header declares how much is sent in this chunk). The mechanism for determining the initial chunk size is up to the server implementation. However some suggestions include: it could be determined by a bounded proportion of the file size, or based on the RTT of the initial handshake, or based on cached information on previous throughput to the same client, or based on current throughput achieved by the server, or a combination of all of these. We will discuss these heuristics in more detail below It is then up to the client to work out appropriate scheduling for retrieving further chunks of this file from the original server and the mirrors. The initial server will continue sending its own scheduled chunks until a GET request is received containing the standard HTTP Range: header, and at this point the server defers to the client's requests. It is up to the client to choose how many servers to use, and what size chunks to request from each. An advanced client is expected to adapt to relative speeds of servers and utilise the bandwidth most effectively, and to pipeline range requests to these servers to avoid idle time on the connections. The client will also include a new checksum verification header to ensure the file on the mirror is the same as from the initial server. 1.2. Requirements Language The key words "MUST", "MUST NOT", "REQUIRED", "SHALL", "SHALL NOT", "SHOULD", "SHOULD NOT", "RECOMMENDED", "MAY", and "OPTIONAL" in this document are to be interpreted as described in RFC 2119 [RFC2119]. 2. HTTP Extension Headers 2.1. Overview The additional functionality required at the HTTP level can be broken down as follows: o Ensure client and server operate same version of Multi-Server HTTP o Retrieve and verify a checksum for a resource o Identify alternatives servers that provide access to the same resource o Retrieve segments of a resource From these requirements, the additional headers required can be Ford & Handley Expires January 7, 2010 [Page 4] Internet-Draft Multi-server HTTP July 2009 derived, and are described in the following sections. NOTE WELL: The names of the extension headers given below are TEMPORARY and FOR DISCUSSION ONLY. These will evolve to meet [RFC2774] if appropriate. The formal definitions below make use of the conventions defined in Section 2.2 of [RFC2616]. 2.2. Requests 2.2.1. X-Multiserver-Version This is used in both request and response to indicate that the host is has multi-server HTTP capability and what version it operates. For this specification, this version field MUST read "0.1". Formally, X-Multiserver-Version = "X-Multiserver-Version" ":" 1*DIGIT "." 1*DIGIT Example: X-Multiserver-Version: 0.1 2.2.2. X-If-Checksum-Match This option is used in a request as a conditional, where the request is only valid if the checksum of the resource matches that specified in this header. The checksum is that already provided by the X-Checksum header in the response (see below). Two checksum functions are provided in this specification, and a compliant implementation MUST implement both. These are MD5 [RFC1321] and SHA-256 [FIPS180-3]. Formally, checksum = quoted-string sum-type = "MD5" | "SHA-256" X-If-Checksum-Match = "X-If-Checksum-Match" ":" 1 sum-type 1 checksum Example: Ford & Handley Expires January 7, 2010 [Page 5] Internet-Draft Multi-server HTTP July 2009 X-If-Checksum-Match: MD5 \ "b1946ac92492d2347c6235b4d2611184" 2.3. Responses 2.3.1. X-Multiserver-Version As for a request, this handshake ensures both hosts will speak the same protocol. 2.3.2. X-Checksum This option is presented by the server to provide a checksum of the whole file, which MUST be used in the X-If-Checksum-Match option to other mirrors. It SHOULD be used by a client to verify the file once all segments have been downloaded. Formally, X-Checksum = "X-Checksum" ":" 1 sum-type 1 checksum Example: X-Checksum: MD5 \ "b1946ac92492d2347c6235b4d2611184" 2.3.3. X-Mirrors This option lists the orginal file, the cache lifetime in seconds, and the list of all mirrors available for the requested URI. This header once per mirror. If the original file ends in a slash ("/"), it indicates that the mirrors will provide all resources under this directory. If so, then all the mirrors MUST also end in a slash. If it does not specify a directory, but instead ends in a file name, then this declares that the mirror exists only for this file. Any further requests for different files must first go to this server to acquire a mirror list, if it exists. The cache lifetime is the time which this mirrored information can be cached for, before it SHOULD be considered stale and SHOULD not be used. Caching the mirror list is most important for mirrors of directories, where small files can be requested directly from the mirrors without needing to request the first chunk from the initial server. Formally: X-Mirrors="X-Mirrors" ":" filename cachetime 1*absoluteURI Ford & Handley Expires January 7, 2010 [Page 6] Internet-Draft Multi-server HTTP July 2009 Example: X-Mirrors: /wibble/download.zip 3600 \ http://www.example2.com/wibble/download.zip \ http://www.example3.com/wibble/download.zip 3. Example Operation So far, this document has defined the functional components that enable multi-server HTTP. This section will draw out an example of how these fit together, and utilising existing HTTP/1.1 features. First, a Multi-Server HTTP client will connect to a server and request a file, while declaring itself multi-server HTTP capable: GET /wibble/download.zip HTTP/1.1 Host: www.example.com X-Multiserver-Version: 0.1 The server will respond with details about this resource, and begin delivering some of the file: HTTP/1.1 200 OK Accept-Ranges: bytes Content-Length: 10240 Content-Type: application/zip Content-Range: bytes 1-10240/2025121 X-Multiserver-Version: 0.1 X-Checksum: MD5 "d6862c992a3d6736ad678cc865dee67f" X-Mirrors: /wibble/download.zip 3600 \ http://www.example2.com/wibble/download.zip \ http://www.example3.com/wibble/download.zip ...data... The client will then continue requesting more blocks from this server: GET /wibble/download.zip HTTP/1.1 Host: www.example.com X-Multiserver-Version: 0.1 Range: 10241-20480 The server will then respond with further blocks of the file: Ford & Handley Expires January 7, 2010 [Page 7] Internet-Draft Multi-server HTTP July 2009 HTTP/1.1 200 OK Accept-Ranges: bytes Content-Length: 10240 Content-Type: application/zip Content-Range: bytes 10241-20480/2025121 X-Multiserver-Version: 0.1 ...data... It will then also connect to the mirrors and request more blocks, such as the following: GET /wibble/download.zip HTTP/1.1 Host: www.example2.com X-Multiserver-Version: 0.1 X-If-Checksum-Match: MD5 "d6862c992a3d6736ad678cc865dee67f" Range: 20481-30720 Assuming the checksum matches correctly, the server will respond as for a normal multiserver HTTP continuation request from the original server. When finished, the client can close all its connections. 4. Full Interaction Specification TBD. This section will define permitted HTTP status codes, behaviour upon errors, etc. 4.1. Error Handling Multi-server HTTP introduces a number of failure modes that do not exist in conventional HTTP. 4.1.1. Checksum Failure It should not be the case that the data checksum fails when the file has finished downloading, as the mirrors MUST NOT return data if it does not correspond to the version of the file specified in the checksum. However, if this occurs due to bugs or hardware data corruption, there is no way to determine which chunks are errored. It would be possible to add mechanism to check with the original server to determine what the checksum would be for a range, but this would add extra complexity for a case that is expected to be extremely rare. In the case of a checksum failure, a client SHOULD re-request any chunks received from mirrors from the original server, but recalculate the file checksum periodically to determine if the Ford & Handley Expires January 7, 2010 [Page 8] Internet-Draft Multi-server HTTP July 2009 errored chunk has now been replaced. 5. Alternative Approaches It is debatable whether ETags could be used in place of checksums, along with the If-Match header in requests. We elected not to follow this as the definition of an ETag is that it must be unique to a resource, however in this case we are accessing multiple resources (URIs) even though they should be the same data. We do not feel comfortable using ETags as this would require the semantics of the field to be changed. 6. Heuristics and Optimizations The performance of Multipath HTTP depends heavily on several factors: o The number of servers used simultaneously. o The ability to pipeline sufficient or sufficiently large range requests to each server so as to avoid connections going idle. o The ability to pipeline sufficiently few or sufficiently small range requests to servers so that all the servers finish their final chunks simultaneously. o The ability to switch between mirrors dynamically so as to use the fastest mirrors at any moment in time Obviously we do not want to use too many simultaneous connections, or other traffic sharing a bottleneck link will be starved. But at the same time, good performance requires that the client can simultaneously download from at least one fast mirror while exploring whether any other mirror is faster. Based on laboratory experiments, we suggest a good default number of simultaneous connections is probably four, with three of these being used for the best three mirrors found so far, and one being used to evaluate whether any other mirror might offer better performance. The size of chunks chosen by the client should be sufficiently large that the chunk request headers and reponse headers represent neglible overhead, and sufficiently large that they can be pipelined effectively without needing a very high rate of chunk requests. At the same time, the amount of time wasted waiting for the last chunk to download from the last server after all the other servers have finished should be minimized. Thus we currently recommend that a chunk size of at least 10KBytes should be used. If the file being Ford & Handley Expires January 7, 2010 [Page 9] Internet-Draft Multi-server HTTP July 2009 transfered is very large, or the download speed very high, this can be increased to perhaps 1MByte. As network bandwidths increase, we expect these numbers to increase appropriately, so that the time to transfer a chunk remains significantly larger than the latency of requesting a chunk from a server. 7. Managing Server Load A goal of mirroring is to manage server load, not just network bandwidth. Thus server operators need simple mechanisms they can use to handle overload conditions gracefully and balance load. Multi- server HTTP provides spreads load automatically across heterogeneous servers so that the fastest servers get to serve the most data. However, if a server is approaching an overload condition, Multi- server HTTP can increase the overall load a little due to the additional processing of chunk requests. Thus, a way for such a server to discard load is desirable. The easiest way to shed load is to use two ports for HTTP; one is used for the initial requests to a server, and the second is used to handle requests for subsequent chunks redirected from other servers. This second port would then be explicitly reported in the URLs specified in the X-Mirrors option. A server that is approaching overload can continue to report mirrors in initial requests sent to it (thus directing load to other mirrors), but simply stop listening on the second port. This avoids it having to handle requests redirected to it from other servers until its load returns to a more manageable level. In the event that all the mirrors are overloaded, this effectively falls back to conventional HTTP, with only requests sent to the original port being serviced. 8. Security Considerations At present, we do not believe there should be any new security issues with this approach. The use of checksums will ensure that the resource on every server is the same, and the final result can also be verified. One concern could be regarding whether we should permit redirects to be used as responses on a mirror request. Although this is deviating from the mirrored resource as originally specified by the initial server, the use of checksums will allow the final file to be verified. 9. Acknowledgements This work builds upon work undertaken by Javier Vela Diago, who also provided validation of the benefits of this approach. It is based loosely on the BitTorrent algorithm, but adapted to the HTTP client/ Ford & Handley Expires January 7, 2010 [Page 10] Internet-Draft Multi-server HTTP July 2009 server architecture. The authors are supported by Trilogy (http://www.trilogy-project.org), a research project (ICT-216372) partially funded by the European Community under its Seventh Framework Program. The views expressed here are those of the author(s) only. The European Commission is not liable for any use that may be made of the information in this document. 10. IANA Considerations None. 11. References 11.1. Normative References [RFC2119] Bradner, S., "Key words for use in RFCs to Indicate Requirement Levels", BCP 14, RFC 2119, March 1997. 11.2. Informative References [FIPS180-3] National Institute of Standards and Technology (NIST), "FIPS 180-3 Secure Hash Standard (SHS)", October 2008, . [RFC1321] Rivest, R., "The MD5 Message-Digest Algorithm", RFC 1321, April 1992. [RFC2616] Fielding, R., Gettys, J., Mogul, J., Frystyk, H., Masinter, L., Leach, P., and T. Berners-Lee, "Hypertext Transfer Protocol -- HTTP/1.1", RFC 2616, June 1999. [RFC2774] Nielsen, H., Leach, P., and S. Lawrence, "An HTTP Extension Framework", RFC 2774, February 2000. Ford & Handley Expires January 7, 2010 [Page 11] Internet-Draft Multi-server HTTP July 2009 Authors' Addresses Alan Ford Roke Manor Research Old Salisbury Lane Romsey, Hampshire SO51 0ZN UK Phone: +44 1794 833 465 Email: alan.ford@roke.co.uk Mark Handley University College London Gower Street London WC1E 6BT UK Email: m.handley@cs.ucl.ac.uk Ford & Handley Expires January 7, 2010 [Page 12]