Network Working Group O. Finkelman Internet-Draft Qwilt Intended status: Informational A. Begen Expires: May 8, 2019 Networked Media November 4, 2018 HTTP Redirects in HTTP Adaptive Streaming draft-finkelman-httpbis-has-redirecting-00 Abstract This document motivates the need for clarifications of client behavior in cases of HTTP redirect to a content delivery network (CDN) when the redirected object contains relative references. This document focuses on HTTP Adaptive Streaming (HAS) use cases, but it might be possible to generalize to other use cases. The goal of this document is to present the current status of things and to explore potential solutions. Status of This Memo This Internet-Draft is submitted in full conformance with the provisions of BCP 78 and BCP 79. Internet-Drafts are working documents of the Internet Engineering Task Force (IETF). Note that other groups may also distribute working documents as Internet-Drafts. The list of current Internet- Drafts is at https://datatracker.ietf.org/drafts/current/. Internet-Drafts are draft documents valid for a maximum of six months and may be updated, replaced, or obsoleted by other documents at any time. It is inappropriate to use Internet-Drafts as reference material or to cite them other than as "work in progress." This Internet-Draft will expire on May 8, 2019. Copyright Notice Copyright (c) 2018 IETF Trust and the persons identified as the document authors. All rights reserved. This document is subject to BCP 78 and the IETF Trust's Legal Provisions Relating to IETF Documents (https://trustee.ietf.org/license-info) in effect on the date of publication of this document. Please review these documents carefully, as they describe your rights and restrictions with respect to this document. Code Components extracted from this document must Finkelman & Begen Expires May 8, 2019 [Page 1] Internet-Draft HTTP Redirects in HTTP Adaptive Streaming November 2018 include Simplified BSD License text as described in Section 4.e of the Trust Legal Provisions and are provided without warranty as described in the Simplified BSD License. Table of Contents 1. Introduction . . . . . . . . . . . . . . . . . . . . . . . . 2 2. Glossary of Terms . . . . . . . . . . . . . . . . . . . . . . 3 3. Current State of Things . . . . . . . . . . . . . . . . . . . 3 4. Objectives . . . . . . . . . . . . . . . . . . . . . . . . . 4 5. Potential Solutions . . . . . . . . . . . . . . . . . . . . . 4 6. Examples and Interpretation of the Current RFCs . . . . . . . 5 6.1. Example 1: Absolute URI to Manifest . . . . . . . . . . . 6 6.2. Example 2: Relative Reference in an Encapsulating Entity 7 6.3. Example 3: Relative Reference with Redirect . . . . . . . 8 7. Conclusion . . . . . . . . . . . . . . . . . . . . . . . . . 8 8. IANA Considerations . . . . . . . . . . . . . . . . . . . . . 9 9. Security Considerations . . . . . . . . . . . . . . . . . . . 9 10. Acknowledgements . . . . . . . . . . . . . . . . . . . . . . 9 11. Contributors . . . . . . . . . . . . . . . . . . . . . . . . 9 12. References . . . . . . . . . . . . . . . . . . . . . . . . . 9 12.1. Normative References . . . . . . . . . . . . . . . . . . 9 12.2. Informative References . . . . . . . . . . . . . . . . . 10 Authors' Addresses . . . . . . . . . . . . . . . . . . . . . . . 10 1. Introduction HTTP redirect, commonly using the "302 Found" response code, is widely used in Content Delivery Networks (CDN) for HTTP Adaptive Streaming (HAS), specifically for HTTP Live Streaming (HLS) [RFC8216] and Dynamic Adaptive Streaming over HTTP (DASH) [MPEG-DASH]. Using HTTP redirect rather than delegation by CNAME is a more powerful and flexible solution. At a high level, the required behavior from a HAS client is that after getting an HTTP redirect to a playlist address that takes the client to a CDN cache, if the playlist contains relative references to the media segments, the client should be 'sticky' and request the subsequent media segments directly from the cache without the need for repeated redirects, which naturally would add undesirable latency. Unfortunately, in some cases, common misinterpretations of the RFC lead to wrong behavior of clients, while in other cases, this required behavior contradicts the HTTP RFC. Thus, we are facing issues with streaming clients that behave differently under different use cases of HTTP redirects when relative references are in use within the playlist. Finkelman & Begen Expires May 8, 2019 [Page 2] Internet-Draft HTTP Redirects in HTTP Adaptive Streaming November 2018 The Streaming Video Alliance (SVA) and DASH Industry Forum (DASH-IF) are attempting to resolve these issues and create clear guidelines describing how video players, browsers and streaming applications should behave in such HTTP redirect cases. A discussion of this use case can also be found in Section 2.2.1 of [RFC6983]. The following sequence explains this step by step: 1. The client gets a URI to the playlist file from the origin website (content provider). This URI may point directly at a CDN address or be CNAMEd to a CDN node. 2. The client requests the playlist file from the CDN. 3. The CDN responds with an HTTP redirect, or a series of redirects that eventually takes the client to the final cache. 4. The client requests the playlist from this final cache. 5. The client parses the relative URIs to media segments from the playlist. 6. The client uses the final cache address to construct the absolute URIs for the media segments that point to the final cache. 7. The client requests the media segments from the final cache. The redirection should still be temporary in the sense that if a client does a refresh, or repeatedly fails to get the playlist or the media files from the cache address, it can go back to the original playlist URI and request it again. 2. Glossary of Terms The key words "MUST", "MUST NOT", "REQUIRED", "SHALL", "SHALL NOT", "SHOULD", "SHOULD NOT", "RECOMMENDED", "NOT RECOMMENDED", "MAY", and "OPTIONAL" in this document are to be interpreted as described in BCP 14 [RFC2119] [RFC8174] when, and only when, they appear in all capitals, as shown here. 3. Current State of Things Currently, there are inconsistencies in the behaviors of HAS clients when constructing the URI of a relative referenced object, read from another resource, which was redirected. In some cases the client constructs these URI relative to the redirected address, while in Finkelman & Begen Expires May 8, 2019 [Page 3] Internet-Draft HTTP Redirects in HTTP Adaptive Streaming November 2018 other cases the client constructs it relative to the original URI of the playlist before the redirection. The result of the latter behavior is that the client returns to the original host for each and every request, and then gets redirected again to the cache address. This has a significant negative impact on the the user experience (due to increased risk of slower media segment downloads). Further, it causes redundant work in the redirecting CDN host. The reasons for this are, in some cases, misinterpretation of the standard and, in other cases, contradictions between this requirement and the existing standard. 4. Objectives Our objectives are to document a clear understanding when a client SHOULD compute the absolute URI for relative reference taken from a playlist using the actual retrieval URI of the playlist, and have a specification that leads to a consistent behavior that meets the HAS requirement, regardless of how the retrieval URI of the playlist was resolved. 5. Potential Solutions The below are high-level initial thoughts in order to get the feedback from the HTTPbis working group. As we see it, the alternatives could be one of the following, depending on the understanding of the current state of things: 1. Have a single consistent interpretation of the existing RFCs showing in which cases the behavior should be as required above. 2. Draft an extension to Section 5 of [RFC3986] to handle the required cases. 3. Create a new HTTP 3xx response code that will be specified to serve the required use cases. 4. Handle it outside the HTTP standard, i.e., in the SVA and DASH- IF. A solution should take under consideration the more advanced use cases of: 1. A chain of redirects to get to the final cache. 2. Redirect back: What happens if there is a need to redirect the client back to the original URI? How does the client behave Finkelman & Begen Expires May 8, 2019 [Page 4] Internet-Draft HTTP Redirects in HTTP Adaptive Streaming November 2018 after such redirect of a specific media object? Is there any difference between this case and another forward redirection? 3. An absolute URI appearing in the middle of a playlist, between relative references (for example, for an advertisement segment) should not break the sequence of things, it should be treated as an absolute URI and the subsequent segments that are relative references should be requested from the cache again. 4. Failover: If the cache does not answer, the client should go back and request the playlist from the original URI, and not request the relative reference object directly as it does not know how to construct an alternative URI for it. 5. The client MAY, on its own consideration, re-request the playlist again from the original URI, and may be redirected again to the same cache, or other cache. In that sense the redirect is temporary. 6. The solution should handle both the use case in which the encapsulating entity was redirected and it includes relative references to the media segment, as in HLS (Section 5.1.3 of [RFC3986]), and the use case that encapsulating entity includes the base URL embedded in it, as in DASH (Section 5.1.1 of [RFC3986]), which may lead to the understanding that DASH services should avoid using an absolute URI as base URL in the Media Presentation Description (MPD), if they wish HTTP redirect to CDN to work properly. 6. Examples and Interpretation of the Current RFCs This is our interpretation of the current state of things using HLS manifest examples, in each example the manifest URI is resolved differently and we observe the implications on the resolution of the relative references contained within the manifest file. In the first example the master manifest URI is given as an absolute URI, while in the second example the master manifest is given itself as a relative reference from the encapsulation that provided it, both lead to the required behavior. In the third example, similar to the second, the manifest is given as a relative reference but the request for the manifest is redirected. This leads to the undesired behavior of the client going back to the original location for every request, requiring redirect for every media chunk. Finkelman & Begen Expires May 8, 2019 [Page 5] Internet-Draft HTTP Redirects in HTTP Adaptive Streaming November 2018 6.1. Example 1: Absolute URI to Manifest Let us take a HLS example and assume the player is passed an absolute URI for the HLS manifest (e.g., a user has entered the manifest address directly into the player): https://video.cdn.example.com/vod/movie/master.m3u8 GET /vod/movie/master.m3u8 Host: video.cdn.example.com HTTP/1.1 302 Found Location: https://edge1.dcdn.example.net/vod/movie/master.m3u8 The client then requests the HLS manifest from the edge cache. GET /vod/movie/master.m3u8 Host: edge1.dcdn.example.net HTTP/1.1 200 OK The client reads the per-bitrate manifest reference from the master manifest. #EXT-X-STREAM-INF:BANDWIDTH=2121091,RESOLUTION=640x360,CODECS="mp4 a.40.2,avc1.4d001f" 1800K/1800_complete.m3u8 The player needs to resolve 1800K/1800_complete.m3u8. Following [RFC3986]: The m3u8 does not have a base URI embedded within it, so, Section 5.1.1 does not apply. The relative reference (and therefore the representation it references) is enclosed in an encapsulating entity, so, Section 5.1.2 applies. Section 5.1.2 says 'Thus, the default base URI of a representation is the base URI of the entity in which the representation is encapsulated.' So what is the base URI of the encapsulating entity (/vod/movie/ master.m3u8)? 1. If we follow [RFC3986] again: The master.m3u8 does not have a base URI embedded within it, so, Section 5.1.1 does not apply. Finkelman & Begen Expires May 8, 2019 [Page 6] Internet-Draft HTTP Redirects in HTTP Adaptive Streaming November 2018 2. The master.m3u8 is not encapsulated in any other entity, so, 5.1.2 does not apply. A URI was used to retrieve the m3u8, so, Section 5.1.3 applies. 3. Section 5.1.3 says 'if a URI was used to retrieve the representation, that URI shall be considered the base URI. Note that if the retrieval was the result of a redirected request, the last URI used (i.e., the URI that resulted in the actual retrieval of the representation) is the base URI.' 4. As the retrieval was the result of a redirected request, the second sentence applies and the last URI is https://edge1.dcdn.example.net/vod/movie/master.m3u8 and so that is the base URI for the manifest (the encapsulating entity). 5. Therefore the base URI for the relative URI 1800K/1800_complete.m3u8 is https://edge1.dcdn.example.net/vod/movie/master.m3u8 and the player should request https://edge1.dcdn.example.net/vod/ movie/1800K/1800_complete.m3u8 6.2. Example 2: Relative Reference in an Encapsulating Entity Let us take the same example, but instead of the player being passed an absolute URI, let us assume the URI is relative and embedded within another entity, i.e., a top-level HTML page https://video.cdn.example.com/index.html. In that case /vod/movie/ master.m3u8 (the encapsulating entity of 1800K/1800_complete.m3u8) is itself encapsulated (in /index.html) and Section 5.1.2 does apply this time, and the base URI of /vod/movie/master.m3u8 depends on the base URI of /index.html The base URI of /index.html (which is not encapsulated in another entity) follows Section 5.1.3, and depends on how /index.html was retrieved. 1. If /index.html was retrieved without redirection, the base URI is https://video.cdn.example.com/index.html and the base URI for the relative URI 1800K/1800_complete.m3u8 is https://video.cdn.example.com/vod/movie/master.m3u8 and the player should request https://video.cdn.example.com/vod/ movie/1800K/1800_complete.m3u8 2. If /index.html was retrieved via a redirect (e.g., to https://edge1.dcdn.example.net/index.html), the base URI is https://edge1.dcdn.example.net/index.html, and the base URI for the relative URI 1800K/1800_complete.m3u8 is https://edge1.dcdn.example.net/vod/movie/master.m3u8 and the Finkelman & Begen Expires May 8, 2019 [Page 7] Internet-Draft HTTP Redirects in HTTP Adaptive Streaming November 2018 player should request https://edge1.dcdn.example.net/vod/ movie/1800K/1800_complete.m3u8 6.3. Example 3: Relative Reference with Redirect In this example, as in Example 2 above, the manifest URI is relative and embedded within another entity within a top-level HTML page https://video.cdn.example.com/index.html. The base URI of /index.html is an absolute URI and the request for /index.html is not redirected, but, in this case, the request for /vod/movie/master.m3u8 is redirected. What would be resulting URI for 1800K/1800_complete.m3u8 in this case ? 1. According to Section 5.1.2 of [RFC3986] 1800K/1800_complete.m3u8 will get its base URI from the base URI of the encapsulating entity which is /vod/movie/master.m3u8. 2. As /vod/movie/master.m3u8 is itself encapsulated within /index.html then according to Section 5.1.2 its base URI is the taken from the base URI of /index.html. Since this base URI is https://video.cdn.example.com/index.html, the base URI of the master.m3u8 is resolved to https://video.cdn.example.com/vod/movie/master.m3u8, though it was redirected and its actual retrieval URI was https://edge1.dcdn.example.net/vod/movie/master.m3u8 3. The 1800K/1800_complete.m3u8, taking its base URI from the the encapsulating master.m3u8, is resolved to https://video.cdn.example.com/vod/movie/1800K/1800_complete.m3u8 One can see that in this case, as Section 5.1.2 does not define what happens if the encapsulating entity was redirected, the straightforward interpretation leads to the undesired result of the 1800_complete.m3u8 being requested from the original URI rather than the redirection URI. 7. Conclusion The examples above show that the behavior under playlist redirect depends on how the playlist URI was resolved, and whether the playlist URI was an absolute URI or a relative one. It also depends on how exactly Section 5.1.2 of [RFC3986] is interpreted in the absence of clear directions for the case that the encapsulating entity itself was a relative reference which was redirected. Finkelman & Begen Expires May 8, 2019 [Page 8] Internet-Draft HTTP Redirects in HTTP Adaptive Streaming November 2018 As explained above, we would like to have a single clear specification defining a consistent client behavior that is 'sticky' after a playlist redirect, regardless of how the playlist URI was resolved. We, therefore, hope that the members of HTTPbis working group could share their views regarding the preferred approach in order to resolve this issue. 8. IANA Considerations There are no IANA considerations. 9. Security Considerations TBC. 10. Acknowledgements The authors would like to thank the members of the Open Caching working group of the Streaming Video Alliance (SVA) for their feedback and especially Ben Niven-Jenkins and Kevin J. Ma for their help in understanding the current state of the standards for the provided examples. 11. Contributors TBC. 12. References 12.1. Normative References [RFC2119] Bradner, S., "Key words for use in RFCs to Indicate Requirement Levels", BCP 14, RFC 2119, DOI 10.17487/RFC2119, March 1997, . [RFC3986] Berners-Lee, T., Fielding, R., and L. Masinter, "Uniform Resource Identifier (URI): Generic Syntax", STD 66, RFC 3986, DOI 10.17487/RFC3986, January 2005, . [RFC8174] Leiba, B., "Ambiguity of Uppercase vs Lowercase in RFC 2119 Key Words", BCP 14, RFC 8174, DOI 10.17487/RFC8174, May 2017, . Finkelman & Begen Expires May 8, 2019 [Page 9] Internet-Draft HTTP Redirects in HTTP Adaptive Streaming November 2018 12.2. Informative References [MPEG-DASH] ISO/IEC 23009-1:2014, "Dynamic Adaptive Streaming over HTTP-- Part 1: Media Presentation Description and Segment Formats", 2014. [RFC6983] van Brandenburg, R., van Deventer, O., Le Faucheur, F., and K. Leung, "Models for HTTP-Adaptive-Streaming-Aware Content Distribution Network Interconnection (CDNI)", RFC 6983, DOI 10.17487/RFC6983, July 2013, . [RFC8216] Pantos, R., Ed. and W. May, "HTTP Live Streaming", RFC 8216, DOI 10.17487/RFC8216, August 2017, . Authors' Addresses Ori Finkelman Qwilt 6, Ha'harash Hod HaSharon 4524079 Israel Email: orif@qwilt.com Ali Begen Networked Media Konya Turkey Email: ali.begen@networked.media Finkelman & Begen Expires May 8, 2019 [Page 10]