Informational F.Wang Internet Draft Beijing Institute of Technology Intended status: Informational Z.Fei Expires: Nov 19, 2015 Beijing Institute of Technology May 19, 2015 QoE Evaluation for HTTP Adaptive Streaming draft-wang-tsvwg-qoe-evaluation-has-00.txt Abstract This document describes a method to evaluate the Quality of Experience (QoE) of real-time video delivered over HTTP Adaptive Streaming (HAS) technology. Not only the end points but also the content providers and network operators can acquire the QoE of HAS by implementing this method. Status of this Memo This Internet-Draft is submitted to IETF in full conformance with the provisions of BCP 78 and BCP 79. Internet-Drafts are working documents of the Internet Engineering Task Force (IETF), its areas, and its working groups. Note that other groups may also distribute working documents as Internet-Drafts. Internet-Drafts are draft documents valid for a maximum of six months and may be updated, replaced, or obsoleted by other documents at any time. It is inappropriate to use Internet-Drafts as reference material or to cite them other than as "work in progress." The list of current Internet-Drafts can be accessed at http://www.ietf.org/1id-abstracts.html The list of Internet-Draft Shadow Directories can be accessed at http://www.ietf.org/shadow.html This Internet-Draft will expire on Sept 18, 2015. Copyright and License Notice Copyright (c) 2014 IETF Trust and the persons identified as the document authors. All rights reserved. This document is subject to BCP 78 and the IETF Trust's Legal Provisions Relating to IETF Documents (http://trustee.ietf.org/license-info) in effect on the date of publication of this document. Please review these documents carefully, as they describe your rights and restrictions with respect to this document. Code Components extracted from this document must include Simplified BSD License text as described in Section 4.e of the Trust Legal Provisions and are provided without warranty as described in the Simplified BSD License. F.Wang Expires Nov 19,2015 [Page 1] Internet-Dratf QoE Evaluation for HTTP Adaptive Streaming May 18, 2015 Table of Contents 1. Introduction . . . . . . . . . . . . . . . . . . . . . . . . . 2 1.1. Method of evaluate QoE for HAS . . . . . . . . . . . . . . 2 2. Conventions used in this document . . . . . . . . . . . . . . 3 3. Chunk-quality metrics . . . . . . . . . . . . . . . . . . . . 3 3.1. The analysis of Quality metrics . . . . . . . . . . . . . 3 3.2. Metric chosen for QoE evaluation . . . . . . . . . . . . . 3 3.3. The chunk-quality tag . . . . . . . . . . . . . . . . . . 4 4. Pooling method . . . . . . . . . . . . . . . . . . . . . . . . 5 5. Security Considerations . . . . . . . . . . . . . . . . . . . 5 6. IANA Considerations . . . . . . . . . . . . . . . . . . . . . 5 7. References . . . . . . . . . . . . . . . . . . . . . . . . . . 6 7.1. Normative References . . . . . . . . . . . . . . . . . . . 6 7.2. Informative References . . . . . . . . . . . . . . . . . . 6 Authors' Addresses . . . . . . . . . . . . . . . . . . . . . . . . 6 1. Introduction HTTP Adaptive Streaming (HAS) has been used widely to deliver real- time video streams, including Apple's HTTP Live Streaming (HLS), Microsoft's HTTP Smooth Streaming (HSS), Adobe's HTTP Dynamic Streaming (HDS), and 3GPP's standardized solution 3GP-DASH [DASH]. Most HAS protocols have defined Manifest File, i.e. Media Presentation Description (MPD) of 3GP-DASH and Playlist file of HLS, which lists the location of various chunks, as well as some other informational tags set such as the way the content has been chunked. Each chunk is specified by its address and associated informational tags set. There will be no change in quality for a single chunk after it has been delivered due to the reliability of HTTP, thus the quality of each chunk can be measured at the source point before multimedia delivering. Based on the above analysis, the quality of each chunk can be as a tag and embedded into the associated informational tags set of Manifest Files. Then the overall QoE for the whole video can be obtained by a pooling model which takes the quality of each received chunk into consideration. 1.1. Method of evaluate QoE for HAS Since, for HAS all Manifest files must be download before the multimedia data playing, by capturing and parsing these files, quality of each chunk can be get easily. For each playing chunk, its corresponding information can be acquired from the HTTP request information, because a URL that included in the HTTP request information, corresponds to a specific presentation. By the above analysis, the quality of a playing chunk can be obtained in real time at any point, and the quality for a certain time can be predicted through pooling methods which take all the segments within the prediction period into consideration. The simplest pooling method is averaging over all played chunks. In this draft, we have proposed F.Wang Expires Nov 19,2015 [Page 2] Internet-Dratf QoE Evaluation for HTTP Adaptive Streaming May 18, 2015 a liner model which has been demonstrated has a higher accuracy in [Pooling method]. 2. Conventions used in this document The key words "MUST", "MUST NOT", "REQUIRED", "SHALL", "SHALL NOT", "SHOULD", "SHOULD NOT", "RECOMMENDED", "MAY", and "OPTIONAL" in this document are to be interpreted as described in RFC 2119 [RFC2119]. In this document, these words will appear with that interpretation only when in ALL CAPS. Lower case uses of these words are not to be interpreted as carrying RFC-2119 significance. A list of acronyms and abbreviations used in this document are presented below. o HAS: HTTP Adaptive Streaming o HLS: HTTP Live Streaming o DASH: Dynamic Adaptive Streaming over HTTP o QoE: Quality of Experience o MOS: Mean of Score o PSNR: Peak Signal to Noise Ratio o dPSNR: differential PSNR 3. Chunk-quality metrics 3.1. The analysis of Quality metrics The quality of each chunk can be measured by subjective or objective methods. It is recognized that subjective methods are time and manpower consuming, which makes these methods hard to utilize in practice. The limitation of subjective methods drives the development of objective methods. Usually the objective methods can be divided into three categories according to the dependency of original video: 1) full-reference methods (FR, in which need complete original content), 2) reduced-reference methods (RR, in which need partial information about original content) 3) no-reference methods (NR, in which need nothing about the original content). FR methods are more simple and accurate than RR and NR methods but not practical for most delivery scenarios due to their dependence of the complete original content. But for HAS delivery scenario, FR methods can be used, since the quality of each segment can be measured at the source point that with original content available. 3.2. Metric chosen for QoE evaluation F.Wang Expires Nov 19,2015 [Page 3] Internet-Dratf QoE Evaluation for HTTP Adaptive Streaming May 18, 2015 Peak Signal to Noise Ratio (PSNR), Structural Similarity (SSIM) and Video Quality Metric (VQM) are three commonly used FR objective metrics for quality analysis. But these methods are sensitive to video content and often used as a measurement metric for different compression versions with a same given original content. The sensitivity of these methods are caused by the different spatial- temporal characteristics in different video content, which will lead to different PSNR range for different content although they have the same QoE range. In order to solve the above problem, this draft present the concept of differential PSNR (dPSNR) based on PSNR. For each segment, a suitable presentation is chosen as the benchmark, then the dPSNR of all presentations of this segment are obtained by subtracting the PSNR value of the benchmark. In order to solve the above problem, this draft present the concept of differential PSNR (dPSNR) based on PSNR. For each segment, a suitable presentation is chosen as the benchmark, then the dPSNR of all presentations of this segment are obtained by subtracting the PSNR value of the benchmark. E.g. if different contents at a content provider are encoded with the same bitrate levels set, for each segment, we can choose the presentation with the highest bitrate level as the benchmark. 3.3. The chunk-quality tag Manifest files contain the URLs and other informational tags for various chunks. In this draft, the following attributes are defined and added to the basic Manifest files: QUALITY TYPE The value is the type of quality metric has been used. VALUE It denotes the value of quality under the given quality metric which can be get from the QUALITY TYPE attribute. DESCRIPTION It denotes some other information for presentation-quality, e.g. if we have chosen dPSNR as the presentation-quality metric, it can describe whether a presentation is chosen as a benchmark, if QUALITY DESCRIPTION=1 the presentation is a benchmark, otherwise not. In this draft, dPSNR has been used as the presentation-quality metric. Thus the QUALITY TYPE: dPSNR. The dPSNR value of each multimedia presentation can be calculated at the content provider point. F.Wang Expires Nov 19,2015 [Page 4] Internet-Dratf QoE Evaluation for HTTP Adaptive Streaming May 18, 2015 A simple example of media play list file for HLS after we define and embed the chunk-quality tag #EXT-X-QoE: #EXTM3U #EXT-X-VERSION: 3 #EXT-X-TARGETDURATION: 8 #EXT-X-MEDIA-SEQUENCE: 2680 #EXT-X-QoE: QUALITY TYPE=dPSNR, VALUE=20, DESCRIPTION=0 #EXTINF: 7.975, https://priv.example.com/fileSequence2680.ts #EXT-X-QoE: QUALITY TYPE=dPSNR, VALUE=18, DESCRIPTION=0 #EXTINF: 7.941, https://priv.example.com/fileSequence2681.ts #EXT-X-QoE: QUALITY TYPE=dPSNR, VALUE=15, DESCRIPTION=0 #EXTINF: 7.975, https://priv.example.com/fileSequence2682.ts 4. Pooling method Since many tests have proved that for a certain time, the quality perceived by the end users not only depends on mean video quality, but also depends much on the segments with the best or worst quality that may leave a deep impression to end users, and the occurrence of quality switching that will distract viewers' attention. The liner model is proposed based on the above analysis, which take more influenced factors into consideration. A simple example to evaluate quality for a certain period is shown as follows, PMOS=a*mean+b*max+c*min+d*std, where a,b,c,d are parameters associated with content type, encoded type and prediction period, mean, max, min, std are calculated influenced factors, which measure the mean quality, maximum quality, minimum quality and standard deviation quality respectively. 5. Security Considerations Since the protocol relies on HTTP Live Streaming, most of the same security considerations apply. See section 11 of draft-pantos- httplive-streaming-13. 6. IANA Considerations Same IANA considerations of HTTP Live Streaming apply. See section 10 of draft-pantos-http-live-streaming-13. F.Wang Expires Nov 19,2015 [Page 5] Internet-Dratf QoE Evaluation for HTTP Adaptive Streaming May 18, 2015 7. References 7.1. Normative References [RFC2119] Bradner, S., "Key words for use in RFCs to Indicate Requirement Levels", BCP 14, RFC 2119, March 1997. [RFC2616] Fielding, R., Gettys, J., Mogul, J., Frystyk, H.,Masinter, L., Leach, P. and T. Berners-Lee, "Hypertext Transfer Protocol -- HTTP/1.1", RFC 2616, June 1999. 7.2. Informative References [DASH] 3GPP TS 26.247 v12.2.0, "Progressive Download and Dynamic Adaptive Streaming over HTTP (3GP-DASH)", Release 12, mar 2014 [Pooling method] Xiaolin Deng, Liang Chen, Fei Wang "A Novel Strategy to Evaluate QoE for Video Service Delivered over HTTP Adaptive Streaming", unpublished Authors' Addresses Fei Wang Beijing Institute of Technology 5 South Zhongguancun Street, Haidian District, Beijing, China Email: fei_wang@bit.edu.cn Zesong Fei Beijing Institute of Technology 5 South Zhongguancun Street, Haidian District, Beijing, China Email: feizesong@bit.edu.cn F.Wang Expires Nov 19,2015 [Page 6] Internet-Dratf QoE Evaluation for HTTP Adaptive Streaming May 18, 2015