Internet DRAFT - draft-vanderauwera-rmcat-video-quality
draft-vanderauwera-rmcat-video-quality
RMCAT Working Group G. Van der Auwera
Internet Draft M. Coban
Intended status: Informational Qualcomm Technologies Inc.
Expires: April 13, 2014 October 13, 2013
RMCAT Video Quality Evaluation and Double Bottleneck Test Scenario
draft-vanderauwera-rmcat-video-quality-00.txt
Abstract
The first part of this document proposes video quality test scenarios
and desired quality behaviors to evaluate RMCAT congestion control
solutions. The purpose is to identify undesired video quality
behaviors. The second part proposes a double bottleneck test scenario
to provide additional insight into the rate allocation behavior of
RMCAT solutions.
Status of this Memo
This Internet-Draft is submitted in full conformance with the
provisions of BCP 78 and BCP 79.
Internet-Drafts are working documents of the Internet Engineering
Task Force (IETF), its areas, and its working groups. Note that
other groups may also distribute working documents as Internet-
Drafts.
Internet-Drafts are draft documents valid for a maximum of six months
and may be updated, replaced, or obsoleted by other documents at any
time. It is inappropriate to use Internet-Drafts as reference
material or to cite them other than as "work in progress."
The list of current Internet-Drafts can be accessed at
http://www.ietf.org/ietf/1id-abstracts.txt
The list of Internet-Draft Shadow Directories can be accessed at
http://www.ietf.org/shadow.html
This Internet-Draft will expire on April 13, 2014.
Copyright Notice
Copyright (c) 2013 IETF Trust and the persons identified as the
document authors. All rights reserved.
Van der Auwera & Coban Expires April 13, 2014 [Page 1]
Internet-Draft RMCAT Video Quality Evaluation October 2013
This document is subject to BCP 78 and the IETF Trust's Legal
Provisions Relating to IETF Documents
(http://trustee.ietf.org/license-info) in effect on the date of
publication of this document. Please review these documents
carefully, as they describe your rights and restrictions with respect
to this document. Code Components extracted from this document must
include Simplified BSD License text as described in Section 4.e of
the Trust Legal Provisions and are provided without warranty as
described in the Simplified BSD License.
Table of Contents
1. Introduction...................................................2
2. Video Quality Test Scenarios...................................3
2.1. Test Scenario A...........................................3
2.1.1. Configuration........................................3
2.1.2. Desired Video Quality Behavior.......................4
2.2. Test Scenario B...........................................4
2.2.1. Configuration........................................4
2.2.2. Desired Video Quality Behavior.......................5
2.3. Test Scenario C...........................................5
2.3.1. Configuration........................................5
2.3.2. Desired Video Quality Behavior.......................6
2.4. Video Coding and Communication Framework Discussion.......6
2.4.1. Test Sequences.......................................6
2.4.2. Configuration........................................6
2.5. Video Quality Assessment..................................7
3. Double Bottleneck Test Scenario................................7
3.1. Configuration.............................................7
4. Security Considerations........................................8
5. IANA Considerations............................................8
6. Conclusions....................................................8
7. References.....................................................9
7.1. Informative References....................................9
8. Acknowledgments................................................9
1. Introduction
The impact of congestion control on media streams is important in
real-world deployments. Therefore, the purpose of this evaluation is
to verify or inspect the video quality of RMCAT congestion control
solutions under different test scenarios. RMCAT based rate
adaptations should result in expected or desired video quality when
implemented in a state-of-the-art video coding and communication
framework. In other words, the purpose is to verify that there are no
unexpected or undesired video quality problems caused by congestion
Van der Auwera & Coban Expires April 13, 2014 [Page 2]
Internet-Draft RMCAT Video Quality Evaluation October 2013
control behaviors, while the main purpose is not to directly compare
RMCAT solutions against each other. It is proposed to perform such
video quality evaluation of candidate RMCAT solutions in addition to
network traffic evaluations.
An additional test scenario is proposed to evaluate the rate
allocation behavior of RMCAT solutions when asymmetric flow
conditions exist. In this test the video quality is not evaluated.
2. Video Quality Test Scenarios
It is proposed to evaluate the visual quality of a subset of test
scenarios described in [1]. The specifics are open for further
discussion in the working group. Specific configuration parameters,
such as bottleneck link rates, should be chosen so that video quality
changes are visible.
The video quality of the following congestion control behaviors is
interesting to evaluate:
o Startup
o Varying bottleneck link bandwidth
o Staggered flow starts
o Background traffic (bursty)
The following are video quality test scenarios based on [1].
2.1. Test Scenario A
This scenario consists of a single bottleneck link and a single media
flow. The purpose is to evaluate video quality under congestion
control startup and varying bottleneck bandwidth behaviors.
2.1.1. Configuration
Topology:
o Single bottleneck link
o Single RMCAT sender and receiver
Bottleneck link rate varies between 1Mbps and 500kbps as follows:
o 0-20s: 1Mbps
Van der Auwera & Coban Expires April 13, 2014 [Page 3]
Internet-Draft RMCAT Video Quality Evaluation October 2013
o 20-80s: 500kbps
o 80-100s: 1Mbps
One-way propagation delay: 10ms
Bottleneck queue type: drop-tail
Bottleneck queue size: 32 packets
Random loss rate over link: 0%
Initial rate: 200kbps
2.1.2. Desired Video Quality Behavior
Startup:
Video quality improves and stabilizes within 4 seconds
Bottleneck bandwidth drops from 1Mbps to 500kbps:
Video quality decreases and stabilizes within 2 seconds
Bottleneck bandwidth increases from 500kbps to 1Mbps:
Video quality improves and stabilizes within 4 seconds
2.2. Test Scenario B
Scenario B consists of a single bottleneck link and two media flows
with different start times. The purpose is to evaluate video quality
under the congestion control behavior when a second competing flow
joins and leaves the bottleneck link.
2.2.1. Configuration
Topology:
o Single bottleneck link
o Two RMCAT senders and receivers
Second flow joins 20s after first flow and leaves after 40s.
Bottleneck link rate is 1Mbps
Van der Auwera & Coban Expires April 13, 2014 [Page 4]
Internet-Draft RMCAT Video Quality Evaluation October 2013
One-way propagation delay: 10ms
Bottleneck queue type: drop-tail
Bottleneck queue size: 32 packets
Random loss rate over link: 0%
2.2.2. Desired Video Quality Behavior
First media flow:
After second flow joins, video quality of first flow decreases and
stabilizes within 4 seconds. After second flow leaves, video quality
increases and stabilizes within 4 seconds.
Second media flow:
At startup, the video quality improves and stabilizes within 4
seconds.
First and second flow:
Video quality of both streams is similar.
2.3. Test Scenario C
This scenario consists of a single bottleneck link with background
traffic. The purpose is to evaluate video quality under congestion
control behavior in the presence of bursty TCP flows.
2.3.1. Configuration
Topology:
o Single bottleneck link
o Single RMCAT sender and receiver
o Single TCP sender to receiver
Bursty TCP flow starts at 0s and stops at 60s.
Media flow joins link after 20s.
Bottleneck link speed is 1Mbps
Van der Auwera & Coban Expires April 13, 2014 [Page 5]
Internet-Draft RMCAT Video Quality Evaluation October 2013
One-way propagation delay: 10ms
Bottleneck queue type: drop-tail
Bottleneck queue size: 32 packets
Random loss rate over link: 0%
Bursty TCP flow parameters: to be defined
2.3.2. Desired Video Quality Behavior
After media flow joins link (>20s):
Video quality improves and stabilizes within 4 seconds.
After TCP flow ends (>60s):
Video quality improves and stabilizes within 4 seconds.
2.4. Video Coding and Communication Framework Discussion
It is proposed that RMCAT solutions are implemented in a state-of-
the-art video coding and communication framework to provide proof-of-
concept evidence. The purpose is to demonstrate that implementations
of the congestion control solutions are feasible and that the video
quality behavior under the above test scenarios is as desired.
The following provides some details about test sequences,
configuration and quality assessment.
2.4.1. Test Sequences
Content types: video telephony and conferencing
Length: 100 seconds
VGA resolution or higher
Frame rate is 15fps or higher
File based feed for repeatability
2.4.2. Configuration
Video codec: up to proponent (AVC/H.264, VP8, HEVC/H.265, VP9, etc.)
Van der Auwera & Coban Expires April 13, 2014 [Page 6]
Internet-Draft RMCAT Video Quality Evaluation October 2013
Error resiliency and concealment mechanisms: allowed
Traffic shaping: allowed
Proponents would be required to implement the congestion control
method exactly as proposed and deliver the target bitrate directly to
the video encoder's rate control without further processing. The
reasoning is that if extra processing is required, then this should
be part of the congestion control method under evaluation.
The video encoder's rate control must achieve the target bit rate
with reasonable accuracy and speed, which could be defined if
necessary.
The source code is to be provided for cross checking by non-
proponents with the exception of proprietary modules that are not
directly relevant for the evaluation.
2.5. Video Quality Assessment
Since the purpose is to determine that the video quality behavior of
the RMCAT solutions is as desired under the described test scenarios,
it is proposed that the assessment is performed by a panel of experts
that are non-proponents, for example, five experts. The experts
report undesired video quality behaviors of the proposed RMCAT
solutions. Alternatively, formal subjective quality testing (MOS) can
be performed, if the MOS results are determined to be useful in the
decision process.
3. Double Bottleneck Test Scenario
This part of the document describes an additional test scenario for
network traffic evaluation (not video quality).
In this scenario, which is based on [2], one media flow encounters
two bottleneck links that are each shared with a second but different
flow. Figure 1 depicts the test setup. Each source Si sends a flow to
its corresponding receiver Ri. The second bottleneck is more
restricted than the first. The purpose is to evaluate the congestion
control's rate distribution among the flows under these asymmetric
conditions.
3.1. Configuration
Topology:
o Two sequential bottleneck links (Figure 1)
Van der Auwera & Coban Expires April 13, 2014 [Page 7]
Internet-Draft RMCAT Video Quality Evaluation October 2013
o Three RMCAT senders and receivers
First bottleneck link speed is 1.5Mbps
Second bottleneck link speed is 1Mbps
One-way propagation delay per link: 10ms
Bottleneck queue type: drop-tail
Bottleneck queue size: 32 packets
Random loss rate over link: 0%
+--+
+--+ |S3| +--+
|S1|=== \ +--+ / ===|R1|
+--+ \\ || // +--+
\\ || //
+--+ Bottleneck +--+ +--+ Bottleneck +--+
|N0|==============|N1|=====|N2|==============|N3|
+--+ Link 1 +--+ +--+ Link 2 +--+
// || \\
+--+ // || \\ +--+
|S2|=== / || \ ===|R3|
+--+ +--+ +--+
|R2|
+--+
Figure 1 Double Bottleneck Test Setup
4. Security Considerations
Security issues have not been discussed in this memo.
5. IANA Considerations
There are no IANA impacts in this memo.
6. Conclusions
This document proposes video quality test scenarios and desired
quality behaviors to evaluate RMCAT congestion control solutions. In
Van der Auwera & Coban Expires April 13, 2014 [Page 8]
Internet-Draft RMCAT Video Quality Evaluation October 2013
addition, a double bottleneck test scenario is proposed to provide
additional insight into the rate allocation behavior of RMCAT
solutions. These evaluation scenarios are provided to be further
discussed in the RMCAT working group.
7. References
7.1. Informative References
[1] "RMCAT Solution Evaluations",
https://sites.google.com/site/ietfrmcatsolutionevaluations/
[2] S. Holmer, "On Fairness, Delay and Signalling of Different
Approaches to Real-time Congestion Control"
8. Acknowledgments
The authors are grateful to Vadim Seregin from Qualcomm for valuable
discussions.
This document was prepared using 2-Word-v2.0.template.dot.
Van der Auwera & Coban Expires April 13, 2014 [Page 9]
Internet-Draft RMCAT Video Quality Evaluation October 2013
Authors' Addresses
Geert Van der Auwera
Qualcomm Technologies Inc.
5775 Morehouse Drive
San Diego, CA 92121
USA
Email: geertv@qti.qualcomm.com
Muhammed Coban
Qualcomm Technologies Inc.
5775 Morehouse Drive
San Diego, CA 92121
USA
Email: mcoban@qti.qualcomm.com
Van der Auwera & Coban Expires April 13, 2014 [Page 10]
