| Network Working Group | Z. Sarker |
| Internet-Draft | Ericsson AB |
| Intended status: Informational | V. Singh |
| Expires: August 18, 2014 | Aalto University |
| X. Zhu | |
| M. A. Ramalho | |
| Cisco Systems | |
| February 14, 2014 |
Test Cases for Evaluating RMCAT Proposals
draft-sarker-rmcat-eval-test-00
The Real-time Transport Protocol (RTP) is used to transmit media in multimedia telephony applications, these applications are typically required to implement congestion control. The RMCAT working group is currently working on candidate algorithms for such interactive real-time multimedia applications. This document describes the test cases to be used in the evaluation of the performance of those candidate algorithms.
This Internet-Draft is submitted in full conformance with the provisions of BCP 78 and BCP 79.
Internet-Drafts are working documents of the Internet Engineering Task Force (IETF). Note that other groups may also distribute working documents as Internet-Drafts. The list of current Internet-Drafts is at http://datatracker.ietf.org/drafts/current/.
Internet-Drafts are draft documents valid for a maximum of six months and may be updated, replaced, or obsoleted by other documents at any time. It is inappropriate to use Internet-Drafts as reference material or to cite them other than as "work in progress."
This Internet-Draft will expire on August 18, 2014.
Copyright (c) 2014 IETF Trust and the persons identified as the document authors. All rights reserved.
This document is subject to BCP 78 and the IETF Trust's Legal Provisions Relating to IETF Documents (http://trustee.ietf.org/license-info) in effect on the date of publication of this document. Please review these documents carefully, as they describe your rights and restrictions with respect to this document. Code Components extracted from this document must include Simplified BSD License text as described in Section 4.e of the Trust Legal Provisions and are provided without warranty as described in the Simplified BSD License.
This memo describes a set of test cases for evaluating candidate RMCAT congestion control algorithm proposals, it is based on the guidelines enumerated in [I-D.ietf-rmcat-eval-criteria] and requirements discussed in [I-D.ietf-rmcat-cc-requirements]. The test cases cover basic usage scenarios and are described using a common structure, which allows for additional test cases to be added to those described herein to accommodate other topologies and/or the modeling of different link characteristics. Each test case incorporates the metrics, evaluation guidelines and parameters described in [I-D.ietf-rmcat-eval-criteria]. It is the intention of this work to capture the consensus of the RMCAT working group participants regarding the test cases upon which the performance of the candidate RMCAT proposals should be evaluated.
The terminology defined in RTP [RFC3550], RTP Profile for Audio and Video Conferences with Minimal Control [RFC3551], RTCP Extended Report (XR) [RFC3611], Extended RTP Profile for RTCP-based Feedback (RTP/AVPF) [RFC4585], Support for Reduced-Size RTCP [RFC5506], and RTP Circuit Breaker algorithm [I-D.ietf-avtcore-rtp-circuit-breakers] apply.
All test cases in this document follow a basic structure allowing implementers to describe a new test scenarios without repeatedly explaining common attributes. The structure includes a general description section that describes the test case and motivations, additionally it defines a set of attributes that characterize the testbed, i.e., the network path between communicating peers and the diverse traffic sources.
+---+ +---+
|S1 |====== \ UPSTREAM --> / =======|R1 |
+---+ \\ // +---+
\\ //
+---+ +-----+ +-----+ +---+
|S2 |=======| A |------------------------------>| B |=======|R2 |
+---+ | |<------------------------------| | +---+
+-----+ +-----+
(...) // \\ (...)
// <-- DOWNSTREAM \\
+---+ // \\ +---+
|Sn |====== / \ ======|Rn |
+---+ +---+
Figure 1: Example of A Testbed Topology
Any attribute can have a set of values (enclosed within "[]"). Each member value of such a set MUST be treated as different value for the same attribute. It is desired to run separate tests for each such attribute value.
The test cases described in this document follow the above structure.
In this test case the bottleneck-link capacity between the two endpoints varies over time. This test is designed to measure the responsiveness of the candidate algorithm. This test tries to address the requirement 1(a) in [I-D.ietf-rmcat-cc-requirements], which requires the algorithm to adapt the flow(s) and provide lower end-to-end latency when there exists:
It should be noted that the exact variation in available capacity due to any of the above depends on the under-lying technologies. Hence, we describe a set of known factors, which may be extended to devise a more specific test case targeting certain behaviour in a certain network environment.
Expected behavior: the candidate algorithms is expected to detect the variation in available capacity and adapt the media stream(s) accordingly. The flows stabilize around their maximum bitrate as the as the maximum link capacity is large enough to accommodate the flows. When the available capacity drops, the flow(s) adapts by decreasing its sending bit rate, and when congestion disappears, the flow(s) are again expected to ramp up.
To evaluate the performance of the candidate algorithms it is expected to log enough information to visualize the following metrics at a fine enough time granularity:
+---+ +---+
|S1 |===== \ UPSTREAM --> / =======|R1 |
+---+ \\ // +---+
\\ //
+-----+ +-----+
| A |------------------------------>| B |
| |<------------------------------| |
+-----+ +-----+
// \\
// <-- DOWNSTREAM \\
+---+ // \\ +---+
|S2 |====== / \ ======|R2 |
+---+ +---+
Figure 2: Testbed Topology for Variable Available Capacity
Testbed Topology: Two (2) media sources S1 and S2 are connected to their corresponding R1 and R2. The media traffic is transported over the upstream path and corresponding feedback/control traffic is transported over the downstream path.
Testbed attributes:
In this case, the application will attempt to ramp up to its maximum bit rate, since the link capacity is limited to a value lower, the congestion control scheme is expected to stabilize the sending bit rate close to the available bottleneck capacity. This situation can occur when the endpoints are connected via thin long networks even though the advertised capacity of the access network may be higher. The test case addresses the requirement 1 and 10 of the [I-D.ietf-rmcat-cc-requirements].
Expected behavior: the candidate algorithm is expected to detect the path capacity constraint, converges to bottleneck link's capacity and adapt the flow to avoid unwanted oscillation when the sending bit rate is approaching the bottleneck link's capacity. The oscillations occur when the media flow(s) attempts to reach its maximum bit rate, overshoots the usage of the available bottleneck capacity, to rectify it reduces the bit rate and starts to ramp up again.
UPSTREAM -->
+---+ +-----+ +-----+ +---+
|S1 |=======| A |------------------------------>| B |=======|R1 |
+---+ | |<------------------------------| | +---+
+-----+ +-----+
<-- DOWNSTREAM
Figure 3: Testbed Topology for Limited Link Capacity
Testbed topology: One media source S1 is connected to corresponding R1. The media traffic is transported over the upstream path and corresponding feedback/control traffic is transported over the downstream path.
To evaluate the performance of the candidate algorithms it is expected to log enough information to visualize the following metrics at a fine enough time granularity:
Testbed attributes:
In this test case, more than one RMCAT media flow shares the bottleneck link and each of them uses the same congestion control algorithm. This is a typical scenario wherein a real-time interactive application sends more than one media flows to the same destination and these flows are multiplexed over the same port. In such a scenario it is likely that the flows will be routed via the same path and need to share the available bandwidth amongst themselves. For the sake of simplicity it is assumed that there are no other competing traffic sources in the bottleneck link and that there is sufficient capacity to accommodate all the flows individually. While this appears to be a variant of the test case defined in Section 4.1 , however it tests the capacity sharing distribution of the candidate algorithm. Whereas, the previous test case measures the stability and responsiveness of the candidate algorithm. This test case particularly addresses the requirements 2,3 and 10 in [I-D.ietf-rmcat-cc-requirements].
Expected behavior: It is expected that the competing flows will converge to an optimum bit rate to accommodate all the flows with minimum possible latency and loss. Specifically, the test introduces three media flows at different time instances, when the second flow appears there should still be room to accommodate another flow on the bottleneck link. Lastly, when the third flow appears the bottleneck link should be saturated.
To evaluate the performance of the candidate algorithms it is expected to log enough information to visualize the following metrics metrics at a fine enough time granularity:
+---+ +---+
|S1 |===== \ UPSTREAM --> / =======|R1 |
+---+ \\ // +---+
\\ //
+---+ +-----+ +-----+ +---+
|S2 |=======| A |------------------------------>| B |=======|R2 |
+---+ | |<------------------------------| | +---+
+-----+ +-----+
// \\
// <-- DOWNSTREAM \\
+---+ // \\ +---+
|S3 |====== / \ ======|R3 |
+---+ +---+
Figure 4: Testbed Topology for Multiple RMCAT Flows
Testbed topology: Three media sources S1, S2, S3 are connected to respective R1, R2, R3. The media traffic is transported over the upstream path and corresponding feedback/control traffic is transported over the downstream path.
Testbed attributes:
In this test case, one or more RMCAT media flow shares the bottleneck link with at least one long lived TCP flows. Long lived TCP flows download data throughout the session and are expected to have infinite amount of data to send and receive. This is a scenario wherein a multimedia application co-exists with a large file download. The test case measures the adaptivity of the candidate algorithm to competing traffic, it addresses the requirement 8 in [I-D.ietf-rmcat-cc-requirements].
Expected behavior: depending on the convergence observed in test case 4.1 and 4.2, the candidate algorithm may be able to avoid congestion collapse. In the worst case, the media stream will fall to the minimum media bit rate.
To evaluate the performance of the candidate algorithms it is expected to log enough information to visualize the following metrics at a fine enough time granularity:
Testbed topology: One (1) media source S1 is connected to corresponding R1, but both endpoints are additionally receiving and sending data, respectively. The media traffic (S1->S2) is transported over the upstream path and corresponding feedback/control traffic is transported over the downstream path. Likewise media traffic (S2->S1) is transported over the downstram path and corresponding feedback/control traffic is transported over the upstream path. The TCP uploads traffic over upstream path and downloads over downstream path. Hence, TCP traffic exits in both path directions.
+--+ +--+ |S1|===== \ UPSTREAM --> / =======|R1| +--+ \\ // +--+ \\ // +-----+ +-----+ +-----+ +-----+ |R_tcp|======| A |---------------------------->| B |=====|R_tcp| +-----+ | |<----------------------------| | +-----+ +-----+ +-----+ // \\ // <-- DOWNSTREAM \\ +--+ // \\ +--+ |R2|====== / \ ======|S2| +--+ +--+
Figure 5: Testbed Topology for TCP vs RMCAT Flows
Testbed attributes:
In this test case, one or more RMCAT media flow shares the bottleneck link with at multiple short-lived TCP flows. Short-lived TCP flows resemble the on/off pattern observed in the web traffic, wherein clients (browsers) connect to a server and download a resource (typically a webpage, few images, text files, etc.) using several TCP connections (up to 4). This scenario shows the performance of the multimedia application when several browser windows are active. The test case measures the adaptivity of the candidate algorithm to competing web traffic, it addresses the requirements 2 in [I-D.ietf-rmcat-cc-requirements].
Depending on the number of short TCP flows, the cross-traffic either appears as a short burst flow or resembles a long TCP flow. The intention of this test is to observe the impact of short-term burst on the behaviour of the candidate algorithm.
To evaluate the performance of the candidate algorithms it is expected to log enough information to visualize the following metrics at a fine enough time granularity:
Testbed topology: One (1) media source S1 is connected to corresponding R1, but both endpoints are additionally receiving and sending data, respectively. The media traffic (S1->S2) is transported over the upstream path and corresponding feedback/control traffic is transported over the downstream path. Likewise media traffic (S2->S1) is transported over the downstram path and corresponding feedback/control traffic is transported over the upstream path. The TCP uploads traffic over upstream path and downloads over downstream path. Hence, TCP traffic exits in both path directions. The topology described here is similar to the one described in Figure 5.
Testbed attributes:
RMCAT WG has been chartered to define algorithms for RTP hence it is assumed that RTCP, RTP header extension or such would be used as signalling means for the adaptation algorithm in the backchannel. Due to asymmetry nature of the link between communicating peers it is possible to observer lack such backchannel information due to impaired backchannel link (even when forward channel might be unimpaired). This test case is designed to observer candidate congestion control behaviour in such an event. This test case addresses requirement number 5 and in particular, requirement number 7.
It is expected that the candidate algorithms should cope with the lack of backchannel information and adapt to minimize the performance degradation of media flows in the forward channel.
It SHOULD be noted that for this test case log is needed for the reference case where the downstream channel has no impairments
To evaluate the performance of the candidate algorithms it is expected to log enough information to visualize the following metrics at a fine enough time granularity:
.
Testbed topology: One (1) media source S1 is connected to corresponding R1, but both endpoints are additionally receiving and sending data, respectively. The media traffic (S1->S2) is transported over the upstream path and corresponding feedback/control traffic is transported over the downstream path. Likewise media traffic (S2->S1) is transported over the downstram path and corresponding feedback/control traffic is transported over the upstream path.
+---+ +---+
|S1 |===== \ UPSTREAM --> / =======|R1 |
+---+ \\ // +---+
\\ //
+-----+ +-----+
| A |------------------------------>| B |
| |<------------------------------| |
+-----+ +-----+
// \\
// <-- DOWNSTREAM \\
+---+ // \\ +---+
|R2 |===== / \ ======|S2 |
+---+ +---+
Figure 6: Testbed Topology for Congested Feedback Link
Testbed attributes:
In this test case, more than one RMCAT media flow shares the bottleneck link, but the end-to-end path latency for each RMCAT flow is different. For the sake of simplicity it is assumed that there are no other competing traffic sources in the bottleneck link and that there is sufficient capacity to accommodate all the flows. While this appears to be a variant of the test case 4.2, it tests the capacity sharing distribution of the candidate algorithm under different RTTs. This test case particularly addresses the requirements 2 [I-D.ietf-rmcat-cc-requirements].
It is expected that the competing flows will converge to an optimum bit rate to accommodate all the flows with minimum possible latency and loss. Specifically, the test introduces five media flows at the same time instance.
To evaluate the performance of the candidate algorithms it is expected to log enough information to visualize the following metrics at a fine enough time granularity:
Testbed Topology: Five (5) media sources S1,S2,..,S5 are connected to their corresponding media sinks R1,R2,..,R5. The media traffic is transported over the upstream path and corresponding feedback/control traffic is transported over the downstream path. The topology is the same as test case 4.3 defined in Section 4.3. The end-to-end path delay for S1-R1 is 10ms, S2-R2 is 25ms, S3-R3 is 50ms, S4-R4 is 100ms, S5-R5 is 150ms.
Testbed attributes:
In this test case, more than one real-time interactive media flows share the link bandwidth and all flows reach to a steady state by utilizing the link capacity in an optimum way. At these stage one of the media flow is paused for a moment. This event will result in more available bandwidth for the rest of the flows and as they are on a shared link. When the paused media flow will resume it would no longer have the same bandwidth share on the link. It has to make it way through the other existing flows in the link to achieve a fair share of the link capacity. This test case is important specially for real-time interactive media which consists of more than one media flows and can pause/resume media flow at any point of time during the session. This test case directly addresses the requirement number 1.B in [I-D.ietf-rmcat-cc-requirements]. One can think it as a variation of test case 4.3. However, it is different as the candidate algorithms can use different strategies to increase it s efficiency, for example the fairness, convergence time, reduce oscillation etc, by capitalizing the fact that they have previous information of the link.
To evaluate the performance of the candidate algorithms it is expected to log enough information to visualize the following metrics at a fine enough time granularity:
Testbed Topology: Same as test case 4.3 defined in Section 4.3
Testbed attributes: The general description of the testbed parameters are same as test case 4.3 with changes in the test specific setup as below-
TBD
Additional cellular network specific test cases are define in [I-D.draft-sarker-rmcat-cellular-eval-test-cases]
TBD
[Editor's Note: We should encourage people to come up with possible WiFi Network specific test cases]
Security issues have not been discussed in this memo.
There are no IANA impacts in this memo.
Much of this document is derived from previous work on congestion control at the IETF.
The content and concepts within this document are a product of the discussion carried out in the Design Team.