Switch Testing for Streaming Media Applications July 2006 Network Working Group J. Welch Internet Draft IneoQuest Technologies Intended Category: Informational Muhammad Waris Sagheer Cisco Systems Sameer Satyam Cisco Systems Javed Asghar Cisco Systems Syed Nawaz Cisco Systems Andre Dufour Agilent Technologies July, 2006 Switch Testing for Streaming Media Applications draft-welch-streaming-test-00.txt Status of this Memo By submitting this Internet-Draft, each author represents that any applicable patent or other IPR claims of which he or she is aware have been or will be disclosed, and any of which he or she becomes aware will be disclosed, in accordance with Section 6 of BCP 79. Internet-Drafts are working documents of the Internet Engineering Task Force (IETF), its areas, and its working groups. Note that other groups may also distribute working documents as Internet- Drafts. Internet-Drafts are draft documents valid for a maximum of six months and may be updated, replaced, or obsoleted by other documents at any time. It is inappropriate to use Internet-Drafts as reference material or to cite them other than a "work in progress." The list of current Internet-Drafts can be accessed at http://www.ietf.org/lid-abstracts.html Welch,Sagheer,Salyam, Expires January 2007 [Page 1] Asghar,Nawaz, Dufour Switch Testing for Streaming Media Applications July 2006 The list of Internet-Draft Shadow Directories can be accessed at http://www.ietf.org/shadow.html This memo provides information for the Internet community. It does not specify an Internet standard. Distribution of this memo is unlimited. The IETF takes no position regarding the validity or scope of any Intellectual Property Rights or other rights that might be claimed to pertain to the implementation or use of the technology described in this document or the extent to which any license under such rights might or might not be available; nor does it represent that it has made any independent effort to identify any such rights. Information on the procedures with respect to rights in RFC documents can be found in BCP 78 and BCP 79. Abstract This memo defines a set of supplementary tests for networking interconnection devices (switches) that can be used to evaluate and compare their performance when used with streaming media. The Media Delivery Index (MDI) [i3] measurement is employed as a convenient stream quality indicator of input and output streams to indicate cumulative stream jitter and packet loss under test load conditions. Typical operating profiles are defined to focus test efforts on common switch applications for unidirectional streaming media to reduce the amount of testing required and to encourage device and system evaluation in advance of deployment through use of a set of reference tests. The supplementary tests defined in this memo are intended for Information only. 1. Introduction There has been considerable progress over the last several years in the development of methods to provide for Quality of Service (QoS) over packet switched networks to improve the delivery of streaming media and other time and packet loss sensitive applications such as [i1], [i2]. QoS mechanisms are required for many practical converged networks involving streaming media applications such as video transport to assure the availability of network bandwidth by Welch,Sagheer,Salyam, Expires January 2007 [Page 2] Asghar,Nawaz, Dufour Switch Testing for Streaming Media Applications July 2006 providing upper limits on the number of flows admitted to a network as well as to bound the packet jitter introduced by the network. These bounds are required to dimension a receiver`s buffer to properly display the video in real time without buffer overflow or underflow. Just as a receiver’s buffer has limited depth and is subject to overflow considerations, so too are intermediate switch node buffers limited in depth and subject to flow variation stresses. QoS implementations and queue management strategies in switch devices vary widely and it has been difficult to compare resultant switch performance in the presence of streaming media. Owing to their real time behavior and persistent nature, streaming media applications are particularly sensitive to packet arrival time jitter and packet loss. Cumulative jitter and loss performance are especially critical in gauging end user perception of compressed video stream quality as any loss is frequently visually perceptible and significant cumulative jitter can cause loss in terminal equipment due to dejitter buffer overflow or underflow. Subjective voice stream perception quality has been shown to be more tolerant of loss for acceptable quality. Due to the wide variety of possible network application uses, traditional data application oriented tests often do not reflect perceived switch performance when used in streaming media applications. Traditional network interconnect device evaluation criteria such as outlined in RFC 2544 specify procedures for evaluating device performance expressed as maximum frame forwarding rates for given frame sizes, frame formats, broadcast/multicast frames, management frames, and during routing updates and with forwarding filters enabled. Bursty traffic is considered as well by specifying test conditions of various length bursts. Device throughput is characterized by the fastest rate that can be sustained without frame loss. Latency measurements are considered by measurement of a frame propagation time while the device is under load. Frame burst handling, system recovery times from oversubscription, and reset recovery times are also detailed. These measurements provide standardized test methods and consistent reporting guides so that device performance can be compared with different vendors and the devices’ suitability for handling specific mixed data applications can be assessed. RFC 2889 extends the methodology of RFC 2544 for MAC layer switches to include “…forwarding performance, congestion control, latency, address handling and filtering.”[RFC 2889] It also includes test Welch,Sagheer,Salyam, Expires January 2007 [Page 3] Asghar,Nawaz, Dufour Switch Testing for Streaming Media Applications July 2006 conditions for fully meshed traffic, consideration for address learning, frame formats and sizes, duplexity, burst size, number of addresses per port, flooding performance, illegal frame handling, and broadcast frame performance. As in RFC 2544, standardized test methods and consistent reporting guides are detailed. Methods exist to control the stream jitter introduced by devices forwarding streaming media [i4] through ingress flow allocation and policing, forwarding management, and link bandwidth allocation. The design and configuration tasks necessary to achieve optimal performance for these functions are often complex. Hence, there is a need for a verification and evaluation procedure for streaming media. This note addresses the need. Traditional methods of evaluating VoIP streams have proven adequate to date. Video streaming media device performance has not been adequately addressed and is the focus of this note. For converged profile tests, it is suggested that existing, conventional VoIP tests be used for verifying the performance of VoIP such as ITU-T P.862 [i7], Y.1541 [i8], and G.107 [i9] along with equipment intended to provide such results in addition to the tests described here. If the defined test profile includes VoIP traffic, such tests should ideally be performed simultaneously with the tests described here. This note proposes and describes the following approach for device evaluation: -Identification of a set of profiles representing common traffic load types, levels, and flow paths to be used during the evaluation of a switch intended for streaming media. These profiles are based on common configurations found in many Cable and Telco streaming media transport networks. Some suggested profiles are included and others may be added to expand the coverage of the tests to include more traffic classes. -Generation of realistic, repeatable traffic load test conditions to characterize switches with the use of the media delivery index (MDI) as the indicator of flow quality for streaming media. MDI, which measures cumulative packet jitter and packet loss, is becoming increasingly common as a metric of streaming media quality. -Use of the MDI Characteristic Curve as the means for showing the results of switch testing for streaming media. Welch,Sagheer,Salyam, Expires January 2007 [Page 4] Asghar,Nawaz, Dufour Switch Testing for Streaming Media Applications July 2006 Determining switch contribution to jitter and loss under known traffic profiles provides users confidence that a particular switch will adequately function in a particular target environment. For example, a video on demand (VoD) server head end environment with known server-delivered traffic, MDI characteristics, flow paths, and known downstream decoder. Specifications can require that a switch and the remaining transport system contribute no more than a maximum MDI footprint (additional jitter and loss) for successful unimpaired stream delivery where success is ultimately measured by a network- unimpaired decode of the stream. Switches characterized in these terms simplify system design and assure streaming media delivery quality for a given range of load conditions. For other than streaming media performance, the existing methods described above can and should be used to represent switch performance. Thus, this note is intended to describe a realistic and practical set of supplementary tests for switches intended to carry streaming media whose results can readily be used to: 1. Compare switches of various design, configuration, setup, and manufacturer, 2. Determine whether so characterized devices will deliver the expected performance for specific system requirements, and 3. Verify that a specific system is configured to deliver expected operational streaming media delivery performance. 4. In addition, these tests can assist in planning and configuring buffer sizes properly over single and multiple hop networks in the presence of bursty streaming traffic. The Media Delivery Index (MDI) measurement is employed as the media quality indicator since it conveniently captures cumulative stream jitter and packet loss performance through its delay factor (DF) and media loss rate (MLR) components. 2. Testing Criteria Welch,Sagheer,Salyam, Expires January 2007 [Page 5] Asghar,Nawaz, Dufour Switch Testing for Streaming Media Applications July 2006 The measure of network switch performance is the magnitude of MDI modification, or MDI footprint, it imposes: this indicates how it has degraded or improved a stream’s MDI. The MDI footprint of a device or network segment is the maximum difference between the MDI delay factor at the input of the device to that measured at the output for the same stream. Taking the maximum delay factor recorded for each stream at the input and comparing it to the corresponding stream’s maximum DF recorded over the test period at the output and, from these maxima, selecting the largest one demonstrates the device’s worst case footprint. Ideally, the DF change will be low indicating that the amount of stream jitter present at the input has not been substantially increased and no loss was introduced. It is important to make the measurements on all streams during the test as some streams can be significantly affected while other streams may not be affected at all based on their relative arrival times at a queue. It is also important that the switch’s load profile including the input active port count (including exactly which ports on a multiport device are active), per port stream count, bit rates, forwarding paths, and delay factors accurately represent the worst case combinations for the intended application, since these factors may change the effects that a switch has on streams. A switch that unduly delays forwarding packets from input queue to output queue or bursts data to its output queue, and subsequently to the port output, exhibits poorer stream performance than one that avoids introducing packet bursts. This poorer bursty performance can accumulate in successive downstream devices and may ultimately result in a downstream queue overflow (or an underflow at the destination decoder) and lost packets. Devices with ports that have per stream rate shaped outputs forward better stream characteristics to downstream nodes reducing the chances of a downstream queue overflow. The MDI Characteristic Curves that result from the tests described in this note can be utilized by a service provider in an iterative fashion to adequately dimension queue depths and behavior, traffic shaping, and other switching infrastructure resources during both lab certification and system commissioning. 3. Profiles Switches should be tested using input traffic profiles representative of the intended applications of the devices. Test results should be plotted as MDI Characteristic Curves as described below. MDI Welch,Sagheer,Salyam, Expires January 2007 [Page 6] Asghar,Nawaz, Dufour Switch Testing for Streaming Media Applications July 2006 Characteristic Curves can be used to determine operating limits for the switch for a given number of input streams which have given levels of MDI. Switch implementation options such as how ASICs and backplanes are partitioned, how architecture tradeoffs are made, how available buffer memory is allocated, etc. can greatly affect the performance being discussed. Unless otherwise specified, manufacturers should quote performance based on worst case configurations with notes on how streaming performance would be affected for other configurations. For example, if selecting output ports on a switch results in worse MDI Characteristic Curve performance if the active output ports are on different blades (modules), then the manufacturer should note the differences and tradeoffs. Alternatively, the device specification can be made such that any configuration of port selection will satisfy the (probably lower performance) specification. Testing time can be shortened and the process of determining MDI Characteristic Curves expedited if only the worst case numbers are specified. These advantages must be weighed against under representing the device performance in more ideal configurations. In any case, the operational hardware and software configurations must be noted along with the MDI Characteristic curve test results. Testing and plotting MDI Characteristic Curves for the following profiles is recommended based on the profiles’ prevalence in existing deployments. These profiles can provide a minimum baseline for device comparison. A relatively small number of test conditions are included to minimize testing time and costs in order to encourage testing and publication. Additional profiles may be used by vendors and/or users to represent other applications as demand warrants. Vendors may wish to expand this list in order to fully characterize a device that is capable of a particularly demanding system configuration. Users may wish to test a configuration that more exactly represents an expected, intended, or anticipated network load. Additional profiles may be included in testing results that are adjusted for port count, stream count, and stream mix by manufacturers, for example, to better represent a device’s capabilities and/or by users to better represent a target user configuration for testing and characterization. The listed profiles in this section should be included in a device characterization so that reference performance levels can be compared from manufacturer to manufacturer. If the number of available switch ports or port speeds or other switch implementation constraints prevent testing a Welch,Sagheer,Salyam, Expires January 2007 [Page 7] Asghar,Nawaz, Dufour Switch Testing for Streaming Media Applications July 2006 particular profile, it should be noted in the testing results. Testing configurations beyond the included profiles is encouraged. Some other typical profiles likely to be of interest include combinations of: SD/MPTS, HD/MPTS, SD/HD/MPTS, various ratios of multicast to unicast as would be encountered in a service provider’s head end consisting of both broadcast and on-demand flows, and flow paths for device ports within a blade and between blades. Profiles may also include redundant architectures based on mesh networks, redundant rings, with and without QoS configurations, VLANs, etc. A supplemental set of profiles would also likely include the reduced bit rates for SD and HD streams when used with MPEG4 part 10 compression. Note that the profiles are intended to describe an input traffic mix representative of a service provider’s or user’s operating environment. A switch’s performance might best be represented by a family of MDI Characteristic Curves for typical or recommended switch configurations. For example, a full featured metro class switch might have sets of MDI Characteristic Curves for VLAN configurations, converged network Diffserv priorities, etc. A lower cost, smaller edge switch may only need MDI Characteristic Curves for the Broadcast head end Profile. The profiles in this section assume 1 Gb/s links which are in common use today. Stream counts used may be scaled with bandwidth if other link speeds are of interest. The stream bit rates specified in this section are stream payload rates and may vary somewhat depending on the specific encapsulation protocols selected. 3.1 All Standard Definition (SD) Streams: This profile includes only SD streams at 3.75 Mb/s destined for uncongested output port(s). 3.75 Mb/s is the typical rate of transmission at which MPEG2 compressed standard definition streams are transported today in cable and telco environments. This profile assumes that all input streams can be destined for any single output port and that the output port will never be oversubscribed. Characteristic Curves for input streams in the following cases are required. Each row below varies the number of input streams per input port. For each of these combinations, the Characteristics are plotted when the streams are transmitted out 1, 2, 4, 8, or 16 ports. This profile is representative of head end applications where all Welch,Sagheer,Salyam, Expires January 2007 [Page 8] Asghar,Nawaz, Dufour Switch Testing for Streaming Media Applications July 2006 streams are encoded as SD and the switch is aggregating potentially hundreds of streams from off-air receivers and encoders or other feeds. The aggregated streams are then forwarded to possibly a metro or regional distribution system using a variety of possible ring, mesh, and point-to-point architectures. In this application the streams use multicast destination addresses. The switch may or may not be specifically configured for multicast operation. The number of streams per output port should be equal and should sum to the total number of input streams. No. of switch No. of Streams per No. of switch input ports Input port output ports 1 250 1, 2, 4, 8, 16 2 125 1, 2, 4, 8, 16 4 62 1, 2, 4, 8, 16 8 31 1, 2, 4, 8, 16 16 15 1, 2, 4, 8, 16 32 7 1, 2, 4, 8, 16 3.2 All High Definition (HD) Streams This profile includes only HD streams at 15.0 Mb/s destined for uncongested output port(s). 15 Mb/s is the typical rate of transmission at which MPEG2 compressed high definition streams are transported today in cable and telco environments. This profile assumes that all input streams can be destined for any single output port and that the output port will never be oversubscribed. MDI Characteristic Curves generated for the following cases are required. Each row below varies the number of input streams per input port. For each of these combinations, the MDI Characteristic Curves are plotted when the streams are transmitted out 1, 2, 4, 8, or 16 ports. This profile is representative of head end applications where all streams are encoded as HD and the switch is aggregating potentially hundreds of streams from off-air receivers and encoders or other feeds. The aggregated streams are then forwarded to possibly a metro or regional distribution system using a variety of possible ring, mesh, and point-to-point architectures. In this application the Welch,Sagheer,Salyam, Expires January 2007 [Page 9] Asghar,Nawaz, Dufour Switch Testing for Streaming Media Applications July 2006 streams use multicast destination addresses. The switch may or may not be specifically configured for multicast operation. The number of streams per output port should be equal and should sum to the total number of input streams. No. of switch No. of Streams per No. of switch input Ports input port output ports 1 60 1, 2, 4, 8, 16 2 30 1, 2, 4, 8, 16 4 15 1, 2, 4, 8, 16 8 7 1, 2, 4, 8, 16 Welch,Sagheer,Salyam, Expires January 2007 [Page 10] Asghar,Nawaz, Dufour Switch Testing for Streaming Media Applications July 2006 3.3 All Multi Program Transport Streams (MPTS) This profile includes only MPTS streams at 38 Mb/s destined for uncongested output port(s). 38 Mb/s is the typical rate of transmission at which multiple programs multiplexed together at a head end into a Multi Program Transport Stream (MPTS) are transported today in cable and telco environments. This profile assumes that all input streams can be destined for any single output port and that the output port will never be oversubscribed. MDI Characteristic Curves for the following cases are required. Each row below varies the number of input streams per input port. For each of these combinations, the MDI Characteristic Curves are plotted when the streams are transmitted out 1, 2, 4, 8, or 16 ports. This profile is representative of head end applications where all streams are multiplexed into MPTSs and the switch is aggregating these streams from the multiplexers. The aggregated streams are then forwarded to possibly a metro or regional distribution system using a variety of possible ring, mesh, and point-to-point architectures. In this application the streams use unicast destination addresses to effect transport to a downstream demultiplexer before transport to an end user. The number of streams per output port should be equal and should sum to the total number of input streams. No. of switch No. of Streams per No. of switch input ports input port output ports 1 25 1, 2, 4, 8 2 12 1, 2, 4, 8 4 6 1, 2, 4, 8 8 2 1, 2, 4, 8 3.4 SD/HD Stream Mix This profile includes an SD/HD stream mix of 120 streams of SD 3.75 Mb/s and 30 streams of HD at 15.0 Mb/s destined for uncongested output port(s). A mix of SD and HD streams are typical in many current head ends where HD streams are still in limited availability or must be limited due to downstream distribution plant limitations. This profile assumes that all input streams can be destined for any Welch,Sagheer,Salyam, Expires January 2007 [Page 11] Asghar,Nawaz, Dufour Switch Testing for Streaming Media Applications July 2006 single output port and that the output port will never be oversubscribed. MDI Characteristic Curves for the following cases are required. Each row below varies the number of input streams per input port. For each of these combinations, the MDI Characteristic Curves are plotted when the streams are transmitted out 1, 2, 4, or 8 ports. This profile is representative of head end applications where there are a mix of streams encoded as SD and HD and the switch is aggregating potentially hundreds of streams from off-air receivers and encoders or other feeds. The aggregated streams are then forwarded to possibly a metro or regional distribution system using a variety of possible ring, mesh, and point-to-point architectures. In this application the streams use multicast destination addresses. The switch may or may not be specifically configured for multicast operation. The number of streams per output port should be equal and should sum to the total number of input streams. No. of switch No. of Streams per No. of switch Input ports input port (SD/HD) output ports 1 120/30 1, 2, 4, 8 2 64/16 1, 2, 4, 8 4 32/8 1, 2, 4, 8 8 16/4 1, 2, 4, 8 3.5 VoD Headend Switch This profile includes an SD/HD stream mix of 120 streams of SD 3.75 Mb/s and 30 streams of HD at 15.0 Mb/s destined for uncongested output port(s). In this application the streams use unicast destination addresses. This profile requires a minimum of 8 simultaneously active input ports representing a small to moderate size VoD server array serving a mix of SD and HD streams. This profile assumes that all input streams from an input port can be destined for any output port and that the output ports will never be oversubscribed. MDI Characteristic Curves for the following cases are required. Each row below varies the number of active input ports. For each row of combinations, the MDI Characteristic Curves are plotted when the streams are transmitted out 8, 16, 24, … up to the maximum number of blade and/or switch output ports. The maximum number of blade and/or switch output ports is determined either by the device manufacturer based on device physical or performance Welch,Sagheer,Salyam, Expires January 2007 [Page 12] Asghar,Nawaz, Dufour Switch Testing for Streaming Media Applications July 2006 limits, or by the user based on a maximum projected configuration. The number and type of streams per output port should be equal and should sum to the number of input streams per input port. Direct mapping of input port traffic to one other output port is used. No. of switch No. of Streams per No. of switch Input ports input port (SD/HD) output ports 8 120/30 8 16 120/30 16 24 120/30 24 3.6 Backbone Ring hub site Switch This profile includes two 10 Gigabit ports representing connections to a backbone with a traffic mix of SD/HD streams, VoIP streams, and various levels of non-streaming data with a mix of broadcast and unicast stream addressing. This profile is intended to represent converged backbone traffic, say on a metropolitan ring, with a hub site accessing and forwarding the backbone traffic for local distribution thus representing a midstream location in a provider network. This profile assumes that 50% of the backbone traffic is video, 20% is VoIP, 20% is Data and that most traffic arriving on one backbone port is forwarded to another backbone port while a percentage of the traffic is forwarded to local Gigabit ports and not forwarded to the second backbone port. Priority marking and handling of classes of traffic such as video, voice, and data may be optionally included in this profile. This profile includes an SD/HD stream mix of 1000 streams of SD 3.75 Mb/s, 300 streams of HD, 10000 VoIP streams at 64 Kb/s, and 2 Gb/s data traffic on the inbound backbone port with all streams employing unicast addressing. The data traffic should consist of 25% utilization each of 64, 512, and 1024, and 1500 byte packets. This profile requires a minimum of 2 active Gigabit output ports representing the traffic being forwarded to the hub site. The hub site traffic consists of 100 SD streams, 20 HD streams, 1000 VoIP streams, and 2% IP data utilization. The remainder of the traffic is forwarded to the downstream backbone port. Thus, direct mapping of Welch,Sagheer,Salyam, Expires January 2007 [Page 13] Asghar,Nawaz, Dufour Switch Testing for Streaming Media Applications July 2006 input port traffic to three other output ports, the backbone output port and two local hub output ports, is defined. This profile assumes that the streams from an input port destined for local Gigabit port(s) will never oversubscribe those ports. Likewise, the profile assumes that the streams from an input port destined for a backbone output port will never oversubscribe that port. MDI Characteristic Curves for the output port cases described below are required. 10 Gb/s backbone ports load: No. of Data No. of backbone No. of Streams per Util backbone output Input ports Input port (SD/HD/VoIP) % ports 1 800/200/10000 20 1 1 Gb/s Hub ports load: No. of Streams per Data No. of output port (SD/HD/VoIP) Util Gigabit output % ports 50/10/500 1% 2 This represents a backbone load of approximately 3Gb/s (SD video) + 3Gb/s (HD video) + 1Gb/s (VoIP) + 2Gb/s (data) = 9 Gb/s and a hub Gigabit drop port with 10% of the backbone load. 3.7 SD/HD Stream Multicast Edge Switch This profile includes an SD/HD stream mix of 120 streams of SD 3.75 Mb/s and 30 streams of HD at 15.0 Mb/s destined for uncongested output port(s). A mix of SD and HD streams are typical in many current head ends where HD streams are still in limited availability or must be limited due to downstream plant limitations. This profile assumes that all input streams can be destined for any single output port and that the output port will never be oversubscribed. MDI Characteristic Curves for the following cases are required. Each row Welch,Sagheer,Salyam, Expires January 2007 [Page 14] Asghar,Nawaz, Dufour Switch Testing for Streaming Media Applications July 2006 below varies the number of input streams per input port. For each of these combinations, the MDI Characteristic Curves are plotted when the streams are transmitted out 1, 2, 4, 8 ports and the maximum number of available output ports for the device under test. This profile is representative of edge multicast applications where there are a mix of streams encoded as SD and HD and the switch is distributing potentially hundreds of streams from up stream distribution networks. The distributed streams are then forwarded to STBs directly or via DSLAMs. In this application the streams use multicast distribution mechanisms and the switch must replicate inbound streams to all requesting ports. This profile should include the configuration in which all output ports are requesting all possible input streams simultaneously. No. of switch input No. of Streams per No. out ports Ports input port (SD/HD) 1 120/30 1, 2, 4, 8, max 2 60/15 1, 2, 4, 8, max 4 30/7 1, 2, 4, 8, max 8 15/3 1, 2, 4, 8, max 4. Test Stream Source The stream sources used for testing should be capable of generating the required number of test streams at the appropriate bit rates and with appropriate addressing and QoS tagging flexibility as described in the test Profiles. They should be able to generate the streams with industry common, streaming media protocol encapsulations such as ISO 13818 Transport Streams over UDP/IP and/or RTP as identified as part of the profile description used for testing. The encapsulations and resultant packet sizes should represent realistic traffic characteristics as described in the profile. The stream sources must also have the capability to increase the MDI Delay Factor (cumulative jitter) of the test streams to be able to create the output MDI Characteristic Curves. A given Delay Factor can be created with a variety of specific inter-packet gap times and burst sizes. For the tests in this document, the stream sources should use the minimum inter-packet gap times compliant with the link layer protocol employed to create elevated Delay Factors. This Welch,Sagheer,Salyam, Expires January 2007 [Page 15] Asghar,Nawaz, Dufour Switch Testing for Streaming Media Applications July 2006 causes the maximum utilization of switch queues when the switches are subject to multiple coincident input streams. The stream sources must also have the capability to synchronize the test streams such that they arrive at the switch being tested simultaneously. This causes the maximum utilization of switch queues and represents worst case test conditions. Such transient conditions can be encountered on realistic deployed networks. 5. Test Stream Analyzer The output stream analyzer used for testing should be capable of identifying analyzing the MDI for all streams simultaneously that the device or system under test produces. 6. Test Procedure Configure a test profile with appropriate stimulus streams and stream monitors and begin transmission. Run each profile test for two minutes to assure no packet loss with this load and then record results. Note that since an MDI Characteristic curve point must have no packet loss, a test can be terminated as soon as packet loss is detected (that is, as soon as the media loss rate (MLR) is greater than zero.) If no loss is detected, the test should run for at least two minutes to determine the maximum stream DF. After at least two minutes of operation, determine the test stream which has experienced the largest increase in DF by comparing each test stream’s maximum output DF measured during the test and subtracting that test stream’s input DF. Record that output stream’s maximum DF on the MDI Characteristic Curve described below. For profiles that include multiple types of input streams such as SD, HD, etc., an MDI Characteristic Curve for each type of stream is required; therefore, the comparison of input to output DF must be executed for each stream of the same type. In the event of loss, decrease the number of input test streams and perform the measurement described above to acquire other DF points on the MDI Characteristic Curve. Welch,Sagheer,Salyam, Expires January 2007 [Page 16] Asghar,Nawaz, Dufour Switch Testing for Streaming Media Applications July 2006 Begin with the input streams’ nominal DF and increase the input streams’ Delay Factor in steps of 2ms and repeat the Profile’s test to obtain other members of the family of Characteristic Curves for the Profile, as shown below. 7. MDI Characteristic Curve The MDI Characteristic Curve is a graph of the output streams’ delay factor vs. the number of lossless output streams that can be achieved for a given test profile, while holding the input Delay Factor constant as shown below. Output DF (ms) 9| ^^^^^^(input DF=5) 8| ^^ 7| ^ 6| ^ 5|^^ ********(input DF=3) 4| *** 3|***** 2| 1| +----+---+---+---+---+---Lossless stream count 10 20 30 40 50 Example MDI Characteristic Curve For the above example, note that for ideal input streams which have low cumulative jitter (DF = 3 curve), 40 is the maximum number of output streams without loss that the tested switch delivers since this is where the input DF = 3 curve terminates. When the streams’ input DF is increased to 5 ms, the tested switch delivers only up to 30 streams without loss. Note also that as the number of input streams increase, the output DF also typically changes. Referring to the input DF=5 curve, while the maximum number of lossless streams is 30, at 30 output streams the output DF has reached 9 ms vs 5 ms when the switch is less loaded and delivering less streams. Welch,Sagheer,Salyam, Expires January 2007 [Page 17] Asghar,Nawaz, Dufour Switch Testing for Streaming Media Applications July 2006 For profiles that include multiple types of input streams such as SD, HD, etc., an MDI Characteristic Curve for each type of stream is required. Optionally, separate MDI Characteristic Curves may be generated based on other stream characteristics such as priority, output port number, destination address type, etc. 8. Application of Test Data MDI Characteristic curves can be used in conjunction with stream source characteristics and stream consuming device characteristics to determine whether the transport system is adequately configured to deliver acceptable stream quality. For example, by measuring the worst case DF of a network device’s input stream and applying that DF to the network device’s MDI Characteristic Curve, the output MDI curve for that device indicates the worst case output performance to be expected for that stream. That value can be compared to the requirements of the downstream stream consuming device to determine if it has an adequate dejitter buffer to deliver lossless performance. 9. Summary A supplementary procedure for characterizing the quality of streaming media flows by network switches and systems was described. The results of the testing are documented in a series of Characteristic Curves representing the MDI footprint of the device under test. These curves describe a realistic, practical, and convenient method of describing the cumulative jitter performance of the device. They can be used to compare the performance of switches of various design, configuration, setup, and manufacturer and to infer whether so characterized devices will deliver the needed streaming media quality. For applications other than streaming media performance, previously existing methods described above can and should be used to represent switch performance. The Media Delivery Index metric is employed as the media quality indicator. It conveniently captures cumulative stream jitter and Welch,Sagheer,Salyam, Expires January 2007 [Page 18] Asghar,Nawaz, Dufour Switch Testing for Streaming Media Applications July 2006 packet loss performance through its delay factor and media loss rate components. Comments are solicited and should be addressed to the authors via Jim.Welch@IneoQuest.com. 10. Security Considerations The measurements identified in this document do not directly affect the security of a network or user. Actions taken in response to these measurements which may affect the available bandwidth of the network or availability of a service are beyond the scope of this document. While most applications for the testing described is envisioned to be performed in a laboratory or pre-deployment setting, performing the measurements described in this document only requires examination of payload header information, such as MPEG transport stream headers and/or RTP headers to determine nominal stream bit rate and sequence number information. Content may be encrypted without affecting these measurements. Therefore, content privacy is not expected to be a concern even on publicly accessible networks. 11. Informative References i1. R. Braden et al., `Resource Reservation Protocol ` Version 1 Functional Specification`, RFC 2205, 1997. i2. V. Raisanen, `Implementing Service Quality in IP Networks`, John Wiley & Sons Ltd., 2003. i3. Welch, Clark, ‘A Proposed Media Delivery Index’, RFC 4445, 2006. i4. Van Jacobson, Kathleen Nichols, Kedar Poduri, Internet Draft draft-ietf-diffserv-pdb-vw-00.txt, July 2000, “The ‘Virtual Wire’ Per-Domain Behavior” i5. S. Bradner, `Benchmarking Methodology for Network Interconnect Devices`, RFC2544, 1999. i6. R. Mandeville, `Benchmarking Methodology for LAN Switching Devices`, RFC2889, 2000. i7. `Perceptual evaluation of speech quality (PESQ): An objective method for end-to-end speech quality assessment of narrow-band telephone networks and speech codecs`, ITU-T P.862, February 2001 i8. `Network performance objectives for IP-based services`, ITU-T Y.1541 Welch,Sagheer,Salyam, Expires January 2007 [Page 19] Asghar,Nawaz, Dufour Switch Testing for Streaming Media Applications July 2006 i9. `The E-model, a computational model for use in transmission planning`, ITU-T G.107, February 2003 12. Acknowledgments The authors gratefully acknowledge the contributions of Marc Todd and Jesse Beeson of IneoQuest Technologies, Inc. 13. Authors' Address James Welch IneoQuest Technologies, Inc 170 Forbes Blvd Mansfield, Massachusetts 02048 508 618 0312 Jim.Welch@ineoquest.com Muhammad Waris Sagheer Cisco Systems, Inc 170 West Tasman Drive San Jose, California 95134-1706 408 853 6682 waris@cisco.com Sameer Satyam Cisco Systems, Inc. 170 West Tasman Drive San Jose, California 95134-1706 408 525 4697 sameers@cisco.com Javed Asghar Cisco Systems 170 West Tasman Drive San Jose, California 95134-1706 408 853 4078 jasghar@cisco.com Syed Nawaz Cisco Systems 170 West Tasman Drive Welch,Sagheer,Salyam, Expires January 2007 [Page 20] Asghar,Nawaz, Dufour Switch Testing for Streaming Media Applications July 2006 San Jose, California 95134-1706 408 853 5453 snawaz@cisco.com Andre Dufour Agilent Technologies 2500-4710 Kingsway Burnaby, BC V5H 4M2 Canada 604 454 3405 adufour@agilent.com 14. Copyright Notice Copyright (C) The Internet Society (2006). This document is subject to the rights, licenses and restrictions contained in BCP 78, and except as set forth therein, the authors retain all their rights. 15. Disclaimer This document and the information contained herein are provided on an 'AS IS' basis and THE CONTRIBUTOR, THE ORGANIZATION HE/SHE REPRESENTS OR IS SPONSORED BY (IF ANY), THE INTERNET SOCIETY AND THE INTERNET ENGINEERING TASK FORCE DISCLAIM ALL WARRANTIES, EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO ANY WARRANTY THAT THE USE OF THE INFORMATION HEREIN WILL NOT INFRINGE ANY RIGHTS OR ANY IMPLIED WARRANTIES OF MERCHANTABILITY OR FITNESS FOR A PARTICULAR PURPOSE.' 16. Intellectual Property The IETF takes no position regarding the validity or scope of any Intellectual Property Rights or other rights that might be claimed to pertain to the implementation or use of the technology described in this document or the extent to which any license under such rights might or might not be available; nor does it represent that it has made any independent effort to identify any such rights. Information on the ISOC's procedures with respect to rights in ISOC Documents can be found in BCP 78 and BCP 79. Copies of IPR disclosures made to the IETF Secretariat and any assurances of licenses to be made available, or the result of an Welch,Sagheer,Salyam, Expires January 2007 [Page 21] Asghar,Nawaz, Dufour Switch Testing for Streaming Media Applications July 2006 attempt made to obtain a general license or permission for the use of such proprietary rights by implementers or users of this specification can be obtained from the IETF on-line IPR repository at http://www.ietf.org/ipr. The IETF invites any interested party to bring to its attention any copyrights, patents or patent applications, or other proprietary rights that may cover technology that may be required to implement this standard. Please address the information to the IETF at ietf- ipr@ietf.org. Welch,Sagheer,Salyam, Expires January 2007 [Page 22] Asghar,Nawaz, Dufour