Internet Engineering Task Force Audio Video Transport WG Internet Draft J.Rosenberg,H.Schulzrinne draft-ietf-avt-rtcptest-00.txt Bell Laboratories,Columbia U. February 9, 1999 Expires: August 1999 Conformance Tests for RTP Scalability Algorithms STATUS OF THIS MEMO This document is an Internet-Draft and is in full conformance with all provisions of Section 10 of RFC2026. Internet-Drafts are working documents of the Internet Engineering Task Force (IETF), its areas, and its working groups. Note that other groups may also distribute working documents as Internet- Drafts. Internet-Drafts are draft documents valid for a maximum of six months and may be updated, replaced, or obsoleted by other documents at any time. It is inappropriate to use Internet- Drafts as reference material or to cite them other than as work in progress. The list of current Internet-Drafts can be accessed at http://www.ietf.org/ietf/1id-abstracts.txt The list of Internet-Draft Shadow Directories can be accessed at http://www.ietf.org/shadow.html. Abstract This document defines a set of tests that can be run on an RTP imple- mentation to determine the level of conformance with the various sca- lability algorithms defined in the draft version of RTP. These tests do not require access to internal implementations, and use black box testing techniques to determine compliance. 1 Introduction In the process of moving from proposed to draft standard, the Real Time Transport Protocol (RTP) [1] has added a number of algorithms to support better scalability. These algorithms include forward recon- sideration, reverse reconsideration, and BYE reconsideration [2,3]. All are aimed at reducing RTCP congestion during periods of rapid change in group membership. These changes are all backwards J.Rosenberg,H.Schulzrinne [Page 1] Internet Draft RTP Test February 9, 1999 compatible with RFC1889. An RTP session with a mix of new and old implementations will function, and the level of improvement in congestion will depend on the fraction of new implementations. In order to progress to draft standard, interoperability of these features must be demonstrated. However, since the algorithms do not make use of new protocol messages, and are purely local computations, some other means of testing is needed. This document proposes a series of black box tests to demonstrate correctness of implementa- tion of these algorithms. 2 Testing Architecture The basic architecture for performing the tests is shown in Figure 1. -------------- | | | test | | instrument | | | -------------- | -------------------------------- LAN | ---------------- | | | RTP | | implementation | | | ---------------- Figure 1: Test Architecture The test instrument is connected on the same LAN as the RTP implemen- tation being tested. It is assumed that the test instrument is pre- configured with the addresses and ports used by the RTP implementa- tion, and is also aware of the RTCP bandwidth and sender/receiver fractions. The tests can be conducted using either multicast or uni- cast. The test instrument must be capable of sending arbitrarily crafted RTP and RTCP packets to the RTP implementation. The instrument should also be capable of receiving packets sent by the RTP implementation, J.Rosenberg,H.Schulzrinne [Page 2] Internet Draft RTP Test February 9, 1999 parsing them, and computing metrics based on those packets. It is furthermore assumed that a number of basic controls over the RTP implementation exist. These controls are: o ability to force the implementation to send or not send RTP packets at any desired point in time; o ability to force the application to terminate its involvement in the RTP session, and for this termination to be known immediately to the test instrument. The second of these is required only for the test of BYE reconsidera- tion, and is the only aspect of these tests not easily implementable by pure automation. It will generally require manual intervention to terminate the session from the RTP implementation and to convey this to the test instrument through some non-RTP means. 3 Basic Behavior The first test is to verify basic correctness of the implementation of the RTCP transmission rules. This basic behavior consists of: o periodic transmission of RTCP packets; o randomization of the interval for RTCP packet transmission; o correct implementation of the randomization interval computa- tions, with unconditional reconsideration. These features are tested in the following manner. The RTP implemen- tation acts as a session receiver, and never sends any RTP packets. The implementation is configured with a large session bandwidth, say 1 Mb/s. This will cause the implementation to use the minimal inter- val of 5s rather than the small interval based on the session bandwidth and membership size. The implementation will generate RTCP packets at this minimal interval, on average. The test instrument generates no packets, but receives the RTCP packets generated by the implementation. When an RTCP packet is received, the time is noted by the test instrument. The difference in time between each pair of sub- sequent packets (called the interval) is computed. These intervals are stored, so that statistics based on these intervals can be com- puted. It is recommended that this observation process operate for at least 20 minutes. An implementation passes this test if the intervals have the follow- ing properties: J.Rosenberg,H.Schulzrinne [Page 3] Internet Draft RTP Test February 9, 1999 o the minimum interval is never less than 2 seconds or more than 2.5 seconds; o the maximum interval is never more than 7 seconds or less than 5.5 seconds; o the average interval is between 4.5 and 5.5 seconds; o the number of intervals between x and x+500ms is less than the number of intervals between x+500ms and x+1s, for any x. In particular, an implementation fails if the packets are sent with a constant interval. 4 Step join backoff The main purpose of the reconsideration algorithm is to avoid a flood of packets that might occur when a large number of users simultane- ously join an RTP session. Reconsideration therefore exhibits a back- off behavior in sending of RTCP packets when group sizes increase. This aspect of the algorithm can be tested in the following manner. The implementation begins operation. The test instrument waits for the arrival of the first RTCP packet. When it arrives, the test instrument notes the time and then immediately sends 100 RTCP RR packets to the implementation, each with a different SSRC and SDES CNAME. The test instrument should ensure that each RTCP packet is of the same length. The instrument should then wait until the next RTCP packet is received from the implementation, and the time of such reception is noted. Without reconsideration, the next RTCP packet will arrive within a short period of time. With reconsideration, transmission of this packet will be delayed. The earliest it can arrive depends on the RTCP session bandwidth, receiver fraction, and average RTCP packet size. The RTP implementation should be using the exponential averag- ing algorithm defined in the specification to compute the average RTCP packet size. Since this is dominated by the received packets (the implementation has only sent one itself), the average will be roughly equal to the length of the SDES packets sent by the test instrument. Therefore, the minimum amount of time between the first and second RTCP packets from the implementation is: T > 101* S/(B*Fr(e-1.5)*2) J.Rosenberg,H.Schulzrinne [Page 4] Internet Draft RTP Test February 9, 1999 Where S is the size of the RTCP packets, B is the RTCP bandwidth (normally five percent of the session bandwidth), Fr is the fraction of RTCP bandwidth allocated to receivers (normally 75 percent), and e is the natural exponent. Without reconsideration, this minimum inter- val is much smaller: Te > MAX(S/(B *Fr *(e-1.5)*2), 2.5/(e-1.5)) B should be chosen sufficiently small so that T is around 60 seconds. Reasonable choices for these parameters are B = 950 bits per second, and S = 1024 bits. An implementation passes this test if the interval between packets is not less than T above, and not more than 3 times T. The test may be repeated for the case when the RTP implementation is a sender. This is accomplished by having the implementation send RTP packets at least once a second. In this case, the interval between the first and second RTCP packets should be no less than: T > S/(B*Fs*(e-1.5)*2) 5 Steady State Behavior In addition to the basic behavior in section 3, an implementation should correctly implement a number of other, slightly more advanced features: o scale the RTCP interval with the group size; o correctly divide bandwidth between senders and receivers; o correctly compute the RTCP interval when the user is a sender The following tests are meant to test these features. 5.1 Scaling the Interval The implementation begins operation as a receiver. The test instru- ment waits for the first RTCP packet from the implementation. When it arrives, the test instrument notes the time, and immediately sends 50 RTCP RR packets and 50 RTCP SR packets to the implementation, each with a different SSRC and SDES CNAME. The test instrument then sends J.Rosenberg,H.Schulzrinne [Page 5] Internet Draft RTP Test February 9, 1999 50 RTP packets, using the 50 SSRC from the RTCP SR packets. The test instrument should ensure that each RTCP packet is of the same length. The instrument should then wait until the next RTCP packet is received from the implementation, and the time of such reception is noted. The difference between the reception of the RTCP packet and the reception of the previous is computed and stored. In addition, after every RTCP packet reception, the 100 RTCP and 50 RTP packets are retransmitted by the test instrument. This ensures that the sender and member status of the 100 users does not time out. The test instrument should collect the interval measurements figures for at least 100 RTCP packets. With 50 senders, the implementation should not try to divide the RTCP banwidth between senders and receivers, but rather group all users together and divide the RTCP bandwidth equally. The test is deemed successful if the average RTCP interval is within 5% of: T = 101* S/B Where S is the size of the RTCP packets sent by the test instrument, and B is the RTCP bandwidth. B should be chosen sufficiently small so that the value of T is on the order of tens of seconds or more. Rea- sonable values are S=1024 bits and B=3.4 kb/s. 5.2 Compensating for Senders The test of section 5.1 is repeated. However, the test instrument sends 10 RTP packets instead of 50, and 10 RTCP SR and 90 RTCP RR instead of 50 of each. In addition, the implementation is made to send at least one RTP packet between transmission of every one of its own RTCP packets. In this case, the average RTCP interval should be within 5% of: T = 11 * S / (B * Fs) Where S is the size of the RTCP packets sent by the test instrument, B is the RTCP bandwidth, and Fs is the fraction of RTCP banwidth allocated for senders (normally 25%). The values for B and S should be chosen small enough so that T is on the order of tens of seconds. Reasonable choices are S=1024 bits and B=1.5 kb/s. J.Rosenberg,H.Schulzrinne [Page 6] Internet Draft RTP Test February 9, 1999 6 Reverse Reconsideration The reverse reconsideration algorithm is effectively the opposite of the normal reconsideration algorithm. It causes the RTCP interval to be reduced more rapidly in response to decreases in the group member- ship. This is advantageous in that it keeps the RTCP information as fresh as possible, and helps avoids some premature timeout problems. These tests verify the basic correctness of reverse reconsideration. 6.1 Test I In this test, the implementation joins the session as a receiver. As soon as it sends its first RTCP packet, the test instrument sends 100 RTCP RR packets, each of the same length S, and a different SDES CNAME and SSRC in each. It then waits for the implementation to send another RTCP packet. Once it does, the test instrument sends 100 BYE packets, each one containing a different SSRC, but matching an SSRC from one of the RTCP packets. Each BYE should also be the same size as the RTCP packets sent by the test instrument. This is easily accomplished by using a BYE reason to pad out the length. The time of the next RTCP packet from the implementation is then noted. The delay T between this (the third RTCP packet) and the previous should be no more than: T < 3 * S/(B * Fr * (e-1.5) * 2) Where S is the size of the RTCP and BYE packets sent by the test instrument, B is the RTCP bandwidth, Fr is the fraction of the RTCP bandwidth allocated to receivers, and e is the natural exponent. B should be chosen such that T is on the order of tens of seconds. A reasonable choice is S=1024 bits and B=168 bits per second. This test demonstrates basic correctness of implementation. An imple- mentation without reverse reconsideration will not send its next RTCP packet for nearly 100 times as long as the above amount. 6.2 Test II In this test, the implementation joins the session as a receiver. As soon as it sends its first RTCP packet, the test instrument sends 100 RTCP RR packets, each of the same length S, followed by 100 BYE pack- ets, also of length S. Each RTCP packet carries a different SDES CNAME and SSRC, and is matched with precisely one BYE packet with the same SSRC. This will cause the implementation to see a rapid increase J.Rosenberg,H.Schulzrinne [Page 7] Internet Draft RTP Test February 9, 1999 and then rapid drop in group membership. The test is deemed succesful if the next RTCP packet shows up T seconds after the first, and T is within: 2.5/(e-1.5) < T < 7.5/(e-1.5) This tests correctness of the maintenance of the pmembers variable. An incorrect implementation might try to execute reverse reconsidera- tion every time a BYE is received, as opposed to only when the group membership drops below pmembers. If an implementation did this, it would end up sending an RTCP packet immediately after receiving the stream of BYE's. For this test to work, B must be chosen to be a large value, around 1Mb/s. 7 BYE Reconsideration The BYE reconsideration algorithm works in much the same fashion as regular reconsideration, except applied to BYE packets. When a user leaves the group, instead of sending a BYE immediately, it may delay transmission of its BYE packet if others are sending BYE's. The test for correctness of this algorithm is as follows. The RTP implementation joins the group as a receiver. The test instrument waits for the first RTCP packet. When the test instrument receives this packet, the test instrument immediately sends 100 RTCP RR pack- ets, each of the same length S, and each containing a different SSRC and SDES CNAME. Once the test instrument receives the next RTCP packet from the implementation, the RTP implementation is made to leave the RTP session, and this information is conveyed to the test instrument through some non-RTP means. The test instrument then sends 100 BYE packets, each with a different SSRC, and each matching an SSRC from a previously transmitted RTCP packet. Each of these BYE packets is also of size S. Immediately following the BYE packets, the test instrument sends 100 RTCP RR packets, using the same SSRC/CNAMEs as the original 100 RTCP packets. The test is deemed successful if the implementation either never sends a BYE, or if it does, the BYE is not received by the test instrument earlier than T seconds after the implementation left the session, where T is: T = 100 * S / ( 2 * (e-1.5) * B * Fr) J.Rosenberg,H.Schulzrinne [Page 8] Internet Draft RTP Test February 9, 1999 S is the size of the RTCP and BYE packets, e is the natural exponent, B is the RTCP bandwidth, and Fr is the RTCP bandwidth fraction for receivers. S and B should be chosen so that T is on the order of 50 seconds. A reasonable choice is S=1024 bits and B=1.1 kb/s. The transmission of the RTCP packets is meant to verify that the implementation is ignoring non-BYE RTCP packets once it decides to leave the group. 8 Author's Addresses Jonathan Rosenberg Lucent Technologies, Bell Laboratories 101 Crawfords Corner Rd. Holmdel, NJ 07733 Rm. 4C-526 email: jdrosen@bell-labs.com Henning Schulzrinne Columbia University M/S 0401 1214 Amsterdam Ave. New York, NY 10027-7003 email: schulzrinne@cs.columbia.edu 9 Bibliography [1] H. Schulzrinne, S. Casner, R. Frederick, and V. Jacobson, "RTP: a transport protocol for real-time applications," Request for Comments (Proposed Standard) 1889, Internet Engineering Task Force, Jan. 1996. [2] J. Rosenberg and H. Schulzrinne, "Timer reconsideration for enhanced RTP scalability," (San Francisco, California), March/April 1998. [3] J. Rosenberg and H. Schulzrinne, "New results in RTP scalabil- ity," Internet Draft, Internet Engineering Task Force, Dec. 1997. Work in progress. J.Rosenberg,H.Schulzrinne [Page 9]