Network Working Group INTERNET-DRAFT Expires in: November 2006 Scott Poretsky Reef Point Systems Vijay Gurbani Lucent Technologies Carol Davids Illinois Institute of Technology May 2006 Terminology for Benchmarking SIP Networking Devices Intellectual Property Rights (IPR) statement: By submitting this Internet-Draft, each author represents that any applicable patent or other IPR claims of which he or she is aware have been or will be disclosed, and any of which he or she becomes aware will be disclosed, in accordance with Section 6 of BCP 79. Status of this Memo Internet-Drafts are working documents of the Internet Engineering Task Force (IETF), its areas, and its working groups. Note that other groups may also distribute working documents as Internet-Drafts. Internet-Drafts are draft documents valid for a maximum of six months and may be updated, replaced, or obsoleted by other documents at any time. It is inappropriate to use Internet-Drafts as reference material or to cite them other than as "work in progress." The list of current Internet-Drafts can be accessed at http://www.ietf.org/ietf/1id-abstracts.txt. The list of Internet-Draft Shadow Directories can be accessed at http://www.ietf.org/shadow.html. Copyright Notice Copyright (C) The Internet Society (2006). ABSTRACT This document provides a terminology for benchmarking SIP performance in networking devices. Terms are included for test components, test setup parameters, and performance benchmark metrics for black-box benchmarking of SIP networking devices. The Performance Benchmark Metrics are obtained for the SIP Control Plane and Media Plane. The terms are intended for use in a companion Methodology document for complete performance characterization of a device in a variety of network conditions making it possible to compare performance of different devices. It is critical to provide Test Setup Parameters and a Methodology document for SIP performance benchmarking because SIP allows a wide range of configuration and operational conditions that can influence performance benchmark measurements. Benchmarks can be applied to a variety of SIP networking device including SIP Servers, Session Border Controllers (SBCs), and Security Gateways (SEGs) as a DUT or in combination as a SUT. Poretsky, Gurbani, Davids [Page 1] INTERNET-DRAFT Benchmarking Terminology for May 2006 SIP Networking Devices Table of Contents 1. Introduction .................................................2 2. Existing definitions .........................................3 3. Term definitions..............................................3 3.1 Test Components........................................... 3.1.1 SIP Control Plane.................................... 3.1.2 SIP Media Plane...................................... 3.1.3 Emulated Agents...................................... 3.1.4 Session Server....................................... 3.1.5 Security Gateway..................................... 3.1.6 Control Session...................................... 3.1.7 Completed Control Session............................ 3.1.8 Session Attempt...................................... 3.1.9 Session Setup........................................ 3.1.10 Session Teardown.................................... 3.2 Test Setup Parameters..................................... 3.2.1 SIP Transport Protocol............................... 3.2.2 Intended Session Duration............................ 3.2.3 Measured Session Duration............................ 3.2.4 Intended Session Rate................................ 3.2.5 Intended Session Attempt Rate........................ 3.2.6 Media Streams per Session............................ 3.2.7 Media Packet Size.................................... 3.2.8 Media Offered Load, per Media Stream................. 3.2.9 Media Offered Load, Aggregate........................ 3.3 Benchmarks................................................ 3.3.1 Successful Session Rate.............................. 3.3.2 Successful Session Attempt Rate...................... 3.3.3 Standing Session Capacity............................ 3.3.4 Session Completion Rate.............................. 3.3.5 Busy Hour Session Connects (BHSC).................... 3.3.6 Busy Hour Session Attempts (BHSA).................... 3.3.7 Session Setup Delay.................................. 3.3.8 Session Teardown Delay............................... 3.3.9 Standing Sessions.................................... 4. IANA Considerations........................................... 5. Security Considerations....................................... 6. Acknowledgements.............................................. 7. Normative References.......................................... 8. Author's Address.............................................. Poretsky, Gurbani, Davids [Page 1] INTERNET-DRAFT Benchmarking Terminology for May 2006 SIP Networking Devices 1. Introduction Service Providers are now planning VoIP and Multimedia network deployments using the IETF developed Session Initiation Protocol (SIP). Through SIP, service providers will be able to have rich service offerings and many even plan to turn off their Public Switch Telephone Network within five years. VoIP has led to development of new networking devices including SIP Servers, Session Border Controllers, and Security Gateways. The mix of voice and IP functions in this variety of devices has produced inconsistencies in vendor reported performance metrics and has caused confusion in the service provider community. SIP allows a wide range of configuration and operational conditions that can influence performance benchmark measurements. When defining SIP performance benchmarks it is critical to also provide definitions for Test Setup Parameters and a corresponding Methodology document for SIP performance benchmarking. This enables benchmarks to be understood, fairly compared, and repeatable. This document provides the benchmarking terms for performance benchmarking the SIP control and media planes. Terms are included for Test Components, Test Setup Parameters, and Benchmarks. All benchmarks are black-box measurements of the SIP Control and Media Planes. It is intended that these terms be used in a companion Methodology document. The benchmarks can be used to compare a variety of SIP networking device including SIP Servers, Session Border Controllers (SBCs), and Security Gateways (SEGs) as a DUT or in combination as a SUT. 2. Existing definitions The key words "MUST", "MUST NOT", "REQUIRED", "SHALL", "SHALL NOT", "SHOULD", "SHOULD NOT", "RECOMMENDED", "MAY", and "OPTIONAL" in this document are to be interpreted as described in BCP 14, RFC 2119. RFC 2119 defines the use of these key words to help make the intent of standards track documents as clear as possible. While this document uses these keywords, this document is not a standards track document. The term Throughput is defined in RFC 2544. Poretsky, Gurbani, Davids [Page 2] INTERNET-DRAFT Benchmarking Terminology for May 2006 SIP Networking Devices 3. Term Definitions 3.1 Test Components 3.1.1 SIP Control Plane Definition: SIP Signaling to establish and terminate a SIP session. Discussion: It is critical to performance benchmark a device's SIP Control Plane. SIP sessions can be established directly between two SIP User Agent, via a Server, or via a series of Servers. Measurement Units: N/A Issues: None See Also: SIP Media Plane Emulated Agents 3.1.2 SIP Media Plane Definition: Media transfer that occurs directly between two SIP end stations after a SIP session is established. Discussion: The Media Plane is analagous to the Data Plane. Packets for the SIP Control Plane and the SIP Media Plane traverse different paths, which can produce variation in performance. For this reason it is necessary to benchmark performance of the SIP Control Plane and the SIP Media Plane. Measurement Units: N/A Issues: None See Also: SIP Control Plane Emulated Agents 3.1.4 Emulated Agents Definition: Device in test topology that initates/responds to SIP signaling as a session endpoint and sources/receives associated media for established connections. This device is a Tester. Discussion: Measurement Units: N/A Issues: None See Also: SIP Media Plane SIP Control Plane 3.1.5 Session Server Definition: Device in test topology that acts as proxy between Emulated Agents. This device is either a DUT or componenet of a SUT. Discussion: The Session Server may be a Proxy Server or Session Border Controller (SBC). Measurement Units: N/A Issues: None See Also: SIP Control Plane 3.1.6 Security Gateway Definition: Device in test topology that provides SIP DoS Protection for the Emulated Agents and Session Server Discussion: Security Gateways may include additional functionality, but that is beyond the scope of this work item. When testing a Security Gateway, the Security Gateway MUST be tested with the Session Server as a SUT. The Session Server may have an integrated Security Gateway so that it is a DUT. Measurement Units: N/A Issues: None See Also: 3.1.7 Control Session Definition: A SIP control session without media that includes setup and teardown from BYE exchanged between Tester and DUT/SUT. Discussion: Measurement Units: N/A Issues: None See Also: 3.1.8 Completed Control Session Definition: A SIP control session with media that includes setup and teardown from BYE exchanged between Tester and DUT/SUT. Discussion: Measurement Units: N/A Issues: None See Also: 3.1.9 Session Attempt Definition: An attempt by the Tester to establish a session on the DUT/SUT. Discussion: Measurement Units: N/A Issues: None See Also: 3.1.10 Session Setup Definition: A successful session establishment betweeen the Tester and DUT/SUT. Discussion: Measurement Units: N/A Issues: None See Also: 3.1.11 Session Teardown Definition: A successful session disconnect betweeen the Tester and DUT/SUT. Discussion: Measurement Units: N/A Issues: None See Also: 3.2 Test Setup Parameters 3.2.1 SIP Transport Protocol Definition: Description of the protocol used for transport of the SIP Control Plane. Discussion: Performance benchmarks may vary for the same SIP networking device as SIP TCP or SIP UDP is used. For this reason it is necessary to measure the SIP Performance Benchmarks with use of SIP TCP and SIP UDP. Performance Benchmarks MUST report the SIP Transport Protocol used to obtain the results. Measurement Units: TCP or UDP Issues: None See Also: 3.2.2 Intended Session Duration Definition: Configuration on the Tester for time from session establishment to BYE. Discussion: The Intended Session Duration is configured on the Tester. This value is used for all sessions. When benchmarking Intended Session Attempt Rate instead of Intended Session Rate then the effective value of the Intended Session Duration is infinite. Measurement Units: seconds Issues: None See Also: Intended Session Rate Intended Session Attempt Rate 3.4.1 Measured Session Duration Definition: Average measurement on the DUT/SUT for time from session establishment to BYE. Discussion: The value of the Measured Session Duration MAY not equal the Intended Session Duration. This parameter requires that the session duration be measured for every session through the test duration. Measurement Units: seconds Issues: None See Also: Intended Session Rate Intended Session Attempt Rate 3.2.3 Intended Session Attempt Rate Definition: Configuration on the Tester for number of sessions to establish per continuous one-second intervals with the sessions not terminated (remain established for test duration). Discussion: the Intended Session Attempt Rate can cause variation in performance benchmark measurements. Measurement Units: session attempts per second (saps) Issues: None See Also: Measured Session Duration Intended Session Rate 3.2.2 Intended Session Rate Definition: Configuration on the Tester for number of sessions to establish per continuous one-second intervals with the sessions terminated at the Measured Session Duration. Discussion: For a fixed value of Intended Session Rate and Intended Session Attempt Rate, the Intended Session Rate is more stressful on the DUT/SUT since it must process sessions setups and teardowns conncurrently during each one-second interval. Measurement Units: sessions per second (sps) Issues: None See Also: Measured Session Duration Intended Session Attempt Rate 3.2.4 Media Streams per Session Definition: Constant number of media streams offered for each session. Discussion: For a single benchmark test, all sessions use the same number of Media Streams per Session. Presence of media streams and the number of media streams per session can cause variation in performance benchmark measurements. The RECOMMENDED values for Media Streams per Session are 0,1,2,3,4, but higher values can be used. Measurement Units: media streams per session (msps) Issues: None See Also: 3.2.5 Media Packet Size Definition: Fixed size of packets used for media. Discussion: Measurement Units: bytes Issues: None See Also: 3.2.6 Media Offered Load, per Media Stream Definition: The constant amount of media traffic offered by the Tester to the DUT/SUT for each media stream. Discussion: For a single benchmark test, all sessions use the same Media Offered Load, per Media Stream. Measurement Units: pps Issues: None See Also: 3.2.7 Media Offered Load, Aggregate Definition: The total amount of media traffic offered by the Tester to the DUT/SUT. Discussion: Measurement Units: pps Issues: None See Also: 3.3 Benchmarks 3.3.1 Successful Session Rate Definition: Maximum number of sessions successfully established per continuous one-second intervals with the sessions not terminated (remain established for test duration). Discussion: This benchmark is obtained with zero failure in which 100% of the sessions introduced by the Tester successfuly establish. The maximum value is obtained by testing to failure. Measurement Units: sessions per second (sps) Issues: None See Also: Intended Session Rate Successful Session Attempt Rate 3.3.2 Successful Session Attempt Rate Definition: Configuration on the Tester for number of sessions to establish per continuous one-second intervals with the sessions terminated at the Measured Session Duration. Discussion: Measurement Units: session attempts per second (saps) Issues: None See Also: Intended Session Attempt Rate Successful Session Rate 3.3.3 Standing Session Capacity Definition: The maximum number of SIP sessions that the DUT/SUT can simultaneously have established. Discussion: The Standing Session Capacity must be reported with the Successful Session Attempt Rate used to reach the maximum. Measurement Units: sessions Issues: None See Also: Successful Session Attempt Rate 3.3.4 Session Completion Rate Definition: The percentage of sessions that successfully establish and terminate for the duration of a benchmarking test. Discussion: Session Completion Rate is a benchmark for session success. The duration for measuring this benchmark is to be specified in the Methodology. When Session Completion Rate is reported, the Successful Session Rate, Measured Session Duration, Media Streams per Session, and Media Offered Load per Media Stream MUST also be reported. Measurement Units: Percentage, % Issues: None See Also: Successful Session Rate Measured Session Duration Media Streams per Session Media Offered Load per Media Stream 3.3.5 Busy Hour Session Connects (BHSC) Definition: The number of sessions that successfully establish and terminate for one-hour. Discussion: When BHSC is reported, the Successful Session Rate, Measured Session Duration, Media Streams per Session, and Media Offered Load per Media Stream MUST also be reported. Measurement Units: Percentage, % Issues: None See Also: Successful Session Rate Measured Session Duration Media Streams per Session Media Offered Load per Media Stream 3.3.6 Busy Hour Session Attempts (BHSA) Definition: The number of sessions that successfully establish for one-hour. Discussion: Sessions do not terminate for the BHSA. When BHSA is reported, the Successful Session Rate, Measured Session Duration, Media Streams per Session, and Media Offered Load per Media Stream MUST also be reported. Measurement Units: Percentage, % Issues: None See Also: Successful Session Rate Measured Session Duration Media Streams per Session Media Offered Load per Media Stream 3.3.7 Session Setup Delay Definition: The average time for a session to establish. Discussion: Time is from the Tester to signal the first INVITE. Session Setup Delay MUST be measured for every established session to calculate the average. Session Setup Delay MUST be measured at the Successful Setup Attempt Rate. Measurement Units: msec Issues: None See Also: Successful Setup Attempt Rate 3.3.8 Session Teardown Delay Definition: The average time for a session to teardown. Discussion: Time is from the Tester to signal the BYE. Session Teardown Delay MUST be measured for every established session to calculate the average. Session Setup Delay MUST be measured with the rate of teardowns configured to the value of the Successful Setup Attempt Rate. Measurement Units: msec Issues: None See Also: Successful Setup Attempt Rate 3.3.9 Standing Sessions Definition: Measurement of the number of sessions concurrently established on the DUT/SUT. Discussion: The number of Standing Sessions is influenced by the Session Duration and the Session Rate (or Session Attempt Rate). Benchmarks MUST be reported with the maximum and average Standing Sessions for the DUT/SUT. In order to determine the maximum and average Standing Sessions on the DUT/SUT for the duration of the test it is necessary to make periodic measurements of the number of Standing Sessions on the DUT/SUT. The recommended value for the measurement period is 1 second. Measurement Units: sessions Issues: None See Also: Session Duration Session Rate Session Attempt Rate Poretsky, Gurbani, Davids [Page ] INTERNET-DRAFT Benchmarking Terminology for May 2006 SIP Networking Devices 4. IANA Considerations This document requires no IANA considerations. 5. Security Considerations Documents of this type do not directly affect the security of Internet or corporate networks as long as benchmarking is not performed on devices or systems connected to production networks. 6. Acknowledgements 7. References 7.1 Normative References [Ba91] Bradner, S. "Benchmarking Terminology for Network Interconnection Devices", RFC1242, July 1991. [Ba99] Bradner, S. and McQuaid, J., "Benchmarking Methodology for Network Interconnect Devices", RFC 2544, March 1999. [Ma98] Mandeville, R., "Benchmarking Terminology for LAN Switching Devices", RFC 2285, February 1998. [Ro02] Rosenberg, J., Schulzrinne, H., Camarillo, G., Johnston, A., Peterson, J., Sparks, R., Handley, M. and E. Schooler, "SIP: Session Initiation Protocol", RFC 3261, June 2002. [Ga05] Garcia-Martin, M., "Input 3rd-Generation Partnership Project (3GPP) Release 5 Requirements on the Session Initiation Protocol (SIP)", RFC 4083, May 2005. [Sp06] Sparks, R., et al, "Session Initiation Protocol (SIP) Torture Test Messages", RFC 4475, May 2006. [Ma06] Malas, D. "SIP Performance Metrics", draft-malas-performance-metrics-01.txt, work in progress, May 2006. [Li06] Lingle, K., Mule, J., Maeng, J., Walker, D., "Management Information Base for the Session Initiation Protocol (SIP)", draft-ietf-sip-mib-10.txt, work in progress, March 2006. 7.2 Informative References None 8. Author's Address Scott Poretsky Reef Point Systems 8 New England Executive Park Burlington, MA 01803 USA Phone: + 1 508 439 9008 EMail: sporetsky@reefpoint.com Vijay Gurbani 2000 Lucent Lane Lucent Technologies Room 6G-440 Naperville, IL 60566 USA Phone: + 1 630 224 0216 Email: vkg@lucent.com Carol Davids Illinois Institute of Technology Rice Campus 201 East Loop Road Wheaton, IL 60187 USA Phone: + 1 630 682 6000 Email: davids@iit.edu Poretsky, Gurbani, Davids [Page ] INTERNET-DRAFT Benchmarking Terminology for May 2006 SIP Networking Devices Full Copyright Statement Copyright (C) The Internet Society (2006). This document is subject to the rights, licenses and restrictions contained in BCP 78, and except as set forth therein, the authors retain all their rights. This document and the information contained herein are provided on an "AS IS" basis and THE CONTRIBUTOR, THE ORGANIZATION HE/SHE REPRESENTS OR IS SPONSORED BY (IF ANY), THE INTERNET SOCIETY AND THE INTERNET ENGINEERING TASK FORCE DISCLAIM ALL WARRANTIES, EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO ANY WARRANTY THAT THE USE OF THE INFORMATION HEREIN WILL NOT INFRINGE ANY RIGHTS OR ANY IMPLIED WARRANTIES OF MERCHANTABILITY OR FITNESS FOR A PARTICULAR PURPOSE. Intellectual Property The IETF takes no position regarding the validity or scope of any Intellectual Property Rights or other rights that might be claimed to pertain to the implementation or use of the technology described in this document or the extent to which any license under such rights might or might not be available; nor does it represent that it has made any independent effort to identify any such rights. Information on the procedures with respect to rights in RFC documents can be found in BCP 78 and BCP 79. Copies of IPR disclosures made to the IETF Secretariat and any assurances of licenses to be made available, or the result of an attempt made to obtain a general license or permission for the use of such proprietary rights by implementers or users of this specification can be obtained from the IETF on-line IPR repository at http://www.ietf.org/ipr. The IETF invites any interested party to bring to its attention any copyrights, patents or patent applications, or other proprietary rights that may cover technology that may be required to implement this standard. Please address the information to the IETF at ietf- ipr@ietf.org. Acknowledgement Funding for the RFC Editor function is currently provided by the Internet Society. Poretsky, Gurbani, Davids [Page]