Network Working Group D. Malas Internet Draft Level 3 Communications Expires: December 2006 June 1, 2006 SIP Performance Metrics draft-malas-performance-metrics-02.txt Status of this Memo By submitting this Internet-Draft, each author represents that any applicable patent or other IPR claims of which he or she is aware have been or will be disclosed, and any of which he or she becomes aware will be disclosed, in accordance with Section 6 of BCP 79. Internet-Drafts are working documents of the Internet Engineering Task Force (IETF), its areas, and its working groups. Note that other groups may also distribute working documents as Internet- Drafts. Internet-Drafts are draft documents valid for a maximum of six months and may be updated, replaced, or obsoleted by other documents at any time. It is inappropriate to use Internet-Drafts as reference material or to cite them other than as "work in progress." The list of current Internet-Drafts can be accessed at http://www.ietf.org/ietf/1id-abstracts.txt The list of Internet-Draft Shadow Directories can be accessed at http://www.ietf.org/shadow.html This Internet-Draft will expire on November 1, 2006. Abstract This document defines the use of industry recommended reliability metrics for use with the SIP. Conventions used in this document The key words "MUST", "MUST NOT", "REQUIRED", "SHALL", "SHALL NOT", "SHOULD", "SHOULD NOT", "RECOMMENDED", "MAY", and "OPTIONAL" in this document are to be interpreted as described in RFC-2119 [1]. Table of Contents 1. Introduction...................................................2 Malas Expires December 1, 2006 [Page 1] Internet-Draft SIP Performance Metrics June 2006 2. SIP Performance Metrics........................................3 2.1. Session Request Delay (SRD)...............................3 2.2. Session Completion Delay (SCD)............................4 2.3. Session Duration Time (SDT)...............................4 2.4. Sessions Per Second (SPS).................................5 2.5. Session Attempts Per Second (SAPS)........................6 2.6. Session Establishment Rate (SER)..........................7 2.7. Session Defects (SD)......................................7 2.8. Ineffective Session Attempts (ISA)........................8 2.9. Session Disconnect Failures (SDF).........................8 2.10. Session Completion Rate (SCR)............................9 2.11. Session Success Rate (SSR)..............................10 3. Data Collection Considerations................................11 4. Security Considerations.......................................11 5. IANA Considerations...........................................11 6. Conclusions...................................................11 7. Acknowledgments...............................................11 8. References....................................................11 8.1. Normative References.....................................11 8.2. Informative References...................................12 Author's Addresses...............................................12 Intellectual Property Statement..................................12 Disclaimer of Validity...........................................13 Copyright Statement..............................................13 Acknowledgment...................................................13 1. Introduction SIP has become a standard among many service providers, vendors, and end users. Although there are many different standards for measuring the performance of signaling protocols, none of these have been adapted for use with SIP. This document is intended for providing a guideline for the above listed entities in providing a standard approach for measuring and reporting SIP performance metrics in a production environment. This will allow a common approach and understanding of expectations between service providers, vendors, and the users of those services. Not all metrics for performance map to all applications of the SIP. This document provides an overview of many different metrics, which may be used as an individual or set of metrics necessary based on the use of SIP. There are many metrics available for determining performance. Although this document contains a number of them, it is not intended to be exhaustive. Instead, it is designed to provide a common sub- set with a common agreed upon definition. This document does not Malas Expires December 1, 2006 [Page 2] Internet-Draft SIP Performance Metrics June 2006 provide any standard or benchmark information regarding IETF recommended performance criteria to compare any results derived from the following described metrics. Although these metrics may be used in a test environment, the IETF Benchmarking Methodology working group is currently working on a draft for this purpose. This draft will align terminology and methodologies where applicable, in order to maintain consistency among the IETF. 2. SIP Performance Metrics The following metrics may be utilized for all applications. In regards to all of the following metrics, message re-transmissions must be excluded in order to provide accurate metric results. 2.1. Session Request Delay (SRD) It is important session setup delay is calculated for both sessions ending in failure and success. In a successful request attempt, SSD is defined as the time interval from the moment the INVITE message containing the necessary information is passed by the originating agent or user to the intended mediation or destination agent until the first provisional response is received indicating an audible or visual status of the INVITE request. In SIP, the message indicating status would be a non-100 Trying provisional message received in response to an INVITE request. In some cases, a non-100 Trying provisional message is not received, but rather a 200 message is received as the first status message instead. In these situations, the 200 message would be used to calculate the interval. In a failed request attempt, the interval is defined from the INVITE request and a non-100 Trying provisional message or a failure indication response. A failure response is described as a 4XX, 5XX, or possible 6XX message. SRD = Time of Status Indicative Response - Time of INVITE SUM (Time of Status Indicative Response - Time of INVITE) ASRD = --------------------------------------------------------- SUM # of INVITE Requests ASRD = Average SRD The following flow provides an example of Session Request Delay: Malas Expires December 1, 2006 [Page 3] Internet-Draft SIP Performance Metrics June 2006 UA1 UA2 | | |INVITE | |---------------------> | | /\ 100| | <---------||----------| | SRD | | || | | \/ 180| | <---------------------| | | 2.2. Session Completion Delay (SCD) SCD is defined as the interval between sending a session completion message, such as a BYE, and receiving the subsequent 2XX acknowledgement. The following flow provides an example of this metric: UA1 UA2 | | |INVITE | |---------------------> | | 100| | <---------------------| | 180/200| | <---------------------| | BYE| |---------------------->| | /\ | | || | | SCD | | || | | \/ 200| |<----------------------| 2.3. Session Duration Time (SDT) SDT is usually calculated as an average and is defined as the duration of a dialog from receipt of a 200 OK response to an INVITE and an associated BYE message indicating dialog completion. SDT = Time of BYE - Time of 200 OK response to INVITE SUM (Time of BYE - Time of 200 OK response to INVITE) ASDT = ------------------------------------------------------ SUM # of INVITE w/ 200OK & BYE Malas Expires December 1, 2006 [Page 4] Internet-Draft SIP Performance Metrics June 2006 ASDT = Average SDT The following flow represents an example of the determination of this metric: UA1 UA2 | | |INVITE | |---------------------> | | 100| | <---------------------| | 180| | <---------------------| | 200| | <---------------------| | /\ | | || | | SDT | | || | |BYE \/ | |---------------------> | 2.4. Sessions Per Second (SPS) SPS is described as the number of sessions, which are setup in an incremental time period of one second. In order for a session to be setup, an INVITE must be processed with a subsequent response from the terminating UA or UAS associated with the initial INVITE. The following flow provides an example of a "session" related to this metric: Malas Expires December 1, 2006 [Page 5] Internet-Draft SIP Performance Metrics June 2006 UA1 UA2 | | |INVITE | |---------------------> | | /\ 100| | <---------||----------| | Session | | || | | \/ 180/200| | <---------------------| | | As described in the message flow, UA1 must receive a non-100 Trying provisional message or 200 in order for the establishment of a session. 2.5. Session Attempts Per Second (SAPS) SAPS is defined as the number of initial INVITE requests received by a UA or UAS per time increment of one second. Usually, this metric is relative to proxy servers and the maximum number of SAPS it is capable of processing before failure conditions begin to occur. In addition, the metric may be used to trend a specific traffic pattern. This metric may be extended to include any initial SIP related requests in addition to INVITE's, such as REGISTER requests, SUBSCRIBE requests, and others as described in [2] and related extensions. The following flow provides an example of a session attempt: UA1 UA2 | | |INVITE | |---------------------> | | | As is described by the figure, a session attempt does not require a response from UA2 in order to constitute an attempt; however, an attempt may be acknowledged in order to validate the terminating UA or UAS received the attempt. Malas Expires December 1, 2006 [Page 6] Internet-Draft SIP Performance Metrics June 2006 2.6. Session Establishment Rate (SER) SER is defined as the number of INVITE requests resulting in a 200 OK response, to the total number of attempted INVITE requests. # of INVITE Requests w/ associated 200OK SER = ---------------------------------------- Total # of INVITE Requests The following flow represents session establishment as described above: UA1 UA2 | | |INVITE | |---------------------> | | /\ 100| | <---------||----------| | || | | Session Established | | || 180| | <---------||----------| | \/ 200| | <---------------------| | | | | 2.7. Session Defects (SD) Session defects provide a subset of SIP failure responses, which consistently indicate a failure in dialog processing. Defects are necessary to provide input to calculations such as Defects per Million (DPM) or other similar metrics. These failure responses are in response to initial session setup requests, such as a new INVITE. The following failure responses provide a guideline for defective criterion: . 500 Server Internal Error . 503 Service Unavailable . 504 Server Timeout This set of failure responses was derived through correlating more granular ISUP failure responses as described in RFC 3398. Malas Expires December 1, 2006 [Page 7] Internet-Draft SIP Performance Metrics June 2006 2.8. Ineffective Session Attempts (ISA) Ineffective session attempts occur when a proxy or agent internally releases a setup request with a failed or congested condition. The following failure responses provide a guideline for this criterion: . 408 Request Timeout . 500 Server Internal Error . 503 Service Unavailable . 504 Server Timeout This set was derived in a similar manner as described in Section 2.7, in addition 408 failure responses can be indicative a congested state with a downstream element. This metric is calculated as a percentage of total session setup requests. The following calculation provides a guideline: # of ISA ISA % = -------------------------- Total # of INVITE Requests 2.9. Session Disconnect Failures (SDF) Session disconnect failures occur when an active session is terminated due to a failure condition that can be identified by a REASON header [5] in a BYE. This occurs, for example, when a user agent server (UAS) is controlling an IP or TDM (Time Division Multiplexing) media gateway, and the media gateway notifies the UAS of a failure condition causing the loss of media related to an established session. The UAS will release the session with a BYE, but should include a REASON header indicating the session was disconnected abnormally. The REASON value is utilized to determine the disconnect was a failure. This metric is calculated as a percentage of total session completed successfully as defined in Section 2.6. The following calculation provides a guideline: # of SDF’s SDF % = --------------------------------- Total # of INVITE Requests Malas Expires December 1, 2006 [Page 8] Internet-Draft SIP Performance Metrics June 2006 2.10. Session Completion Rate (SCR) A session completion, as described in this metric, is defined as a SIP dialog, which completes without failing due to a lack of response from an intended proxy, UAS, or UA. A session completes successfully when it begins with a setup request and ends with a session completion message. This metric is only used when at least one proxy is involved in the dialog. The following dialog [4] describes a successful session completion: Alice Proxy 1 Proxy 2 Bob | | | | | INVITE | | | |--------------->| | | | 407 | | | |<---------------| | | | ACK | | | |--------------->| | | | INVITE | | | |--------------->| INVITE | | | 100 |--------------->| INVITE | |<---------------| 100 |--------------->| | |<---------------| | | | | 180 | | | 180 |<---------------| | 180 |<---------------| | |<---------------| | 200 | | | 200 |<---------------| | 200 |<---------------| | |<---------------| | | | ACK | | | |--------------->| ACK | | | |--------------->| ACK | | | |--------------->| | Both Way RTP Media | |<================================================>| | | | BYE | | | BYE |<---------------| | BYE |<---------------| | |<---------------| | | | 200 | | | |--------------->| 200 | | | |--------------->| 200 | | | |--------------->| | | | | Malas Expires December 1, 2006 [Page 9] Internet-Draft SIP Performance Metrics June 2006 The following dialog describes an unsuccessful session completion: Alice Proxy 1 Proxy 2 Bob | | | | | INVITE | | | |--------------->| | | | 407 | | | |<---------------| | | | ACK | | | |--------------->| | | | INVITE | | | |--------------->| INVITE | | | 100 |--------------->| INVITE | |<---------------| 100 |--------------->| | |<---------------| | | | | INVITE | | | |--------------->| | | | | | | | INVITE | | | |--------------->| | | | | | | 408 | | | 408 |<---------------| | |<---------------| ACK | | | |--------------->| | | ACK | | | |--------------->| | | This metric is calculated as a percentage of total sessions completed successfully. The following calculation provides a guideline: # of Successfully Completed Sessions SCR % = --------------------------------------- Total # of Session Attempts 2.11. Session Success Rate (SSR) Session success rate is a subjective metric, but is included for usage to combine metrics providing a description of the overall service perspective a vendor or provider. It is defined as the percentage of successfully completed sessions compared to sessions, which fail due to ISA or SDF. The following calculation provides a guideline: SSR = 100% - (ISA% + SDF%) Malas Expires December 1, 2006 [Page 10] Internet-Draft SIP Performance Metrics June 2006 3. Data Collection Considerations The input necessary for these calculations may be collected in a number of different manners. It may be collected or retrieved from call detail records (CDR) or raw signaling information on a proxy, UA, or UAS. The information may also be transmitted through use of SNMP traps as described in the work in progress SIP MIB draft [6], or through a potential undefined new performance metric event package [3] retrieved via SUBSCRIBE requests. 4. Security Considerations Security should be considered in the aspect of securing the relative data utilized in providing input to the above calculations. All other aspects of security should be considered as described in [2]. 5. IANA Considerations There are no IANA considerations at this time. 6. Conclusions The proposed guideline provides a description of common performance metrics, and their defined use with SIP. The use of these metrics will provide a common viewpoint across all vendors, service providers, and customers. These metrics will likely be utilized in production SIP environments for providing input regarding Key Performance Indicators (KPI) and Service Level Agreement (SLA) indications. 7. Acknowledgments I would like to thank John Hearty for his efforts in scrubbing through the draft and providing insight regarding clarification of certain aspects described throughout the document. 8. References 8.1. Normative References [1] Bradner, S., "Key words for use in RFCs to Indicate Requirement Levels", BCP 14, RFC 2119, March 1997. Malas Expires December 1, 2006 [Page 11] Internet-Draft SIP Performance Metrics June 2006 [2] Rosenberg, J., Schulzrinne, H., Camarillo, G., Johnston, A., Peterson, J., Sparks, R., Handley, M. and E. Schooler, "SIP: Session Initiation Protocol", RFC 3261, June 2002. [3] Roach, A., “Session Initiation Protocol (SIP)-Specific Event Notification”, RFC 3265, June 2002. [4] Johnston, A., Donovan, S., Sparks, R., Cunningham, C., and K. Summers, "Session Initiation Protocol (SIP) Basic Call Flow Examples", BCP 75, RFC 3665, December 2003. [5] Schulzrinne, H., Oran, D., Camarillo, G., “The Reason Header Field for the Sessions Initiation Protocol (SIP)”, RFC 3326, December 2002. [6] Lingle, K., Mule, J., Maeng, J., Walker, D., “Management Information Base for the Session Initiation Protocol (SIP)”, draft-ietf-sip-mib-10, Work in Progress. 8.2. Informative References Author's Addresses Daryl Malas Level 3 Communications LLC 1025 Eldorado Blvd. Broomfield, CO 80021 USA EMail: daryl.malas@level3.com Intellectual Property Statement The IETF takes no position regarding the validity or scope of any Intellectual Property Rights or other rights that might be claimed to pertain to the implementation or use of the technology described in this document or the extent to which any license under such rights might or might not be available; nor does it represent that it has made any independent effort to identify any such rights. Information on the procedures with respect to rights in RFC documents can be found in BCP 78 and BCP 79. Copies of IPR disclosures made to the IETF Secretariat and any assurances of licenses to be made available, or the result of an attempt made to obtain a general license or permission for the use of such proprietary rights by implementers or users of this Malas Expires December 1, 2006 [Page 12] Internet-Draft SIP Performance Metrics June 2006 specification can be obtained from the IETF on-line IPR repository at http://www.ietf.org/ipr. The IETF invites any interested party to bring to its attention any copyrights, patents or patent applications, or other proprietary rights that may cover technology that may be required to implement this standard. Please address the information to the IETF at ietf-ipr@ietf.org. Disclaimer of Validity This document and the information contained herein are provided on an "AS IS" basis and THE CONTRIBUTOR, THE ORGANIZATION HE/SHE REPRESENTS OR IS SPONSORED BY (IF ANY), THE INTERNET SOCIETY AND THE INTERNET ENGINEERING TASK FORCE DISCLAIM ALL WARRANTIES, EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO ANY WARRANTY THAT THE USE OF THE INFORMATION HEREIN WILL NOT INFRINGE ANY RIGHTS OR ANY IMPLIED WARRANTIES OF MERCHANTABILITY OR FITNESS FOR A PARTICULAR PURPOSE. Copyright Statement Copyright (C) The Internet Society (2006). This document is subject to the rights, licenses and restrictions contained in BCP 78, and except as set forth therein, the authors retain all their rights. Acknowledgment Funding for the RFC Editor function is currently provided by the Internet Society. Malas Expires December 1, 2006 [Page 13]