Network Working Group INTERNET-DRAFT Expires in: October 2005 Scott Poretsky Quarry Technologies Shankar Rao Qwest Communications February 2005 Terminology for Accelerated Stress Benchmarking Intellectual Property Rights (IPR) statement: By submitting this Internet-Draft, I certify that any applicable patent or other IPR claims of which I am aware have been disclosed, or will be disclosed, and any of which I become aware will be disclosed, in accordance with RFC 3668. Status of this Memo This document is an Internet-Draft and is in full conformance with all provisions of Section 10 of RFC2026. Internet-Drafts are working documents of the Internet Engineering Task Force (IETF), its areas, and its working groups. Note that other groups may also distribute working documents as Internet- Drafts. Internet-Drafts are draft documents valid for a maximum of six months and may be updated, replaced, or obsoleted by other documents at any time. It is inappropriate to use Internet-Drafts as reference material or to cite them other than as "work in progress." The list of current Internet-Drafts can be accessed at http://www.ietf.org/ietf/1id-abstracts.txt The list of Internet-Draft Shadow Directories can be accessed at http://www.ietf.org/shadow.html. ABSTRACT This document provides the Terminology for performing Stress Benchmarking of networking devices. The three phases of the Stress Test: Startup, Instability and Recovery are defined along with the benchmarks and configuration terms associated with the each phase. Also defined are the Benchmark Planes fundamental to stress testing configuration, setup and measurement. The terminology is to be used with the companion framework and methodology documents. Table of Contents 1. Introduction ............................................... 3 2. Existing definitions ....................................... 3 3. Term definitions............................................ 4 3.1 General Terms............................................. 4 Poretsky and Rao [Page 1] INTERNET-DRAFT Terminology for Accelerated Stress Benchmarking February 2005 3.1.1 Benchmark Planes...................................... 4 3.1.2 Configuration Sets.................................... 5 3.1.3 Startup Conditions.................................... 5 3.1.4 Instability Conditions................................ 6 3.1.5 Aggregate Forwarding Rate............................. 6 3.1.6 Controlled Session Loss............................... 7 3.1.7 Uncontrolled Session Loss............................. 7 3.2 Benchmark Planes.......................................... 8 3.2.1 Control Plane......................................... 8 3.2.2 Data Plane............................................ 8 3.2.3 Management Plane...................................... 8 3.2.4 Security Plane........................................ 9 3.3 Startup...................................................10 3.3.1 Startup Phase.........................................10 3.3.2 Benchmarks............................................10 3.3.2.1 Stable Aggregate Forwarding Rate..................10 3.3.2.2 Stable Latency....................................11 3.3.2.3 Stable Session Count..............................11 3.3.3 Control Plane.........................................12 3.3.3.1 Control Plane Configuration Set...................12 3.3.3.2 Control Plane Startup Conditions..................13 3.3.4 Data Plane............................................13 3.3.4.1 Data Plane Configuration Set......................13 3.3.4.2 Traffic Profile...................................13 3.3.5 Management Plane......................................14 3.3.5.1 Management Plane Configuration Set................14 3.3.6 Security Plane........................................15 3.3.6.1 Security Plane Configuration Set..................15 3.3.6.2 Security Plane Startup Conditions.................16 3.4 Instability...............................................16 3.4.1 Instability Phase.....................................16 3.4.2 Benchmarks............................................17 3.4.2.1 Unstable Aggregate Forwarding Rate................17 3.4.2.2 Degraded Aggregate Forwarding Rate................17 3.4.2.3 Average Degraded Aggregate Forwarding Rate........18 3.4.2.4 Unstable Latency..................................18 3.4.2.5 Unstable Uncontrolled Sessions Lost...............19 3.4.3 Instability Conditions................................19 3.4.3.1 Control Plane Instability Conditions..............19 3.4.3.2 Data Plane Instability Conditions.................20 3.4.3.3 Management Plane Instability Conditions...........20 3.4.3.4 Security Plane Instability Conditions.............20 3.5 Recovery..................................................21 3.5.1 Recovery Phase........................................21 3.5.2 Benchmarks............................................21 3.5.2.1 Recovered Aggregate Forwarding Rate...............21 3.5.2.2 Recovered Latency.................................22 3.5.2.3 Recovery Time.....................................22 3.5.2.4 Recovered Uncontrolled Sessions Lost..............23 3.5.2.5 Variability Benchmarks............................23 Poretsky and Rao [Page 2] INTERNET-DRAFT Terminology for Accelerated Stress Benchmarking February 2005 4. Security Considerations.....................................24 5. Normative References........................................24 6. Informative References......................................24 7. Author's Address............................................25 Appendix 1 - White Box Benchmarks..............................25 1. Introduction Routers in an operational network are simultaneously configured with multiple protocols and security policies while forwarding traffic and being managed. To accurately benchmark a router for deployment it is necessary to test that router in operational conditions by simultaneously configuring and scaling network protocols and security policies, forwarding traffic, and managing the device. It is helpful to accelerate these network operational conditions so that the router under test can be benchmarked with faster test duration. Testing a router in accelerated network conditions is known as Accelerated Stress Testing. This document provides the Terminology for performing Stress Benchmarking of networking devices. The three phases of the Stress Test: Startup, Instability and Recovery are defined along with the benchmark and configuration terms associated with the each phase. Benchmarks for stress testing are defined using the Aggregate Forwarding Rate and control plane Session Count during each phase of the test. Also defined are the Benchmark Planes fundamental to stress testing configuration, setup and measurement. These are the Control Plane, Data Plane, Management Plane and Security Plane For each plane, the Configuration Set, Startup Conditions, and Instability Conditions are defined. White Box benchmarks are provided in Appendix 1 for additional DUT behavior measurements. The terminology is to be used with the companion methodology document [4]. The sequence of phases, actions, and benchmarks are shown in Table 1. 2. Existing definitions RFC 1242 "Benchmarking Terminology for Network Interconnect Devices" and RFC 2285 "Benchmarking Terminology for LAN Switching Devices" should be consulted before attempting to make use of this document. For the sake of clarity and continuity this RFC adopts the template for definitions set out in Section 2 of RFC 1242. Definitions are indexed and grouped together in sections for ease of reference. The key words "MUST", "MUST NOT", "REQUIRED", "SHALL", "SHALL NOT", "SHOULD", "SHOULD NOT", "RECOMMENDED", "MAY", and "OPTIONAL" in this document are to be interpreted as described in BCP 14, RFC 2119 [Br97]. RFC 2119 defines the use of these key words to help make the intent of standards track documents as clear as possible. While this document uses these keywords, this document is not a standards track document. Poretsky and Rao [Page 3] INTERNET-DRAFT Terminology for Accelerated Stress Benchmarking February 2005 Table 1. Phase Sequence and Benchmarks III. Recovery Phase II. Instability Phase I. Startup Phase <-----------------<---<-------------------<----<--------------< Remove Instability Achieve Configuration Apply Startup Conditions Set Conditions Benchmark: Benchmark: Benchmark: Recovered Aggregate Unstable Aggregate Stable Aggregate Forwarding Rate Forwarding Rate Forwarding Rate Degraded Aggregate Forwarding Rate Average Degraded Forwarding Rate Recovered Latency Unstable Latency Startup Latency Recovered Uncontrolled Recovered Uncontrolled Stable Session Count Sessions Lost Sessions Lost Recovery Time 3. Term definitions 3.1 General Terms 3.1.1 Benchmark Planes Definition: The features, conditions, and behavior for the Accelerated Stress Benchmarking. Discussion: There are four Benchmark Planes: Control Plane, Data Plane, Management Plane, and Security Plane as shown in Figure 1. The Benchmark Planes define the Configuration, Startup Conditions, Instability Conditions, and Failure Conditions used for the test. Measurement units: N/A Issues: None See Also: Control Plane Data Plane Management Plane Security Plane Poretsky and Rao [Page 4] INTERNET-DRAFT Terminology for Accelerated Stress Benchmarking February 2005 ___________ ___________ | Control | | Management| | Plane |___ ___| Plane | | | | | | | ----------- | | ----------- \/ \/ ___________ ___________ | Security | | |<-----------| Plane | | DUT | | | |--->| |<---| ----------- | ----------- | | | | ___________ | | | Data | | |--->| Plane |<---| | | ----------- Figure 1. Router Accelerated Stress Benchmarking Planes 3.1.2 Configuration Sets Definition: The features and scaling limits used during the Accelerated Stress Benchmarking. Discussion: There are four Configuration Sets: Control Plane Configuration Set, Data Plane Configuration Set, Management Plane Configuration Set, and Security Plane Configuration Set. Measurement units: N/A Issues: None See Also: Control Plane Configuration Set Data Plane Configuration Set Management Plane Configuration Set Security Plane Configuration Set 3.1.3 Startup Conditions Definition: Test conditions that occur at the start of the Accelerated Life Benchmark to establish conditions for the remainder of the test. Poretsky and Rao [Page 5] INTERNET-DRAFT Terminology for Accelerated Stress Benchmarking February 2005 Discussion: Startup Conditions may cause stress on the DUT and produce failure. Startup Conditions are defined for the Control Plane and Security Plane. Measurement units: N/A Issues: None See Also: Control Plane Startup Conditions Data Plane Startup Conditions Management Plane Startup Conditions Security Plane Startup Conditions 3.1.4 Instability Conditions Definition: Test conditions that occur during the Accelerated Stress Benchmark to produce instability and stress the DUT. Discussion: Instability Conditions are applied to the DUT after the Startup Conditions have completed. Instability Conditions occur for the Control Plane, Data Plane, Management Plane, and Security Plane. Measurement units: N/A Issues: None See Also: Control Plane Instability Conditions Data Plane Instability Conditions Management Plane Instability Conditions Security Plane Instability Conditions 3.1.5 Aggregate Forwarding Rate Definition: Sum of forwarding rates for all interfaces on the DUT during the Startup Phase. Discussion: Each interface of the DUT forwards traffic at some measured rate. The Aggregate Forwarding Rate is the sum of forwarding rates for all interfaces on the DUT. Poretsky and Rao [Page 6] INTERNET-DRAFT Terminology for Accelerated Stress Benchmarking February 2005 Measurement units: pps Issues: None See Also: Startup Phase 3.1.6 Controlled Session Loss Definition: Control Plane sessions that are intentionally brought down during the Stress test. Discussion: The test equipment is able to control protocol session state with the DUT. Measurement units: None Issues: None See Also: Uncontrolled Session Loss 3.1.7 Uncontrolled Session Loss Definition: Control Plane sessions that are in the down state but were not intentionally brought down during the Stress test. Discussion: The test equipment is able to control protocol session state with the DUT. The test equipment is also to monitor for sessions lost with the DUT which the test equipment itself did not intentionally bring down. Measurement units: N/A Issues: None See Also: Controlled Session Loss Poretsky and Rao [Page 7] INTERNET-DRAFT Terminology for Accelerated Stress Benchmarking February 2005 3.2 Benchmark Planes 3.2.1 Control Plane Definition: The Description of the control protocols enabled for the Accelerated Stress Benchmarking. Discussion: The Control Plane defines the Configuration, Startup Conditions, and Instability Conditions of the control protocols. Control Plane protocols may include routing protocols, multicast protocols, and MPLS protocols. These can be enabled or disabled for a benchmark test. Measurement units: N/A Issues: None See Also: Benchmark Planes Control Plane Configuration Set Control Plane Startup Conditions Control Plane Instability Conditions 3.2.2 Data Plane Definition: The data traffic profile used for the Accelerated Stress Benchmarking. Discussion: The Data Plane defines the Configuration, Startup Conditions, and Instability Conditions of the data traffic. The Data Plane includes the traffic and interface profile. Measurement Units: N/A See Also: Benchmark Planes Data Plane Configuration Set Data Plane Startup Conditions Data Plane Instability Conditions 3.2.3 Management Plane Definition: The Management features and tools used for the Accelerated Stress Benchmarking. Poretsky and Rao [Page 8] INTERNET-DRAFT Terminology for Accelerated Stress Benchmarking February 2005 Discussion: A key component of the Accelerated Stress Benchmarking is the Management Plane to assess manageability of the router under stress. The Management Plane defines the Configuration, Startup Conditions, and Instability Conditions of the management protocols and features. The Management Plane includes SNMP, Logging/Debug, Statistics Collection, and management configuration sessions such as telnet, SSH, and serial console. SNMP Gets SHOULD be performed continuously. Management configuration sessions should be open simultaneously and be repeatedly open and closed. Open management sessions should have valid and invalid configuration and show commands entered. Measurement units: N/A Issues: None See Also: Benchmark Planes Management Plane Configuration Set Management Plane Startup Conditions Management Plane Instability Conditions 3.2.4 Security Plane Definition: The Security features used during the Accelerated Stress Benchmarking. Discussion: The Security Plane defines the Configuration, Startup Conditions, and Instability Conditions of the security features and protocols. The Security Plane includes the ACLs, Firewall, Secure Protocols, and User Login. Tunnels for those such as IPsec should be established and flapped. Policies for Firewalls and ACLs should be repeatedly added and removed from the configuration via telnet, SSH, or serial management sessions. Measurement units: N/A Issues: None See Also: Benchmark Planes Security Plane Configuration Set Security Plane Startup Conditions Security Plane Instability Conditions Poretsky and Rao [Page 9] INTERNET-DRAFT Terminology for Accelerated Stress Benchmarking February 2005 3.3 Startup 3.3.1 Startup Phase Definition The portion of the benchmarking test in which the Startup Conditions are generated with the DUT. This begins with the attempt to establish the first session and ends when the last Control Plane session is established. Discussion: The Startup Phase is the first Phase of the benchmarking test preceding the Instability Phase and Recovery Phase. It is specified by the Configuration Sets and Startup Conditions for each Benchmark Plane. The Startup Phase ends and Instability Phase may begin when the Configuration Sets are achieved with the DUT. Measurement Units: None Issues: The 'last control plane session is established' may not be a sufficient indicator that steady-state is achieved and Instability Conditions can be applied to begin the Instability Phase. See Also: Benchmark Plane Configuration Sets Startup Conditions Instability Phase Recovery Phase 3.3.2 Benchmarks 3.3.2.1 Stable Aggregate Forwarding Rate Definition: Average rate of traffic forwarded by the DUT during the Startup Phase. Discussion: Stable Aggregate Forwarding Rate is the calculated average of the Aggregate Forwarding Rates measured during the Startup Phase. Measurement units: pps Poretsky and Rao [Page 10] INTERNET-DRAFT Terminology for Accelerated Stress Benchmarking February 2005 Issues: The act of the DUT establishing the Startup Conditions could influence the forwarding rate in certain implementations so that this "baseline" for the remainder of the test is lowered. The alternative is to change the definition of Stable Aggregate Forwarding Rate so that it measured during the Startup Phase, but after Startup Conditions are achieved. The disadvantage of this definition would be that it loses measurement of any impact that establishing Startup Conditions would have on forwarding rate. When comparing the Startup Aggregate Forwarding Rate benchmark of two devices it is preferred to know the impact establishing Startup Conditions has on Forwarding Rate. The definition was therefore selected so that Stable Aggregate Forwarding Rate is calculated from measurement samples throughout the entire Startup Phase. See Also: Startup Phase Aggregate Forwarding Rate 3.3.2.2 Stable Latency Definition: Average measured latency of traffic forwarded by the DUT during the Startup Phase. Discussion: Stable Latency is the calculated average Latency during the Startup Phase. Measurement units: seconds Issues: None See Also: Startup Phase Stable Aggregate Forwarding Rate 3.3.2.3 Stable Session Count Definition: Total number of control plane sessions/adjacencies established and maintained by the DUT during the Startup Phase and prior to Instability Conditions being initiated. Discussion: This measurement should be made after the Control Plane Startup Conditions are applied to the DUT. Poretsky and Rao [Page 11] INTERNET-DRAFT Terminology for Accelerated Stress Benchmarking February 2005 Measurement units: sessions Issues: None See Also: Startup Phase 3.3.3 Control Plane 3.3.3.1 Control Plane Configuration Set Definition: The routing protocols and scaling values used for the Accelerated Life Benchmarking. Discussion: Control Plane Configuration Set is shown in Figure 2 and specifies the Routing Protocols, Multicast, and MPLS configuration. Specific protocols can be enabled or disabled for a benchmark test. Measurement units: N/A Issues: None See Also: Data Plane Configuration Set Management Configuration Set Security Configuration Set ____________ ____________ ____________ | Routing | | Multicast | | MPLS | | Protocols |___ | Protocols | __| Protocols | | | | | | | | | ------------ | ------------ | ------------ | | | | | | | \/ | | ___________ | | | | | |------->| DUT |<------| ``| | ----------- Figure 2. Control Plane Configuration Module Poretsky and Rao [Page 12] INTERNET-DRAFT Terminology for Accelerated Stress Benchmarking February 2005 3.3.3.2 Control Plane Startup Conditions Definition: Control Plane conditions that occur at the start of the Accelerated Stress Benchmarking to establish conditions for the remainder of the test. Discussion: Startup Conditions may cause stress on the DUT and produce failure. Startup Conditions for the Control Plane include session establishment rate, number of sessions established and number of routes learned. Measurement units: N/A Issues: None See Also: Startup Conditions Security Plane Startup Conditions Control Plane Configuration Set 3.3.4 Data Plane 3.3.4.1 Data Plane Configuration Set Definition: The data traffic profile enabled for the Accelerated Stress Benchmarking. Discussion: Data Plane Configuration Set includes the Traffic Profile and interfaces used for the Accelerated Stress Benchmarking. Measurement Units: N/A Issues: None See Also: Traffic Profile 3.3.4.2 Traffic Profile Definition The characteristics of the Offered Load to the DUT used for the Accelerated Stress Benchmarking. Discussion The Traffic Profile specifies the number of packet size(s), packet rate per interface, number of flows, and encapsulation used for the offered load to the DUT. Poretsky and Rao [Page 13] INTERNET-DRAFT Terminology for Accelerated Stress Benchmarking February 2005 Measurement Units: Traffic Profile is reported as follows: Parameter Units --------- ------ Packet Size(s) bytes Packet Rate(interface) array of packets per second Number of Flows number Encapsulation(flow) array of encapsulation type Issues: None See Also: Data Plane Configuration Set 3.3.5 Management Plane 3.3.5.1 Management Plane Configuration Set Definition: The router management features enabled for the Accelerated Stress Test. Discussion: A key component of the Accelerated Stress Test is the Management Configuration Set to assess manageability of the router under stress. The Management Configuration Set defines the management configuration of the DUT. Features that are part of the Management Configuration Set include Telnet access, SNMP, Logging/Debug, and Statistics Collection, and services such as FTP, as shown in Figure 3. These features should be enabled throughout the Stress test. SNMP Gets should be made continuously with multiple FTP and Telnet sessions operating simultaneously. FTP sessions should be opened and closed at varying intervals and get and put files while open. Telnet sessions should be opened and closed at varying intervals and enter valid and invalid show and configuration commands while open. Measurement units: N/A Issues: None See Also: Control Plane Configuration Set Data Plane Configuration Set Security Plane Configuration Set Poretsky and Rao [Page 14] INTERNET-DRAFT Terminology for Accelerated Stress Benchmarking February 2005 ____________ ____________ | | | Logging/ | | SNMP | __| Debug | | | | | | ------------ | ------------ | | | | \/ | ___________ | | | | | DUT |<---| | | ----------- | | \/ ___________ | Packet | | Statistics| | Collector | | | ----------- Figure 3. Management Plane Configuration Set 3.3.6 Security Plane 3.3.6.1 Security Plane Configuration Set Definition: Security features and scaling enabled for the Accelerated Stress Test. Discussion: The Security Plane Configuration Set includes the configuration and scaling of ACLs, Firewall, IPsec, and User Access, as shown in Figure 4. Tunnels should be established and policies configured. Instability is introduced by flapping tunnels and configuring and removing policies. ____________ ____________ ____________ | | | Secure | | User | |ACL/Firewall| | Protocol | __| Access | | | | | | | | ------------ ------------ | ------------ | | | | | | | \/ | | ___________ | | | | | |------->| DUT |<--------| | | ----------- Figure 4. Security Configuration Module Poretsky and Rao [Page 15] INTERNET-DRAFT Terminology for Accelerated Stress Benchmarking February 2005 Measurement units: N/A Issues: None See Also: ACL Configuration Set Secure Protocol Configuration Set Password Login Configuration Set 3.3.6.2 Security Plane Startup Conditions Definition: Security Plane conditions that occur at the start of the Accelerated Stress Benchmarking to establish conditions for the remainder of the test. Discussion: Startup Conditions may cause stress on the DUT and produce failure. Startup Conditions for the Security Plane include session establishment rate, number of sessions established and number of policies learned, and number of user access sessions opened. Measurement units: N/A Issues: None See Also: Startup Conditions Data Plane Startup Conditions Management Plane Startup Conditions Security Plane Startup Conditions 3.4 Instability 3.4.1 Instability Phase Definition: The portion of the benchmarking test in which the Instability Conditions are offered to the DUT. Discussion: The Instability Phase is the middle Phase of of the benchmarking test following the Startup Phase and preceding the Recovery Phase. Measurement Units: None Poretsky and Rao [Page 16] INTERNET-DRAFT Terminology for Accelerated Stress Benchmarking February 2005 Issues: None See Also: Instability Conditions Startup Phase Recovery Phase 3.4.2 Benchmarks 3.4.2.1 Unstable Aggregate Forwarding Rate Definition: Rate of traffic forwarded by the DUT during the Instability Phase. Discussion: Unstable Aggregated Forwarding Rate is an instantaneous measurement of the Aggregate Forwarding Rate during the Instability Phase. Measurement units: pps Issues: None See Also: Instability Conditions Aggregate Forwarding Rate 3.4.2.2 Degraded Aggregate Forwarding Rate Definition: The reduction in Aggregate Forwarding Rate during the Instability Phase. Discussion: The Degraded Aggregate Forwarding Rate is calculated for each measurement of the Unstable Aggregate Forwarding Rate. The Degraded Aggregate Forwarding Rate is calculated by subtracting each measurement of the Unstable Aggregate Forwarding Rate from the Stable Aggregate Forwarding Rate, such that Degraded Forwarding Rate = Stable Aggregate Forwarding Rate - Unstable Aggregate Forwarding Rate Ideally, the Degraded Aggregate Forwarding Rate is zero. Measurement Units: pps Poretsky and Rao [Page 17] INTERNET-DRAFT Terminology for Accelerated Stress Benchmarking February 2005 Issues: None See Also: Instability Phase Unstable Aggregate Forwarding Rate 3.4.2.3 Average Degraded Aggregate Forwarding Rate Definition DUT Benchmark that is the calculated average of the obtained Degraded Forwarding Rates. Discussion: Measurement Units: pps Issues: None See Also: Degraded Aggregate Forwarding Rate 3.4.2.4 Unstable Latency Definition: The average increase in measured packet latency during the Instability Phase compared to the Startup Phase. Discussion: Latency SHOULD be measured at a fixed interval during the Instability Phase. Unstable Latency is the difference between Stable Latency and the average Latency measured during the Instability Phase. It is expected that there be an increase in average latency from the Startup Phase to the Instability phase, but it is possible that the difference be zero. The Unstable Latency cannot be a negative number. Measurement units: seconds Issues: None See Also: Instability Phase Stable Latency Poretsky and Rao [Page 18] INTERNET-DRAFT Terminology for Accelerated Stress Benchmarking February 2005 3.4.2.5 Unstable Uncontrolled Sessions Lost Definition: Control Plane sessions that are in the down state but were not intentionally brought down during the Instability Phase. Discussion: The test equipment is able to control protocol session state with the DUT. The test equipment is also to monitor for sessions lost with the DUT which the test equipment itself did not intentionally bring down. Measurement units: sessions Issues: None See Also: Controlled Session Loss Uncontrolled Session Loss 3.4.3 Instability Conditions 3.4.3.1 Control Plane Instability Conditions Definition: Control Plane conditions that occur during the Accelerated Stress Benchmark to produce instability and stress the DUT. Discussion: Control Plane Instability Conditions are experienced by the DUT after the Startup Conditions have completed. Control Plane Instability Conditions experienced by the DUT include session loss, route withdrawal, and route cost changes. Measurement units: N/A Issues: None See Also: Instability Conditions Data Plane Instability Conditions Management Plane Instability Conditions Security Plane Instability Conditions Poretsky and Rao [Page 19] INTERNET-DRAFT Terminology for Accelerated Stress Benchmarking February 2005 3.4.3.2 Data Plane Instability Conditions Definition: Data Plane conditions that occur during the Accelerated Stress Benchmark to produce instability and stress the DUT. Discussion: Data Plane Instability Conditions are experienced by the DUT after the Startup Conditions have completed. Data Plane Instability Conditions experienced by the DUT include interface shutdown, link loss, and overloaded links. Measurement units: N/A Issues: None See Also: Instability Conditions Control Plane Instability Conditions Management Plane Instability Conditions Security Plane Instability Conditions 3.4.3.3 Management Plane Instability Conditions Definition: Management Plane conditions that occur during the Accelerated Life Benchmark to produce instability and stress the DUT. Discussion: Management Plane Instability Conditions are experienced by the DUT after the Startup Conditions have completed. Management Plane Instability Conditions experienced by the DUT include repeated FTP of large files. Measurement units: N/A Issues: None See Also: Instability Conditions Control Plane Instability Conditions Data Plane Instability Conditions Security Plane Instability Conditions 3.4.3.4 Security Plane Instability Conditions Definition: Security Plane conditions that occur during the Accelerated Life Benchmark to produce instability and stress the DUT. Poretsky and Rao [Page 20] INTERNET-DRAFT Terminology for Accelerated Stress Benchmarking February 2005 Discussion: Security Plane Instability Conditions are experienced by the DUT after the Startup Conditions have completed. Security Plane Instability Conditions experienced by the DUT include session loss and uninitiated policy changes. Measurement units: N/A Issues: None See Also: Instability Conditions Control Plane Instability Conditions Data Plane Instability Conditions Management Plane Instability Conditions 3.5 Recovery 3.5.1 Recovery Phase Definition: The portion of the benchmarking test in which the Startup Conditions are generated with the DUT, but the Instability Conditions are no longer offered to the DUT. Discussion: The Recovery Phase is the final Phase of the benchmarking test following the Startup Phase and Instability Phase. Startup Conditions must not be Restarted. Measurement Units: None Issues: None See Also: Startup Conditions Startup Phase Instability Conditions Instability Phase 3.5.2 Benchmarks 3.5.2.1 Recovered Aggregate Forwarding Rate Definition Rate of traffic forwarded by the DUT during the Recovery Phase. Poretsky and Rao [Page 21] INTERNET-DRAFT Terminology for Accelerated Stress Benchmarking February 2005 Discussion: Recovered Aggregate Forwarding Rate is an instantaneous measurement of the Aggregate Forwarding Rate during the Recovery Phase. Ideally, each measurement of the Recovered Aggregate Forwarding Rate equals the Stable Aggregate Forwarding Rate because the Instability Conditions do not exist in both the Startup and Recovery Phases. Measurement Units: pps Issues: None See Also: Aggregate Forwarding Rate Recovery Phase Recovered Aggregate Forwarding Rate Startup Phase Stable Aggregate Forwarding Rate 3.5.2.2 Recovered Latency Definition: The average increase in measured packet latency during the Recovery Phase compared to the Startup Phase. Discussion: Latency SHOULD be measured at a fixed interval during the Recovery Phase. Unstable Latency is the difference between Stable Latency and the average Latency measured during the Recovery Phase. It is expected that there be no increase in average latency from the Startup Phase to the Recovery Phase. The Recovered Latency cannot be a negative number. Measurement units: seconds Issues: None See Also: Recovery Phase Stable Latency 3.5.2.3 Recovery Time Definition The amount of time for the Recovered Aggregate Forwarding Rate to become equal to the Stable Aggregate Forwarding Rate. Poretsky and Rao [Page 22] INTERNET-DRAFT Terminology for Accelerated Stress Benchmarking February 2005 Discussion Recovery Time is measured beginning at the instant the Instability Phase ends until the Recovered Aggregate Forwarding Rate equals the Stable Aggregate Forwarding Rate for a minimum duration of 180 consecutive seconds. Measurement Units: seconds Issues: None See Also: Recovered Aggregate Forwarding Rate Stable Aggregate Forwarding Rate 3.5.2.4 Recovered Uncontrolled Control Plane Sessions Lost Definition: Control Plane sessions that are in the down state but were not intentionally brought down during the Recovery Phase. Discussion: The test equipment is able to control protocol session state with the DUT. The test equipment is also to monitor for sessions lost with the DUT which the test equipment itself did not intentionally bring down. Measurement units: sessions Issues: None See Also: Controlled Session Loss Uncontrolled Session Loss 3.5.2.5 Variability Benchmarks Definition: The difference between the measured Benchmarks of the same DUT over multiple iterations. Discussion: Ideally, the benchmarks measured should be the same for multiple iterations with the same DUT. Configuration Sets Instability conditions SHOULD be held constant for this benchmark. Whether the DUT can exhibit such predictable and repeatable behavior is an important benchmark in itself. Poretsky and Rao [Page 23] INTERNET-DRAFT Terminology for Accelerated Stress Benchmarking February 2005 Measurement units: As applicable to each Benchmark. The results are to be presented in a table format for successive Iterations. Ideally, the differences should be zero. Issues: None See Also: Startup Period Instability Period Recovery Period 4. Security Considerations Documents of this type do not directly effect the security of the Internet or of corporate networks as long as benchmarking is not performed on devices or systems connected to operating networks. 5. Normative References [1] Bradner, S., Editor, "Benchmarking Terminology for Network Interconnection Devices", RFC 1242, October 1991. [2] Mandeville, R., "Benchmarking Terminology for LAN Switching Devices", RFC 2285, June 1998. [3] Bradner, S. and McQuaid, J., "Benchmarking Methodology for Network Interconnect Devices", RFC 2544, March 1999. [4] Poretsky, S. and Rao, S., "Methodology for Accelerated Stress Benchmarking", draft-ietf-bmwg-acc-bench-meth-01, work in progress, February 2005. 6. Informative References [RFC3871] RFC 3871 "Operational Security Requirements for Large Internet Service Provider (ISP) IP Network Infrastructure. G. Jones, Ed.. IETF, September 2004. [NANOG25] "Core Router Evaluation for Higher Availability", Scott Poretsky, NANOG 25, June 8, 2002, Toronto, CA. [IEEECQR] "Router Stress Testing to Validate Readiness for Network Deployment", Scott Poretsky, IEEE CQR 2003. [CONVMETH] Poretsky, S., "Benchmarking Methodology for IGP Data Plane Route Convergence", draft-ietf-bmwg-igp-dataplane-conv-meth-05, work in progress, February 2005. [CONVTERM] Poretsky, S., "Benchmarking Terminology for IGP Data Plane Route Convergence", draft-ietf-bmwg-igp-dataplane-conv-term-05, work in progress, February 2005. Poretsky and Rao [Page 24] INTERNET-DRAFT Terminology for Accelerated Stress Benchmarking February 2005 7. Author's Address Scott Poretsky Quarry Technologies 8 New England Executive Park Burlington, MA 01803 USA Phone: + 1 781 395 5090 EMail: sporetsky@quarrytech.com Shankar Rao Qwest Communications Denver, CO USA Phone: + 1 303 437 6643 Email: shankar.rao@qwest.com Appendix 1. White Box Benchmarking Terminology Minimum Available Memory Definition: Minimum DUT Available Memory during the duration of the Accelerated Stress Test. Discussion: It is necessary to monitor DUT memory to measure this benchmark. Measurement units: bytes Issues: None See Also: Maximum CPU Utilization Maximum CPU Utilization Definition: Maximum DUT CPU utilization during the duration of the Accelerated Stress Test. Discussion: It is necessary to monitor DUT CPU Utilization to measure this benchmark. Measurement units: % Issues: None See Also: Minimum Available Memory Poretsky and Rao [Page 25] INTERNET-DRAFT Terminology for Accelerated Stress Benchmarking February 2005 Intellectual Property Statement The IETF takes no position regarding the validity or scope of any Intel- lectual Property Rights or other rights that might be claimed to pertain to the implementation or use of the technology described in this docu- ment or the extent to which any license under such rights might or might not be available; nor does it represent that it has made any independent effort to identify any such rights. Information on the procedures with respect to rights in RFC documents can be found in BCP 78 and BCP 79. Copies of IPR disclosures made to the IETF Secretariat and any assurances of licenses to be made available, or the result of an attempt made to obtain a general license or permission for the use of such proprietary rights by implementers or users of this specification can be obtained from the IETF on-line IPR repository at http://www.ietf.org/ipr. The IETF invites any interested party to bring to its attention any copyrights, patents or patent applications, or other proprietary rights that may cover technology that may be required to implement this stan- dard. Please address the information to the IETF at ietf-ipr@ietf.org. Disclaimer of Warranty This document and the information contained herein are provided on an "AS IS" basis and THE CONTRIBUTOR, THE ORGANIZATION HE/SHE REPRESENTS OR IS SPONSORED BY (IF ANY), THE INTERNET SOCIETY AND THE INTERNET ENGINEERING TASK FORCE DISCLAIM ALL WARRANTIES, EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO ANY WARRANTY THAT THE USE OF THE INFORMA- TION HEREIN WILL NOT INFRINGE ANY RIGHTS OR ANY IMPLIED WARRANTIES OF MERCHANTABILITY OR FITNESS FOR A PARTICULAR PURPOSE. Copyright Statement Copyright (C) The Internet Society (2005). This document is subject to the rights, licenses and restrictions contained in BCP 78, and except as set forth therein, the authors retain all their rights. Poretsky and Rao [Page 26]