Network Working Group INTERNET-DRAFT Expires in: July 2004 Scott Poretsky Quarry Technologies Brent Imhoff Wiltel Communications January 2004 Terminology for Benchmarking IGP Data Plane Route Convergence Status of this Memo This document is an Internet-Draft and is in full conformance with all provisions of Section 10 of RFC2026. Internet-Drafts are working documents of the Internet Engineering Task Force (IETF), its areas, and its working groups. Note that other groups may also distribute working documents as Internet- Drafts. Internet-Drafts are draft documents valid for a maximum of six months and may be updated, replaced, or obsoleted by other documents at any time. It is inappropriate to use Internet-Drafts as reference material or to cite them other than as "work in progress." The list of current Internet-Drafts can be accessed at http://www.ietf.org/ietf/1id-abstracts.txt The list of Internet-Draft Shadow Directories can be accessed at http://www.ietf.org/shadow.html. Table of Contents 1. Introduction .................................................2 2. Existing definitions .........................................2 3. Term definitions..............................................3 3.1 Convergence Event.........................................3 3.2 Network Convergence.......................................3 3.3 Route Convergence.........................................4 3.4 Full Convergence..........................................4 3.5 Convergence Packet Loss...................................5 3.6 Convergence Event Instant.................................5 3.7 Convergence Recovery Instant..............................6 3.8 Rate-Derived Convergence Time.............................6 3.9 Convergence Event Transition..............................7 3.10 Convergence Recovery Transition..........................7 3.11 Loss-Derived Convergence Time............................8 3.12 Sustained Forwarding Convergence Time...................................9 Poretsky, Imhoff [Page 1] INTERNET-DRAFT Benchmarking Terminology for January 2004 IGP Data Plane Route Convergence 3.13 Restoration Convergence Time.............................9 3.14 Packet Sampling Interval.................................10 3.15 Local Interface..........................................10 3.16 Neighbor Interface.......................................11 3.17 Remote Interface.........................................11 3.18 Preferred Egress Interface...............................11 3.19 Next-Best Egress Interface...............................12 3.20 Stale Forwarding.........................................12 4. Security Considerations.......................................12 5. References....................................................13 6. Author's Address..............................................13 7. Full Copyright Statement......................................14 1. Introduction This draft describes the terminology for benchmarking IGP Route Convergence. The motivation and applicability for this benchmarking is provided in [1]. The methodology to be used for this benchmarking is described in [2]. The methodology and terminology to be used for benchmarking route convergence can be applied to any link-state IGP such as ISIS [3] and OSPF [4]. The data plane is measured to obtain black-box (externally observable) convergence benchmarking metrics. The purpose of this document is to introduce new terms required to complete execution of the IGP Route Convergence Methodology [2]. An example of Route Convergence as observed and measured from the data plane is shown in Figure 1. The graph in Figure 1 shows Forwarding Rate versus Time. Time 0 on the X-axis is on the far right of the graph. The components of the graph and metrics are defined in the Term Definitions section of this document. Recovery Convergence Event Time = 0sec Maximum ^ ^ ^ Forwarding Rate--> ----\ Packet /--------------- \ Loss /<----Convergence Convergence------->\ / Event Transition Recovery Transition \ / \_____/<------100% Packet Loss X-axis = Time Y-axis = Forwarding Rate Figure 1. Convergence Graph 2. Existing definitions For the sake of clarity and continuity this RFC adopts the template for definitions set out in Section 2 of RFC 1242. Definitions are indexed and grouped together in sections for ease of reference. The key words "MUST", "MUST NOT", "REQUIRED", "SHALL", "SHALL NOT", "SHOULD", "SHOULD NOT", "RECOMMENDED", "MAY", and "OPTIONAL" in this document are to be interpreted as described in RFC 2119. Poretsky, Imhoff [Page 2] INTERNET-DRAFT Benchmarking Terminology for January 2004 IGP Data Plane Route Convergence 3. Term Definitions 3.1 Convergence Event Definition: The occurrence of a planned or unplanned action in the network that results in a change to an entry in the route table. Discussion: Convergence Events include link loss, routing protocol session loss, router failure, and better next-hop. Measurement Units: N/A Issues: None See Also: Convergence Packet Loss Convergence Event Instant 3.2 Network Convergence Definition: The completion of updating of all routing tables, including the FIB, in all routers throughout the network. Discussion: Network Convergence can be approximated to the sum of Route Convergence for all routers in the network. Network Convergence can be determined by recovery of the forwarding rate to equal the offer load, no stale forwarding, and no blenders[5][6]. Measurement Units: N/A Issues: None See Also: Route Convergence Stale Forwarding Poretsky, Imhoff [Page 3] INTERNET-DRAFT Benchmarking Terminology for January 2004 IGP Data Plane Route Convergence 3.3 Route Convergence Definition: Recovery from a Convergence Event indicated by the DUT forwarding rate equal to the offered load. Discussion: Route Convergence is the action of all components of the router being updated with the most recent route change(s) including the RIB and FIB, along with software and hardware tables. Route Convergence can be observed externally by the rerouting of data Traffic to a new egress interface. Measurement Units: N/A Issues: None See Also: Network Convergence Full Convergence Convergence Event 3.4 Full Convergence Definition: Route Convergence for an entire FIB. Discussion: When benchmarking convergence it is useful to measure the time to converge an entire route table. For example, a Convergence Event can be produced for an OSPF table of 5000 routes so that the time to converge routes 1 through 5000 is measured. Measurement Units: N/A Issues: None See Also: Network Convergence Route Convergence Convergence Event Poretsky, Imhoff [Page 4] INTERNET-DRAFT Benchmarking Terminology for January 2004 IGP Data Plane Route Convergence 3.5 Convergence Packet Loss Definition: The amount of packet loss produced by a Convergence Event until Route Convergence occurs. Discussion: Packet loss can be observed as a reduction of forwarded traffic from the maximum forwarding rate. Measurement Units: number of packets Issues: None See Also: Route Convergence Convergence Event Rate-Derived Convergence Time Loss-Derived Convergence Time 3.6 Convergence Event Instant Definition: The time instant that a Convergence Event occurs. Discussion: Convergence Event Instant is observable from the data plane as the precise time that the device under test begins to exhibit packet loss. Measurement Units: hh:mm:ss:uuu Issues: None See Also: Convergence Event Convergence Packet Loss Convergence Recovery Instant Poretsky, Imhoff [Page 5] INTERNET-DRAFT Benchmarking Terminology for January 2004 IGP Data Plane Route Convergence 3.7 Convergence Recovery Instant Definition: The time instant that Full Convergence is measured and maintained for at least an additional five seconds. Discussion: Convergence Recovery Instant is measurable from the data plane as the precise time that the device under test achieves Full Convergence. Measurement Units: hh:mm:ss:uuu Issues: None See Also: Convergence Packet Loss Convergence Event Instant 3.8 Rate-Derived Convergence Time Definition: The amount of time for Convergence Packet Loss to persist upon occurrence of a Convergence Event until occurrence of Route Convergence. Discussion: Rate-Derived Convergence Time can be measured as the time difference from the Convergence Event Instant to the Convergence Recovery Instant, as shown with Equation 1. (eq 1) Rate-Derived Convergence Time = Convergence Recovery Instant - Convergence Event Instant. Rate-Derived Convergence Time should be measured at the maximum forwarding rate. Failure to achieve Full Convergence results in a Rate-Derived Convergence Time benchmark of infinity. Measurement Units: seconds/milliseconds Issues: None See Also: Convergence Packet Loss Convergence Recovery Instant Convergence Event Instant Full Convergence Poretsky, Imhoff [Page 6] INTERNET-DRAFT Benchmarking Terminology for January 2004 IGP Data Plane Route Convergence 3.9 Convergence Event Transition Definition: The characteristic of a router in which forwarding rate gradually reduces to zero after a Convergence Event. Discussion: The Convergence Event Transition is best observed for Full Convergence. Measurement Units: seconds/milliseconds Issues: None See Also: Convergence Event Rate-Derived Convergence Time Convergence Packet Loss Convergence Recovery Transition 3.10 Convergence Recovery Transition Definition: The characteristic of a router in which forwarding rate gradually increases to equal the offered load. Discussion: The Convergence Recovery Transition is best observed for Full Convergence. Measurement Units: seconds/milliseconds Issues: None See Also: Full Convergence Rate-Derived Convergence Time Convergence Packet Loss Convergence Event Transition Poretsky, Imhoff [Page 7] INTERNET-DRAFT Benchmarking Terminology for January 2004 IGP Data Plane Route Convergence 3.11 Loss-Derived Convergence Time Definition: The amount of time it takes for Route Convergence to to be achieved as calculated from the Convergence Packet Loss. Discussion: Loss-Derived Convergence Time can be calculated from Convergence Packet Loss that occurs due to a Convergence Event and Route Convergence, as shown with Equation 2. (eq 2) Loss-Derived Convergence Time = Convergence Packets Loss / Forwarding Rate NOTE: Units for this measurement are packets / packets/second = seconds Measurement Units: seconds/milliseconds Issues: Loss-Derived Convergence time gives a better than actual result when converging many routes simultaneously. Rate-Derived Convergence Time takes the Convergence Recovery Transition into account, but Loss-Derived Convergence Time ignores the Route Convergence Recovery Transition because it is obtained from the measured Convergence Packet Loss. Ideally, the Convergence Event Transition and Convergence Recovery Transition are instantaneous so that the Rate-Derived Convergence Time = Loss-Derived Convergence Time. However, router implementations are less than ideal. For these reasons the preferred reporting benchmark for IGP Route Convergence is the Rate-Derived Convergence Time. Guidelines for reporting Loss-Derived Convergence Time are provided in [2]. See Also: Route Convergence Convergence Packet Loss Rate-Derived Convergence Time Convergence Event Transition Convergence Recovery Transition Poretsky, Imhoff [Page 8] INTERNET-DRAFT Benchmarking Terminology for January 2004 IGP Data Plane Route Convergence 3.12 Sustained Forwarding Convergence Time Definition: The amount of time for Route Convergence to be achieved for cases in which there is no packet loss. Discussion: Sustained Forwarding Convergence Time is the IGP Route Convergence benchmark to be used for Convergence Events that produce a change in next-hop without packet loss. Measurement Units: seconds/milliseconds Issues: None See Also: Route Convergence Rate-Derived Convergence Time Loss-Derived Convergence Time 3.13 Restoration Convergence Time Definition: The amount of time for the router under test to restore traffic to the original outbound port after recovery from a Convergence Event. Discussion: Restoration Convergence Time is the amount of time to Converge back to the original outbound port. This is achieved by recovering from the Convergence Event, such as restoring the failed link. Restoration Convergence Time is measured using the Rate-Derived Convergence Time calculation technique, as provided in Equation 1. It is possible, but not desired to have the Restoration Convergence Time differ from the Rate-Derived Convergence Time. Measurement Units: seconds or milliseconds Issues: None See Also: Convergence Event Rate-Derived Convergence Time Poretsky, Imhoff [Page 9] INTERNET-DRAFT Benchmarking Terminology for January 2004 IGP Data Plane Route Convergence 3.14 Packet Sampling Interval Definition: The rate at which the tester (test equipment) polls to make measurements for arriving packet flows. Discussion: Metrics measured at the Packet Sampling Interval include packets received and Convergence Packet Loss. Measurement Units: seconds or milliseconds Issues: Packet Sampling Interval can influence the Convergence Graph. This is particularly true as implementations achieve Full Convergence in less than 1 second. The Convergence Event Transition and Convergence Recovery Transition can become exaggerated when the Packet Sampling Interval is too long. This will produce a larger than actual Rate-Derived Convergence Time. The recommended value for configuration of the Packet Sampling Interval is provided in [2]. See Also: Convergence Packet Loss Convergence Event Transition Convergence Recovery Transition 3.15 Local Interface Definition: An interface on the DUT. Discussion: None Measurement Units: N/A Issues: None See Also: Neighbor Interface Remote interface Poretsky, Imhoff [Page 10] INTERNET-DRAFT Benchmarking Terminology for January 2004 IGP Data Plane Route Convergence 3.16 Neighbor Interface Definition: The interface on the neighbor router or tester that is directly linked to the DUT's Local Interface. Discussion: None Measurement Units: N/A Issues: None See Also: Local Interface Remote interface 3.17 Remote Interface Definition: An interface on a neighboring router that is not directly connected to any interface on the DUT. Discussion: None Measurement Units: N/A Issues: None See Also: Local interface Neighbor Interface 3.18 Preferred Egress Interface Definition: The outbound interface on DUT to the preferred next-hop. Discussion: Preferred Egress Interface is the egress interface prior to a Convergence Event Measurement Units: N/A Issues: None See Also: Next-Best Egress Interface Poretsky, Imhoff [Page 11] INTERNET-DRAFT Benchmarking Terminology for January 2004 IGP Data Plane Route Convergence 3.19 Next-Best Egress Interface Definition: The outbound interface on DUT to the second-best next-hop. Discussion: Next-Best Egress Interface is the egress interface after a Convergence Event. Measurement Units: N/A Issues: None See Also: Preferred Egress Interface 3.20 Stale Forwarding Definition: Forwarding of traffic to route entries that no longer exist or to route entries with next-hops that are no longer preferred. Discussion: Stale Forwarding can be caused by a Convergence Event and is also known as a "black-hole" since it may produce packet loss. Stale Forwarding exists until Network Convergence is achieved. Measurement Units: N/A Issues: None See Also: Network Convergence 4. Security Considerations Documents of this type do not directly affect the security of Internet or corporate networks as long as benchmarking is not performed on devices or systems connected to operating networks. Poretsky, Imhoff [Page 12] INTERNET-DRAFT Benchmarking Terminology for January 2004 IGP Data Plane Route Convergence 5. References [1] Poretsky, S., "Benchmarking Applicability for IGP Data Plane Route Convergence", draft-ietf-bmwg-igp-dataplane-conv-app-02, work in progress, January 2004. [2] Poretsky, S., "Benchmarking Methodology for IGP Data Plane Route Convergence", draft-ietf-bmwg-igp-dataplane-conv-meth-02, work in progress, January 2004. [3] Callon, R., "Use of OSI IS-IS for Routing in TCP/IP and Dual Environments", RFC 1195, December 1990. [4] Moy, J., "OSPF Version 2", RFC 2328, IETF, April 1998. [5] S. Casner, C. Alaettinoglu, and C. Kuan, "A Fine-Grained View of High Performance Networking", NANOG 22, May 2001. [6] L. Ciavattone, A. Morton, and G. Ramachandran, "Standardized Active Measurements on a Tier 1 IP Backbone", IEEE Communications Magazine, pp90-97, June, 2003. 6. Author's Address Scott Poretsky Quarry Technologies 8 New England Executive Park Burlington, MA 01803 USA Phone: + 1 781 395 5090 EMail: sporetsky@quarrytech.com Brent Imhoff WilTel Communications 3180 Rider Trail South Bridgeton, MO 63045 USA Phone: +1 314 595 6853 EMail: brent.imhoff@wcg.com Poretsky, Imhoff [Page 13] INTERNET-DRAFT Benchmarking Terminology for January 2004 IGP Data Plane Route Convergence 7. Full Copyright Statement Copyright (C) The Internet Society (1998). All Rights Reserved. This document and translations of it may be copied and furnished to others, and derivative works that comment on or otherwise explain it or assist in its implementation may be prepared, copied, published and distributed, in whole or in part, without restriction of any kind, provided that the above copyright notice and this paragraph are included on all such copies and derivative works. However, this document itself may not be modified in any way, such as by removing the copyright notice or references to the Internet Society or other Internet organizations, except as needed for the purpose of developing Internet standards in which case the procedures for copyrights defined in the Internet Standards process must be followed, or as required to translate it into languages other than English. The limited permissions granted above are perpetual and will not be revoked by the Internet Society or its successors or assigns. This document and the information contained herein is provided on an "AS IS" basis and THE INTERNET SOCIETY AND THE INTERNET ENGINEERING TASK FORCE DISCLAIMS ALL WARRANTIES, EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO ANY WARRANTY THAT THE USE OF THE INFORMATION HEREIN WILL NOT INFRINGE ANY RIGHTS OR ANY IMPLIED WARRANTIES OF MERCHANTABILITY OR FITNESS FOR A PARTICULAR PURPOSE. Poretsky, Imhoff [Page 14] Network Working Group INTERNET-DRAFT Expires in: July 2004 Scott Poretsky Quarry Technologies Brent Imhoff Wiltel Communications January 2004 Benchmarking Methodology for IGP Data Plane Route Convergence Status of this Memo This document is an Internet-Draft and is in full conformance with all provisions of Section 10 of RFC2026. Internet-Drafts are working documents of the Internet Engineering Task Force (IETF), its areas, and its working groups. Note that other groups may also distribute working documents as Internet- Drafts. Internet-Drafts are draft documents valid for a maximum of six months and may be updated, replaced, or obsoleted by other documents at any time. It is inappropriate to use Internet-Drafts as reference material or to cite them other than as "work in progress." The list of current Internet-Drafts can be accessed at http://www.ietf.org/ietf/1id-abstracts.txt The list of Internet-Draft Shadow Directories can be accessed at http://www.ietf.org/shadow.html. Table of Contents 1. Introduction ...............................................2 2. Existing definitions .......................................2 3. Test Setup..................................................2 3.1 Test Topologies............................................2 3.2 Test Considerations........................................4 3.2.1 IGP Selection............................................4 3.2.2 BGP Configuration........................................4 3.2.3 IGP Route Scaling........................................5 3.2.4 Timers...................................................5 3.2.5 Convergence Time Metrics.................................5 3.2.6 Offered Load.............................................5 3.2.7 Interface Types..........................................5 3.3 Reporting Format...........................................6 4. Test Cases..................................................6 Poretsky, Imhoff [Page 1] INTERNET-DRAFT Benchmarking Methodology for January 2004 IGP Data Plane Route Convergence 4.1 Convergence Due to Link Failure............................6 4.1.1 Convergence Due to Local Interface Failure...............6 4.1.2 Convergence Due to Neighbor Interface Failure............7 4.1.3 Convergence Due to Remote Interface Failure..............7 4.2 Convergence Due to PPP Session Failure.....................8 4.3 Convergence Due to IGP Adjacency Failure...................9 4.4 Convergence Due to Route Withdrawal........................9 4.5 Convergence Due to Cost Change.............................10 4.6 Convergence Due to ECMP Member Interface Failure...........10 4.7 Convergence Due to Parallel Link Interface Failure.........11 5. Security Considerations.....................................12 6. References..................................................12 7. Author's Address............................................12 8. Full Copyright Statement....................................13 1. Introduction This draft describes the methodology for benchmarking IGP Route Convergence. The applicability of this testing is described in [1] and the new terminology that it introduces is defined in [2]. Service Providers use IGP Convergence time as a key metric of router design and architecture. Customers of Service Providers observe convergence time by packet loss, so IGP Route Convergence is considered a Direct Measure of Quality (DMOQ). The test cases in this document are black-box tests that emulate the network events that cause route convergence, as described in [1]. The black-box test designs benchmark the data plane accounting for all of the factors contributing to convergence time, as discussed in [1]. The methodology (and terminology) for benchmarking route convergence can be applied to any link-state IGP such as ISIS [3] and OSPF [4]. 2. Existing definitions For the sake of clarity and continuity this RFC adopts the template for definitions set out in Section 2 of RFC 1242. Definitions are indexed and grouped together in sections for ease of reference. The key words "MUST", "MUST NOT", "REQUIRED", "SHALL", "SHALL NOT", "SHOULD", "SHOULD NOT", "RECOMMENDED", "MAY", and "OPTIONAL" in this document are to be interpreted as described in RFC 2119. 3. Test Setup 3.1 Test Topologies Figure 1 shows the test topology to measure IGP Route Convergence due to local Convergence Events such as SONET Link Failure, PPP Session Failure, IGP Adjacency Failure, Route Withdrawal, and route cost change. These test cases discussed in section 4 provide route convergence times that account for the Event Detection time, SPF Processing time, and FIB Update time. These times are measured by observing packet loss in the data plane. Poretsky, Imhoff [Page 2] INTERNET-DRAFT Benchmarking Methodology for January 2004 IGP Data Plane Route Convergence --------- Ingress Interface --------- | |<------------------------------| | | | | | | | Preferred Egress Interface | | | DUT |------------------------------>|Tester | | | | | | |~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~>| | | | Next-Best Egress Interface | | --------- --------- Figure 1. IGP Route Convergence Test Topology for Local Changes Figure 2 shows the test topology to measure IGP Route Convergence time due to remote changes in the network topology. These times are measured by observing packet loss in the data plane. In this topology the three routers are considered a System Under Test (SUT). NOTE: All routers in the SUT must be the same model and identically configured. ----- ----------- | | Preferred | | ----- |R2 |---------------------->| | | |-->| | Egress Interface | | | | ----- | | |R1 | | Tester | | | ----- | | | |-->| | Next-Best | | ----- |R3 |~~~~~~~~~~~~~~~~~~~~~~>| | ^ | | Egress Interface | | | ----- ----------- | | |-------------------------------------- Ingress Interface Figure 2. IGP Route Convergence Test Topology for Remote Changes Figure 3 shows the test topology to measure IGP Route Convergence time with members of an ECMP Set. These times are measured by observing packet loss in the data plane. In this topology, the DUT is configured with each Egress interface as a member of an ECMP set and the Tester emulates multiple next-hop routers (emulates one router for each member). Poretsky, Imhoff [Page 3] INTERNET-DRAFT Benchmarking Methodology for January 2004 IGP Data Plane Route Convergence --------- Ingress Interface --------- | |<--------------------------------| | | | | | | | ECMP Set Interface 1 | | | DUT |-------------------------------->| Tester| | | . | | | | . | | | | . | | | |~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~>| | | | ECMP Set Interface N | | --------- --------- Figure 3. IGP Route Convergence Test Topology for ECMP Convergence Figure 4 shows the test topology to measure IGP Route Convergence time with members of a Parallel Link. These times are measured by observing packet loss in the data plane. In this topology, the DUT is configured with each Egress interface as a member of a Parallel Link and the Tester emulates the single next-hop router. --------- Ingress Interface --------- | |<--------------------------------| | | | | | | | Parallel Link Interface 1 | | | DUT |-------------------------------->| Tester| | | . | | | | . | | | | . | | | |~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~>| | | | Parallel Link Interface N | | --------- --------- Figure 4. IGP Route Convergence Test Topology for Parallel Link Convergence 3.2 Test Considerations 3.2.1 IGP Selection The test cases described in section 4 can be used for ISIS or OSPF. The Route Convergence test methodology for both is identical. The IGP adjacencies are established on the Preferred Egress Interface and Next-Best Egress Interface. 3.2.2 BGP Configuration The obtained results for IGP Route Convergence may vary if BGP routes are installed. It is recommended that the IGP Convergence times be benchmarked without BGP routes installed. Poretsky, Imhoff [Page 4] INTERNET-DRAFT Benchmarking Methodology for January 2004 IGP Data Plane Route Convergence 3.2.3 IGP Route Scaling The number of IGP routes will impact the measured IGP Route Convergence because convergence for the entire IGP route table is measured. For results similar to those that would be observed in an operational network it is recommended that the number of installed routes closely approximate that for routers in the network. 3.2.4 Timers There are some timers that will impact the measured IGP Convergence time. The following timers should be configured to the minimum value prior to beginning execution of the test cases: Timer Recommended Value ----- ----------------- SONET Failure Indication Delay <10milliseconds IGP Hello Timer 1 second IGP Dead-Interval 3 seconds LSA Generation Delay 0 LSA Flood Packet Pacing 0 LSA Retransmission Packet Pacing 0 SPF Delay 0 3.2.5 Convergence Time Metrics The recommended value for the Packet Sampling Interval [2] is 100 milliseconds. Rate-Derived Convergence Time [2] is the preferred benchmark for IGP Route Convergence. This benchmark must always be reported when the Packet Sampling Interval [2] <= 100 milliseconds. If the test equipment does not permit the Packet Sampling Interval to be set as low as 100 msec, then both the Rate-Derived Convergence Time and Loss-Derived Convergence Time [2] must be reported. 3.2.6 Offered Load An offered Load of maximum forwarding rate at a fixed packet size is recommended for accurate measurement. The duration of offered load must be greater than the convergence time. 3.2.7 Interface Types All test cases in this methodology document may be executed with any interface type. SONET is recommended and specifically mentioned in the procedures because it can be configured to have no or negligible affect on the measured convergence time. Ethernet (10Mb, 100Mb, 1Gb, and 10Gb) is not preferred since broadcast media are unable to detect loss of host and rely upon IGP Hellos to detect session loss. Poretsky, Imhoff [Page 5] INTERNET-DRAFT Benchmarking Methodology for January 2004 IGP Data Plane Route Convergence 3.3 Reporting Format For each test case, it is recommended that the following reporting format be completed: Parameter Units --------- ----- IGP (ISIS or OSPF) Interface Type (GigE, POS, ATM, etc.) Packet Size bytes IGP Routes number of IGP routes Packet Sampling Interval seconds or milliseconds IGP Timer Values SONET Failure Indication Delay seconds or milliseconds IGP Hello Timer seconds or milliseconds IGP Dead-Interval seconds or milliseconds LSA Generation Delay seconds or milliseconds LSA Flood Packet Pacing seconds or milliseconds LSA Retransmission Packet Pacing seconds or milliseconds SPF Delay seconds or milliseconds Benchmarks Rate-Derived Convergence Time seconds or milliseconds Loss-Derived Convergence Time seconds or milliseconds Restoration Convergence Time seconds or milliseconds 4. Test Cases 4.1 Convergence Due to Link Failure 4.1.1 Convergence Due to Local Interface Failure Objective To obtain the IGP Route Convergence due to a local link failure event at the DUT's Local Interface. Procedure 1. Advertise matching IGP routes from Tester to DUT on Preferred Egress Interface [2] and Next-Best Egress Interface [2] using the topology shown in Figure 1. Set the cost of the routes so that the Preferred Egress Interface is the preferred next-hop. 2. Send traffic at maximum forwarding rate to destinations matching all IGP routes from Tester to DUT on Ingress Interface [2]. 3. Verify traffic routed over Preferred Egress Interface. 4. Remove SONET on DUT's Local Interface [2] by performing an administrative shutdown of the interface. 5. Measure Rate-Derived Convergence Time [2] as DUT detects the link down event and converges all IGP routes and traffic over the Next-Best Egress Interface. 6. Restore SONET on DUT's Local Interface by administratively enabling the interface. 7. Measure Restoration Convergence Time [2] as DUT detects the link up event and converges all IGP routes and traffic back to the Preferred Egress Interface. Poretsky, Imhoff [Page 6] INTERNET-DRAFT Benchmarking Methodology for January 2004 IGP Data Plane Route Convergence Results The measured IGP Convergence time is influenced by the Local SONET indication, SPF delay, SPF Holdtime, SPF Execution Time, Tree Build Time, and Hardware Update Time. 4.1.2 Convergence Due to Neighbor Interface Failure Objective To obtain the IGP Route Convergence due to a local link failure event at the Tester's Neighbor Interface. Procedure 1. Advertise matching IGP routes from Tester to DUT on Preferred Egress Interface [2] and Next-Best Egress Interface [2] using the topology shown in Figure 1. Set the cost of the routes so that the Preferred Egress Interface is the preferred next-hop. 2. Send traffic at maximum forwarding rate to destinations matching all IGP routes from Tester to DUT on Ingress Interface [2]. 3. Verify traffic routed over Preferred Egress Interface. 4. Remove SONET on Tester's Neighbor Interface [2] connected to DUT' s Preferred Egress Interface. 5. Measure Rate-Derived Convergence Time [2] as DUT detects the link down event and converges all IGP routes and traffic over the Next-Best Egress Interface. 6. Restore SONET on Tester's Neighbor Interface connected to DUT's Preferred Egress Interface. 7. Measure Restoration Convergence Time [2] as DUT detects the link up event and converges all IGP routes and traffic back to the Preferred Egress Interface. Results The measured IGP Convergence time is influenced by the Local SONET indication, SPF delay, SPF Holdtime, SPF Execution Time, Tree Build Time, and Hardware Update Time. 4.1.3 Convergence Due to Remote Interface Failure Objective To obtain the IGP Route Convergence due to a Remote Interface failure event. Procedure 1. Advertise matching IGP routes from Tester to SUT on Preferred Egress Interface [2] and Next-Best Egress Interface [2] using the topology shown in Figure 2. Set the cost of the routes so that the Preferred Egress Interface is the preferred next-hop. 2. Send traffic at maximum forwarding rate to destinations matching all IGP routes from Tester to DUT on Ingress Interface [2]. 3. Verify traffic is routed over Preferred Egress Interface. 4. Remove SONET on Tester's Neighbor Interface [2] connected to SUT' s Preferred Egress Interface. Poretsky, Imhoff [Page 7] INTERNET-DRAFT Benchmarking Methodology for January 2004 IGP Data Plane Route Convergence 5. Measure Rate-Derived Convergence Time [2] as SUT detects the link down event and converges all IGP routes and traffic over the Next-Best Egress Interface. 6. Restore SONET on Tester's Neighbor Interface connected to SUT's Preferred Egress Interface. 7. Measure Restoration Convergence Time [2] as SUT detects the link up event and converges all IGP routes and traffic over the Preferred Egress Interface. Results The measured IGP Convergence time is influenced by the SONET failure indication, LSA/LSP Flood Packet Pacing, LSA/LSP Retransmission Packet Pacing, LSA/LSP Generation time, SPF delay, SPF Holdtime, SPF Execution Time, Tree Build Time, and Hardware Update Time. The additional convergence time contributed by LSP Propagation can be obtained by subtracting the Rate-Derived Convergence Time measured in 4.1.2 (Convergence Due to Neighbor Interface Failure) from the Rate-Derived Convergence Time measured in this test case. 4.2 Convergence Due to PPP Session Failure Objective To obtain the IGP Route Convergence due to a Local PPP Session failure event. Procedure 1. Advertise matching IGP routes from Tester to DUT on Preferred Egress Interface [2] and Next-Best Egress Interface [2] using the topology shown in Figure 1. Set the cost of the routes so that the IGP routes along the Preferred Egress Interface is the preferred next-hop. 2. Send traffic at maximum forwarding rate to destinations matching all IGP routes from Tester to DUT on Ingress Interface [2]. 3. Verify traffic routed over Preferred Egress Interface. 4. Remove PPP session from Tester's Neighbor Interface [2] connected to Preferred Egress Interface. 5. Measure Rate-Derived Convergence Time [2] as DUT detects the PPP session down event and converges all IGP routes and traffic over the Next-Best Egress Interface. 6. Restore PPP session on DUT's Preferred Egress Interface. 7. Measure Restoration Convergence Time [2] as DUT detects the session up event and converges all IGP routes and traffic over the Preferred Egress Interface. Results The measured IGP Convergence time is influenced by the PPP failure indication, SPF delay, SPF Holdtime, SPF Execution Time, Tree Build Time, and Hardware Update Time. Poretsky, Imhoff [Page 8] INTERNET-DRAFT Benchmarking Methodology for January 2004 IGP Data Plane Route Convergence 4.3 Convergence Due to IGP Adjacency Failure Objective To obtain the IGP Route Convergence due to a Local IGP Adjacency failure event. Procedure 1. Advertise matching IGP routes from Tester to DUT on Preferred Egress Interface [2] and Next-Best Egress Interface [2] using the topology shown in Figure 1. Set the cost of the routes so that the Preferred Egress Interface is the preferred next-hop. 2. Send traffic at maximum forwarding rate to destinations matching all IGP routes from Tester to DUT on Ingress Interface [2]. 3. Verify traffic routed over Preferred Egress Interface. 4. Remove IGP adjacency from Tester's Neighbor Interface [2] connected to Preferred Egress Interface. 5. Measure Rate-Derived Convergence Time [2] as DUT detects the IGP session failure event and converges all IGP routes and traffic over the Next-Best Egress Interface. 6. Restore IGP session on DUT's Preferred Egress Interface. 7. Measure Restoration Convergence Time [2] as DUT detects the session up event and converges all IGP routes and traffic over the Preferred Egress Interface. Results The measured IGP Convergence time is influenced by the IGP Hello Interval, IGP Dead Interval, SPF delay, SPF Holdtime, SPF Execution Time, Tree Build Time, and Hardware Update Time. 4.4 Convergence Due to Route Withdrawal Objective To obtain the IGP Route Convergence due to Route Withdrawal. Procedure 1. Advertise matching IGP routes from Tester to DUT on Preferred Egress Interface [2] and Next-Best Egress Interface [2] using the topology shown in Figure 1. Set the cost of the routes so that the Preferred Egress Interface is the preferred next-hop. 2. Send traffic at maximum forwarding rate to destinations matching all IGP routes from Tester to DUT on Ingress Interface [2]. 3. Verify traffic routed over Preferred Egress Interface. 4. Tester withdraws all IGP routes from DUT's Local Interface on Preferred Egress Interface. Poretsky, Imhoff [Page 9] INTERNET-DRAFT Benchmarking Methodology for January 2004 IGP Data Plane Route Convergence 6. Re-advertise IGP routes to DUT's Preferred Egress Interface. 7. Measure Restoration Convergence Time [2] as DUT converges all IGP routes and traffic over the Preferred Egress Interface. Results The measured IGP Convergence time is the SPF Processing and FIB Update time as influenced by the SPF delay, SPF Holdtime, SPF Execution Time, Tree Build Time, and Hardware Update Time. 4.5 Convergence Due to Cost Change Objective To obtain the IGP Route Convergence due to route cost change. Procedure 1. Advertise matching IGP routes from Tester to DUT on Preferred Egress Interface [2] and Next-Best Egress Interface [2] using the topology shown in Figure 1. Set the cost of the routes so that the Preferred Egress Interface is the preferred next-hop. 2. Send traffic at maximum forwarding rate to destinations matching all IGP routes from Tester to DUT on Ingress Interface [2]. 3. Verify traffic routed over Preferred Egress Interface. 4. Tester increases cost for all IGP routes at DUT's Preferred Egress Interface so that the Next-Best Egress Interface has lower cost and becomes preferred path. 5. Measure Rate-Derived Convergence Time [2] as DUT detects the cost change event and converges all IGP routes and traffic over the Next-Best Egress Interface. 6. Re-advertise IGP routes to DUT's Preferred Egress Interface with original lower cost metric. 7. Measure Restoration Convergence Time [2] as DUT converges all IGP routes and traffic over the Preferred Egress Interface. Results There should be no measured packet loss for this case. 4.6 Convergence Due to ECMP Member Interface Failure Objective To obtain the IGP Route Convergence due to a local link failure event of an ECMP Member. Procedure 1. Configure ECMP Set as shown in Figure 3. 2. Advertise matching IGP routes from Tester to DUT on each ECMP member. Poretsky, Imhoff [Page 10] INTERNET-DRAFT Benchmarking Methodology for January 2004 IGP Data Plane Route Convergence 3. Send traffic at maximum forwarding rate to destinations matching all IGP routes from Tester to DUT on Ingress Interface [2]. 4. Verify traffic routed over all members of ECMP Set. 5. Remove SONET on Tester's Neighbor Interface [2] connected to one of the DUT's ECMP member interfaces. 6. Measure Rate-Derived Convergence Time [2] as DUT detects the link down event and converges all IGP routes and traffic over the other ECMP members. 7. Restore SONET on Tester's Neighbor Interface connected to DUT's ECMP member interface. 8. Measure Restoration Convergence Time [2] as DUT detects the link up event and converges IGP routes and some distribution of traffic over the restored ECMP member. Results The measured IGP Convergence time is influenced by the Local SONET indication, Tree Build Time, and Hardware Update Time. 4.7 Convergence Due to Parallel Link Interface Failure Objective To obtain the IGP Route Convergence due to a local link failure event for a Member of a Parallel Link. Procedure 1. Configure Parallel Link as shown in Figure 4. 2. Advertise matching IGP routes from Tester to DUT on each Parallel Link member. 3. Send traffic at maximum forwarding rate to destinations matching all IGP routes from Tester to DUT on Ingress Interface [2]. 4. Verify traffic routed over all members of Parallel Link. 5. Remove SONET on Tester's Neighbor Interface [2] connected to one of the DUT's Parallel Link member interfaces. 6. Measure Rate-Derived Convergence Time [2] as DUT detects the link down event and converges all IGP routes and traffic over the other Parallel Link members. 7. Restore SONET on Tester's Neighbor Interface connected to DUT's Parallel Link member interface. 8. Measure Restoration Convergence Time [2] as DUT detects the link up event and converges IGP routes and some distribution of traffic over the restored Parallel Link member. Results The measured IGP Convergence time is influenced by the Local SONET indication, Tree Build Time, and Hardware Update Time. Poretsky, Imhoff [Page 11] INTERNET-DRAFT Benchmarking Methodology for January 2004 IGP Data Plane Route Convergence 5. Security Considerations Documents of this type do not directly affect the security of the Internet or corporate networks as long as benchmarking is not performed on devices or systems connected to operating networks. 6. References [1] Poretsky, S., "Benchmarking Applicability for IGP Convergence", draft-ietf-bmwg-igp-dataplane-conv-app-02, work in progress, January 2004. [2] Poretsky, S., Imhoff, B., "Benchmarking Terminology for IGP Convergence", draft-ietf-bmwg-igp-dataplane-conv-term-02, work in progress, January 2004 [3] Callon, R., "Use of OSI IS-IS for Routing in TCP/IP and Dual Environments", RFC 1195, December 1990. [4] Moy, J., "OSPF Version 2", RFC 2328, IETF, April 1998. 7. Author's Address Scott Poretsky Quarry Technologies 8 New England Executive Park Burlington, MA 01803 USA Phone: + 1 781 395 5090 EMail: sporetsky@quarrytech.com Brent Imhoff WilTel Communications 3180 Rider Trail South Bridgeton, MO 63045 USA Phone: +1 314 595 6853 EMail: brent.imhoff@wcg.com Poretsky, Imhoff [Page 12] INTERNET-DRAFT Benchmarking Methodology for January 2004 IGP Data Plane Route Convergence 8. Full Copyright Statement Copyright (C) The Internet Society (1998). All Rights Reserved. This document and translations of it may be copied and furnished to others, and derivative works that comment on or otherwise explain it or assist in its implementation may be prepared, copied, published and distributed, in whole or in part, without restriction of any kind, provided that the above copyright notice and this paragraph are included on all such copies and derivative works. However, this document itself may not be modified in any way, such as by removing the copyright notice or references to the Internet Society or other Internet organizations, except as needed for the purpose of developing Internet standards in which case the procedures for copyrights defined in the Internet Standards process must be followed, or as required to translate it into languages other than English. The limited permissions granted above are perpetual and will not be revoked by the Internet Society or its successors or assigns. This document and the information contained herein is provided on an "AS IS" basis and THE INTERNET SOCIETY AND THE INTERNET ENGINEERING TASK FORCE DISCLAIMS ALL WARRANTIES, EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO ANY WARRANTY THAT THE USE OF THE INFORMATION HEREIN WILL NOT INFRINGE ANY RIGHTS OR ANY IMPLIED WARRANTIES OF MERCHANTABILITY OR FITNESS FOR A PARTICULAR PURPOSE. Poretsky, Imhoff [Page 13] Network Working Group INTERNET-DRAFT Expires in: July 2004 Scott Poretsky Quarry Technologies January 2004 Benchmarking Applicability for IGP Data Plane Route Convergence Status of this Memo This document is an Internet-Draft and is in full conformance with all provisions of Section 10 of RFC2026. Internet-Drafts are working documents of the Internet Engineering Task Force (IETF), its areas, and its working groups. Note that other groups may also distribute working documents as Internet- Drafts. Internet-Drafts are draft documents valid for a maximum of six months and may be updated, replaced, or obsoleted by other documents at any time. It is inappropriate to use Internet-Drafts as reference material or to cite them other than as "work in progress." The list of current Internet-Drafts can be accessed at http://www.ietf.org/ietf/1id-abstracts.txt The list of Internet-Draft Shadow Directories can be accessed at http://www.ietf.org/shadow.html. ABSTRACT This draft describes the applicability of IGP Route Convergence benchmarking methodology [1] and IGP Route Convergence benchmarking terminology [2]. The methodology and terminology is to be used for benchmarking route convergence and can be applied to any link-state IGP such as ISIS [3] and OSPF [4]. The data plane is measured to obtain the convergence benchmarking metrics described in [1]. Table of Contents 1. Introduction ...............................................2 2. Existing definitions .......................................2 3. Factors for IGP Route Convergence Time......................2 4. Network Events that Cause Route Convergence.................3 5. Use of Data Traffic for IGP Route Convergence Benchmarking..3 6. Security Considerations.....................................4 7. Acknowledgements............................................4 8. References..................................................4 Poretsky [Page 1] INTERNET-DRAFT Benchmarking Applicability for January 2004 IGP Data Plane Route Convergence 9. Author's Address............................................5 10. Full Copyright Statement...................................5 1. Introduction IGP Convergence is a critical performance parameter. Customers of Service Providers use packet loss due to IGP Convergence as a key metric of their network service quality. Service Providers use IGP Convergence time as a key metric of router design and architecture. Fast network convergence can be optimally achieved through deployment of fast converging routers. The fundamental basis by which network users and operators benchmark convergence is packet loss, which is an externally observable event having direct impact on their application performance. IGP Route Convergence is a Direct Measure of Quality (DMOQ) when benchmarking the data plane. For this reason it is important to develop a standard router benchmarking methodology and terminology for measuring IGP convergence that uses the data plane as described in [1] and [2]. This document describes all of the factors that influence a convergence measurement and how a purely black box test can be designed to account for all of these factors. This enables accurate benchmarking and evaluation for route convergence time. 2. Existing definitions For the sake of clarity and continuity this RFC adopts the template for definitions set out in Section 2 of RFC 1242. Definitions are indexed and grouped together in sections for ease of reference. The key words "MUST", "MUST NOT", "REQUIRED", "SHALL", "SHALL NOT", "SHOULD", "SHOULD NOT", "RECOMMENDED", "MAY", and "OPTIONAL" in this document are to be interpreted as described in RFC 2119. 3. Factors for IGP Route Convergence Time There are four major categories of factors contributing to the measured Router IGP Convergence Time. As discussed in [5], [6], [7], [8] and [9], these categories are Event Detection, SPF Processing, IGP Advertisement, and FIB Update. These have numerous components that influence the convergence time. These are listed as follow: -Event Detection- SONET failure indication time PPP failure indication time IGP Hello Dead Interval -SPF Processing- SPF Delay Time SPF Hold time SPF Execution time Poretsky [Page 2] INTERNET-DRAFT Benchmarking Applicability for January 2004 IGP Data Plane Route Convergence -IGP Advertisement- LSA/LSP Flood Packet Pacing LSA/LSP Retransmission Packet Pacing LSA/LSP Generation time -FIB Update- Tree Build time Hardware Update time The contribution of each of these factors listed above will vary with each router vendors' architecture and IGP implementation. It is therefore necessary to design a convergence test that considers all of these components, not just one or a few of these components. The additional benefit of designing a test for all components is that it enables black-box testing in which knowledge of the routers' internal implementations is not required. It is then possible to make valid use of the convergence benchmarking metrics when comparing routers from different vendors. 4. Network Events that Cause Convergence There are different types of network events that can cause IGP convergence. These network events are administrative link removal, unplanned link failure, line card failure, and route changes such as withdrawal, flap, next-hop change, and cost change. When benchmarking a router it is important to measure the convergence time for local and remote occurrence of these network events. The convergence time measured will vary whether the network event occurred locally or remotely due to varying combinations of factors listed in the previous sections. This behavior makes it possible to design purely black-box tests that isolate measurements for each of the components of convergence time. 5. Use of Data Plane for IGP Route Convergence Benchmarking Customers of service providers use packet loss as the metric to calculate convergence time. Packet loss is an externally observable event having direct impact on customers' application performance. For this reason it is important to develop a standard router benchmarking methodology and terminology that is a Direct Measure of Quality (DMOQ)for measuring IGP convergence. Such a methodology uses the data plane as described in [1] and [2]. An additional benefit of using packet loss for calculation of IGP Route Convergence time is that it enables black-box tests to be designed. Data traffic can be offered to the device under test (DUT), an emulated network event can be forced to occur, and packet loss can be externally measured to calculate the convergence time. Knowledge of the DUT architecture and IGP implementation is not required. There is no need to rely on the DUT to produce the test results. There is no need to build intrusive test harnesses for the DUT. Poretsky [Page 3] INTERNET-DRAFT Benchmarking Applicability for January 2004 IGP Data Plane Route Convergence Use of data traffic and measurement of packet loss on the data plane also enables Route Convergence methodology test cases that consider the time for the Route Controller to update the FIB on the forwarding engine of the hardware. A router is not fully converged until all components are updated and traffic is rerouted to the correct egress interface. As long as there is packet loss, routes have not converged. It is possible to send diverse traffic flows to destinations matching every route in the FIB so that the time it takes for the router to converge an entire route table can be benchmarked. 6. Security Considerations Documents of this type do not directly effect the security of the Internet or of corporate networks as long as benchmarking is not performed on devices or systems connected to operating networks. 7. Acknowledgements Thanks to Curtis Villamizar for sharing so much of his knowledge and experience through the years. Also, special thanks to the many Network Engineers and Network Architects at the Service Providers who are always eager to discuss Route Convergence. 8. References [1] Poretsky, S., "Benchmarking Methodology for IGP Data Plane Route Convergence", draft-ietf-bmwg-igp-dataplane-conv-meth-01, work in progress, October 2004. [2] Poretsky, S., "Benchmarking Terminology for IGP Data Plane Route Convergence", draft-ietf-bmwg-igp-dataplane-conv-term-01, work in progress, October 2004. [3] Callon, R., "Use of OSI IS-IS for Routing in TCP/IP and Dual Environments", RFC 1195, December 1990. [4] Moy, J., "OSPF Version 2", RFC 2328, IETF, April 1998. [5] Villamizar, C., "Convergence and Restoration Techniques for ISP Interior Routing", NANOG 25, October 2002. [6] Katz, D., "Why are we Scared of SPF? IGP Scaling and Stability", NANOG 25, October 2002. [7] Filsfils, C., "Deploying Tight-SLA Services on an Internet Backbone: ISIS Fast Convergence and Differentiated Services Design (tutorial)", NANOG 25, October 2002. Poretsky [Page 4] INTERNET-DRAFT Benchmarking Applicability for January 2004 IGP Data Plane Route Convergence [8] Alaettinoglu, C. and Casner, S., "ISIS Routing on the Qwest Backbone: a Recipe for Subsecond ISIS Convergence", NANOG 24, October 2002. [9] Alaettinoglu, C., Jacobson, V., and Yu, H., "Towards Millisecond IGP Convergence", NANOG 20, October 2000. 9. Author's Address Scott Poretsky Quarry Technologies 8 New England Executive Park Burlington, MA 01803 USA Phone: + 1 781 395 5090 EMail: sporetsky@quarrytech.com 10. Full Copyright Statement Copyright (C) The Internet Society (1998). All Rights Reserved. This document and translations of it may be copied and furnished to others, and derivative works that comment on or otherwise explain it or assist in its implementation may be prepared, copied, published and distributed, in whole or in part, without restriction of any kind, provided that the above copyright notice and this paragraph are included on all such copies and derivative works. However, this document itself may not be modified in any way, such as by removing the copyright notice or references to the Internet Society or other Internet organizations, except as needed for the purpose of developing Internet standards in which case the procedures for copyrights defined in the Internet Standards process must be followed, or as required to translate it into languages other than English. The limited permissions granted above are perpetual and will not be revoked by the Internet Society or its successors or assigns. This document and the information contained herein is provided on an "AS IS" basis and THE INTERNET SOCIETY AND THE INTERNET ENGINEERING TASK FORCE DISCLAIMS ALL WARRANTIES, EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO ANY WARRANTY THAT THE USE OF THE INFORMATION HEREIN WILL NOT INFRINGE ANY RIGHTS OR ANY IMPLIED WARRANTIES OF MERCHANTABILITY OR FITNESS FOR A PARTICULAR PURPOSE. Poretsky [Page 5]