Network Working Group J. H. Dunn INTERNET-DRAFT C. E. Martin Expires: February, 2001 ANC, Inc. July, 2000 Framework for Router Benchmarking Status of this Memo This document is an Internet-Draft and is in full conformance with all provisions of Section 10 of RFC2026. Internet-Drafts are working documents of the Internet Engineering Task Force (IETF), its areas, and its working groups. Note that other groups may also distribute working documents as Internet-Drafts. Internet-Drafts are draft documents valid for a maximum of six months and may be updated, replaced, or obsoleted by other documents at any time. It is inappropriate to use Internet- Drafts as reference material or to cite them other than as "work in progress." To view the entire list of current Internet-Drafts, please check the "1id-abstracts.txt" listing contained in the Internet-Drafts Shadow Directories on ftp.is.co.za (Africa), ftp.nordu.net (Northern Europe), ftp.nis.garr.it (Southern Europe), munnari.oz.au (Pacific Rim), ftp.ietf.org (US East Coast), or ftp.isi.edu (US West Coast). The list of current Internet-Drafts can be accessed at http://www.ietf.org/ietf/1id-abstracts.txt The list of Internet-Draft Shadow Directories can be accessed at http://www.ietf.org/shadow.html. Copyright Notice Copyright (C) The Internet Society (1999). All Rights Reserved. Abstract This memo discusses and proposes a framework for the development of IP performance benchmarking methodologies in the case of systems under test (SUT) running IETF routing protocols. The intent of this document is to facilitate the use of existing metrics developed by the BMWG and other working groups. This will be accomplished by specifying router configuration and state parameters and characterizing their effect on IP packet forwarding in terms of these existing metrics. The terms defined in this memo will be used in addition to terms defined in RFCs 1242, 2285, and 2544 and 2761. This memo is a product of the Benchmarking Methodology Working Group (BMWG) of the Internet Engineering Task Force (IETF). Dunn & Martin [Page 1] INTERNET-DRAFT Framework for Router Benchmarking July 2000 I. Background. 1. Introduction. The purpose of this document is to define a general framework for particular methodologies to be developed by the Benchmarking Methodology Working Group (BMWG) of the Operational Requirements Area. This memo extends existing work, specifically RFC 2330 "Framework for IP Performance Metrics", May 1998. The goal of this effort is to produce a set of metrics and methodologies, based on existing metrics, to characterize the effects of a router's configuration and state on IP packet forwarding performance as defined in RFC 2544 "Benchmarking Methodology for Network Interconnect Devices". Metrics will be defined in accordance with the framework specified in RFC 1242 "Benchmarking Terminology for Network Interconnect Devices". Methodologies will be defined in accordance with the framework specified in RFC 2544. This effort will focus on the effects of the following configuration and states parameter sets on IP packet forwarding performance: 1. Static configuration parameters, e.g., route cache vs. total available memory size 2. Dynamic configuration parameters, e.g., BGP4 MinRouteAdvertisementInterval 3. Static states, e.g., response to BGP4 NLRI updates 4. Dynamics states, e.g., response to conflicting BGP4 NLRI updates (route flapping) Metrics will be defined which characterize both the impact on IP packet forwarding performance and router response to route updates based on these states. Packet forwarding performance assessment will be based on metrics described in RFCs 1242, 2285 and 2761. The assessment of router response will be based on the values of MIB objects described in RFC 1850 and the upcoming BGP MIB document. This document will reference Internet vocabulary, clearly describing Internet components such as routers, routing protocols, and router MIB element definitions. Any additional router related vocabulary necessary to develop router metrics will be defined in this document. Measurement uncertainties and errors will be described, including how they relate to the analytical framework shared by many aspects of the Internet engineering discipline. Dunn & Martin [Page 2] INTERNET-DRAFT Framework for Router Benchmarking July 2000 2. Existing Definitions. RFC 1242 "Benchmarking Terminology for Network Interconnect Devices" and RFC 2330 "Framework for IP Performance Metrics" should be consulted before attempting to make use of this document. RFC 2544 "Benchmarking Methodology for Network Interconnect Devices" contains discussions of a number of terms relevant to the benchmarking of switching devices and should be consulted. RFC 2285 "Benchmarking Terminology for LAN Switching Devices" contains a number of terms pertaining to traffic distributions and datagram interarrival. RFC 2761 "Terminology for ATM Benchmarking" contains a number terms pertaining to traffic management [TM4.0, TM4.1]. RFC 1812 "Requirements for IP Version 4 Routers" contains an excellent glossary with which it is assumed that the reader is familiar. For the sake of clarity and continuity, this RFC adopts the template for definitions set out in Section 2 of RFC 1242. 3. Requirements In this document, the words that are used to define the significance of each particular requirement are capitalized. These words are: * "MUST" This word, or the words "REQUIRED" and "SHALL" mean that the item is an absolute requirement of the specification. * "SHOULD" This word or the adjective "RECOMMENDED" means that there may exist valid reasons in particular circumstances to ignore this item, but the full implications should be understood and the case carefully weighed before choosing a different course. * "MAY" This word or the adjective "OPTIONAL" means that this item is truly optional. One vendor may choose to include the item because a particular marketplace requires it or because it enhances the product, for example; another vendor may omit the same item. An implementation is not compliant if it fails to satisfy one or more of the MUST requirements for the protocols it implements. An implementation that satisfies all the MUST and all the SHOULD requirements for its protocols is said to be "unconditionally compliant"; one that satisfies all the MUST requirements but not all the SHOULD requirements for its protocols is said to be "conditionally compliant". 4. Criteria for Router Metrics RFC 2330 "Framework for IP Performance Metrics" contains a number of Dunn & Martin [Page 3] INTERNET-DRAFT Framework for Router Benchmarking July 2000 relevant criteria for performance metrics. In addition, router performance metrics should be independent of router architecture and routing protocol. As with IP performance metrics, router metrics will be dependent on the parameter sets mentioned in section I.1 of this document. Since all metrics must produce reproducible and self- consistent results, router metrics are only meaningful in the case where these parameter sets are stable enough for measurements to be performed. As a result, auxiliary observations may be required to determine whether the router is in a stable state. II. Definitions The definitions presented in this section have been divided into two groups. The first group is formal definitions, which are required in the definitions of the performance metrics but are not themselves strictly metrics. These definitions are subsumed from other work done in other working groups both inside and outside the IETF. They are provided as a courtesy to the reader. 1. Formal Definitions 1.1. Definition Format (from RFC 1242) Term to be defined. Definition: The specific definition for the term. Discussion: A brief discussion of the term, its application and any restrictions on measurement procedures. Specification: The working group and document in which the terms are specified and are listed in the references section. 1.2. Generic Definitions. 1.2.1. Forwarding Decision Definition: The process by which a router determines, based on information in the FIB, what the disposition of a packet will be. Discussion: The forwarding decision is made by the router's forwarder and is used by the forwarder to correctly forward the packet. Dunn & Martin [Page 4] INTERNET-DRAFT Framework for Router Benchmarking July 2000 Specification: N/A 1.2.2. Routing Information Entry Definition: A single entry in the FIB that provides all information required for the forwarder to make a forwarding decision. Discussion: A routing information entry may provide forwarding information mapping a single network, a class of networks, a specific data link layer type or other datagram attribute to a unique egress port on the router. Specification: N/A 1.2.2. Router Port Definition: A single point of datagram entry and exit, characterized by a single data link layer connection to other network equipment, access speed and, perhaps, access controls. Discussion: A router port is the primary source of datagrams to the forwarder and destination of datagrams from the forwarder. Specification: N/A 1.3. Static Configuration Parameters. Definition: Static Configuration Parameters are those attributes of a router, which are not altered by information gathered during the operation of a routing protocol. 1.3.1. Forwarding Information Base Cache Definition: A dedicated portion of the FIB, which contains the most current or most often accessed routing information. Discussion: The cache provides a method for rapidly accessing routing information without resorting to a full scale search of the FIB. Specification: N/A 1.3.2. Forwarding Information Base Cache Size Dunn & Martin [Page 5] INTERNET-DRAFT Framework for Router Benchmarking July 2000 Definition: The number of routing information entries allocated to the FIB cache. Discussion: The size of the cache allocation significantly effects router performance. If the cache allocation is not large enough, a full scale search of the FIB is required in order to make forwarding decisions. If the cache allocation is a significant percentage of the FIB size, the cache search differs very little from a full scale FIB search. 1.3.3. Forwarding Information Base Size Definition: The maximum number of routing information entries the FIB can contain. Discussion: This parameter impacts the maximum number of reachable networks supported by the router; however, the relationship between the two parameter is not always linear. Specification: N/A 1.4. Dynamic Configuration Parameters. Definition: Dynamic Configuration Parameters are those attributes of a router, which are altered by information gathered during the operation of a routing protocol. 1.3.2. Number of Reachable Networks Definition: The current number of networks that the routing information in the FIB allows the router to reach. Discussion: The number of reachable networks will depend on static parameters, such as the maximum number of routing entries in the FIB, and dynamic parmeters, such as whether the FIB accurately reflects the state of the network and route aggregation. Specification: N/A 3. Security Considerations. As this document is solely for providing terminology and describes Dunn & Martin [Page 6] INTERNET-DRAFT Framework for Router Benchmarking July 2000 neither a protocol nor an implementation, there are no security considerations associated with this document. 4. Notices Internet Engineering Task Force The IETF takes no position regarding the validity or scope of any intellectual property or other rights that might be claimed to pertain to the implementation or use of the technology described in this document or the extent to which any license under such rights might or might not be available; neither does it represent that it has made any effort to identify any such rights. Information on the IETFs procedures with respect to rights in standards-track and standards-related documentation can be found in BCP-11. Copies of claims of rights made available for publication and any assurances of licenses to be made available, or the result of an attempt made to obtain a general license or permission for the use of such proprietary rights by implementors or users of this specification can be obtained from the IETF Secretariat. The IETF invites any interested party to bring to its attention any copyrights, patents or patent applications, or other proprietary rights, which may cover technology that may be required to practice this standard. Please address the information to the IETF Executive Director. 5. Disclaimer Copyright (C) The Internet Society (1999). All Rights Reserved. This document and translations of it may be copied and furnished to others, and derivative works that comment on or otherwise explain it or assist in its implementation may be prepared, copied, published and distributed, in whole or in part, without restriction of any kind, provided that the above copyright notice and this paragraph are included on all such copies and derivative works. However, this document itself may not be modified in any way, such as by removing the copyright notice or references to the Internet Society or other Internet organizations, except as needed for the purpose of developing Internet standards in which case the procedures for copyrights defined in the Internet Standards process must be followed, or as required to translate it into languages other than English. The limited permissions granted above are Dunn & Martin [Page 7] INTERNET-DRAFT Framework for Router Benchmarking July 2000 perpetual and will not be revoked by the Internet Society or its successors or assigns. This document and the information contained herein is provided on an "AS IS" basis and THE INTERNET SOCIETY AND THE INTERNET ENGINEERING TASK FORCE DISCLAIMS ALL WARRANTIES, EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO ANY WARRANTY THAT THE USE OF THE INFORMATION HEREIN WILL NOT INFRINGE ANY RIGHTS OR ANY IMPLIED WARRANTIES OF MERCHANTABILITY OR FITNESS FOR A PARTICULAR PURPOSE. 6. References [IPv4-Router] Baker, F., "Requirements for IP Version 4 Routers", RFC 1812, June 1995 7. Editors Addresses Jeffrey Dunn Advanced Network Consultants, Inc. 4214 Crest Place, Ellicott City, MD 21043 USA Phone: +1 (410) 750-1700, E-mail: Jeffrey.Dunn@worldnet.att.net Cynthia Martin Advanced Network Consultants, Inc. 4214 Crest Place, Ellicott City, MD 21043 USA Phone: +1 (410) 750-1700, E-mail: Cynthia.E.Martin@worldnet.att.net Dunn & Martin [Page 8]