Network Working Group K. Dubray INTERNET-DRAFT Bay Networks Expiration Date: September 1997 March 1997 Terminology for IP Multicast Benchmarking Status of this Memo This document is an Internet-Draft. Internet-Drafts are working documents of the Internet Engineering Task Force (IETF), its areas, and its working groups. Note that other groups may also distribute working documents as Internet-Drafts. Internet-Drafts are draft documents valid for a maximum of six months and may be updated, replaced, or obsoleted by other documents at any time. It is inappropriate to use Internet- Drafts as reference material or to cite them other than as ``work in progress.'' To learn the current status of any Internet-Draft, please check the ``1id-abstracts.txt'' listing contained in the Internet- Drafts Shadow Directories on ftp.is.co.za (Africa), nic.nordu.net (Europe), munnari.oz.au (Pacific Rim), ds.internic.net (US East Coast), or ftp.isi.edu (US West Coast). Abstract The purpose of this draft is to add terminology specific to the benchmarking of multicast IP forwarding devices. It builds upon the tenets set forth in RFC 1242, RFC 1944, and other IETF Benchmarking Methodology Working Group (BMWG) effort and extends them to the multicast paradigm. 1. Introduction Network forwarding devices are being required to take a single frame and support delivery to a number of destinations having membership to a particular group. As such, multicast support may place a different burden on the resources of these network forwarding devices than with unicast or broadcast traffic types. By clearly identifying benchmarks and related terminology in this document, it is hoped that detailed methodologies can be generated in subsequent documents. Taken in tandem, these two efforts endeavor to assist the clinical, empirical, and consistent characterization of certain aspects of multicast technologies and their individual implementations. [While primarily directed towards intermediate IP multicast forwarding devices on LANs, elements of this text may or may not be applicable to other media as well.] Dubray, K. Expires September 1997 [Page 1] INTERNET-DRAFT Multicast Benchmarking Terminology March 1997 2. Definition Format This section cites the template suggested by RFC 1242 in the specification of a term to be defined. Term to be defined. Definition: The specific definition for the term. Discussion: A brief discussion of the term, its application and any restrictions on measurement procedures. Measurement units: Units used to record measurements of this term, if applicable. [Issues:] List of issues or conditions that effect this term. This field is optional in this draft. [See Also:] List of other terms that are relevant to the discussion of this term. This field is optional in this draft. 2.1 Existing Terminology This document draws on existing terminology defined in other BMWG work. Examples include, but are not limited to: Throughput (RFC 1242, section 3.17) Latency (RFC 1242, section 3.8) Constant Load (RFC 1242, section 3.4) Frame Loss Rate (RFC 1242, section 3.6) Overhead behavior (RFC 1242, section 3.11) 3. Table of Defined Terms 3.1 General Nomenclature 3.1.1 Device Under Test (DUT). 3.1.2 System Under Test (SUT). 3.1.3 Target Rate. 3.1.4 Offered Rate. 3.1.5 Forwarding Rate. 3.1.6 Maximum Forwarding Rate (MFR). 3.1.7 Flow. 3.1.8 Group Flow. 3.1.9 Service Flow. Dubray, K. Expires September 1997 [Page 2] INTERNET-DRAFT Multicast Benchmarking Terminology March 1997 3.2 Throughput 3.2.1 Mixed Flow Throughput (MFT). 3.2.2 Scaled Group Throughput (SGT). 3.2.3 Aggregated Multicast Throughput (AMT) 3.2.4 Translational Throughput (TT) 3.3 Fairness 3.4 Forwarding Latency 3.4.1 Multicast Latency 3.4.2 Min/Max Multicast Latency 3.5 Overhead 3.5.1 Group Join Delay. 3.5.2 Group Leave Delay. 3.6 Capacity 3.6.1 Multicast Group Capacity. 3.1 General Nomenclature This section will present general terminology to be used in this and other documents. 3.1.1 Device Under Test (DUT). Definition: The network forwarding device being tested. Discussion: Measurement units: Not applicable. Issue: This definition is being moved to the BMWG LAN switch terminology draft. This item will be deleted from this multicast terminology draft upon migration. 3.1.2 System Under Test (SUT). Definition: The collective set of network devices being tested as a singular entity. Dubray, K. Expires September 1997 [Page 3] INTERNET-DRAFT Multicast Benchmarking Terminology March 1997 Discussion: A system under test may be comprised of a variety of networking devices. Some devices may be active in the forwarding decision making process, such as routers or switches; other devices may be passive such as CSU/DSUs. Regardless of constituent components, the system is treated as a "black box" to which stimuli is offered and response measured. Measurement units: Not applicable. Issue: This definition is being moved to the BMWG LAN switch terminology draft. This item will be deleted from this multicast terminology draft upon migration. 3.1.3 Target Rate. Definition: The requested rate at which the test device attempts to offer the DUT or SUT test traffic. Discussion: There are networks events (e.g., collisions) that may preclude the test device from delivering the requested rate to the SUT. In this case, differentiation is made between target rate and offered rate. Measurement units: Frames per second. Issue: This definition is being moved to the BMWG LAN switch terminology draft as "intended rate". This item will be deleted multicast terminology draft upon migration. 3.1.4 Offered Rate. Definition: The actual resultant rate at which the test device is successful in offering test traffic to the SUT. Discussion: Contrast with Target Rate. Note relationship to Forwarding Rate. Measurement units: Frames per second. Dubray, K. Expires September 1997 [Page 4] INTERNET-DRAFT Multicast Benchmarking Terminology March 1997 Issue: This definition is being moved to the BMWG LAN switch terminology draft. This item will be deleted from this multicast terminology draft upon migration. 3.1.5 Forwarding Rate. Definition: The rate at which the SUT has been observed to successfully forward test traffic to the traffic's correct destination(s) in response to a particular offered rate. Discussion: Note the specification of "correct destination(s)" in the definition. The reporting of a forwarding rate MUST correspond to an associated Offered Rate. Frame loss is not a constraint when reporting Forwarding Rate. Measurement units: Frames per second. Issue: This definition is being moved to the BMWG LAN switch terminology draft. This item will be deleted from this multicast terminology draft upon migration. 3.1.6 Maximum Forwarding Rate (MFR). Definition: The rate at which the SUT has been observed to successfully forward test traffic to the traffic's correct destination(s) in response to the test device's maximum offered rate. Discussion: Because a DUT's maximum forwarding rate does not always equal the largest forwarding rate of the DUT, this metric can sometimes indicate oversubscription or congestion internal to the DUT/SUT. For example, consider the following table: Test Device DUT Offered Rate Forwarding Rate ------------- --------------- 1. 14,880 fps 7,400 fps 2. 13,880 fps 8,472 fps 3. 12,880 fps 12,880 fps Dubray, K. Expires September 1997 [Page 5] INTERNET-DRAFT Multicast Benchmarking Terminology March 1997 The tester's maximum offered rate is 14,880 frames per second, as indicated in line 1. Per the definition, the corresponding MFR for the DUT is 7,440 fps - not the 12,880 fps indicated in line 3. When reporting the MFR, the corresponding test device's maximum offered load MUST be cited. This is due to the fact that not all test devices deliver the maximum usable bandwidth. In the case when the test device is able to exceed the maximum, legal bandwidth, the test results SHOULD reflect that the test was conducted in a overload condition. Measurement units: Frames per second. Issue: This definition is being moved to the BMWG LAN switch terminology draft as "Forwarding Rate at Maximum Offered Load." This item will be deleted from this multicast terminology draft upon migration. 3.1.7 Flow. Definition: An equivalence class of packets comprising one or more data streams. Discussion: In the scope of this document, Flow will be considered a logical identifier used to discriminate between a set or sets of packets offered the DUT. For example, one flow may identify a set of unicast packets offered to the DUT. Another flow may differentiate the multicast packets destined to multicast group X. Yet another flow may distinguish the set of multicast packets destined to multicast group Y. Measurement units: Not applicable. Issue: This definition does not seem to be inconsistent with section 1.2 of the RSVP draft, but may not map directly to work being done in the RTFM space. Dubray, K. Expires September 1997 [Page 6] INTERNET-DRAFT Multicast Benchmarking Terminology March 1997 3.1.8 Group Flow. Definition: A specific type of flow where the packets comprising the flow are destined to a particular multicast group. Discussion: Measurement units: Not applicable. 3.1.9 Service Flow. Definition: A specific type of flow where the packets comprising the flow require particular treatment or treatments by the network forwarding devices along the path to the packets' destination(s). Discussion: Measurement units: Not applicable. 3.2 Throughput. This section presents terminology relating to the characterization of the packet forwarding ability of a DUT/SUT in a multicast environment. It extends the concept of throughput presented in RFC 1242. 3.2.1 Mixed Flow Throughput (MFT). Definition: The maximum rate at which none of the offered frames, comprised from a unicast flow and a multicast flow, to be forwarded are dropped by the device. Discussion: Often times, throughput is collected on a homogenous traffic type - though the packets' destinations may vary, the packets follow the same packet forwarding path through the DUT. Based on the RFC 1242 definition for throughput, the Mixed Flow Throughput benchmark attempts to characterize the DUT's ability to process both unicast and multicast frames in the same aggregated traffic stream. Dubray, K. Expires September 1997 [Page 7] INTERNET-DRAFT Multicast Benchmarking Terminology March 1997 Measurement units: Frames per second Issues: Related methodology may have to address the ratio of unicast packets to multicast packets. 3.2.2 Scaled Group Throughput (SGT). Definition: The maximum number of multicast groups that a DUT/SUT can support and still yield the same throughput as supporting a single multicast group. Discussion: A desirable attribute of many Internet mechanisms is the ability to "scale." This benchmark seeks to demonstrate the ability of a SUT to scale the number of multicast groups upwards while holding it to the RFC 1242 definition of throughput for a single multicast group. Measurement units: Number of multicast groups. Issues: The corresponding methodology (or even the definition itself) may have to reflect the impact that the pairing (source, group) has on many multicast routing protocols. 3.2.3 Aggregated Multicast Throughput (AMT) Definition: The maximum rate at which none of the offered frames to be forwarded through N destination interfaces of the same multicast group are dropped. Discussion: Another "scaling" type of exercise, designed to identify the DUT/SUT's ability to handle traffic as a function of the multicast destination ports it is required to support. Measurement units: The ordered pair (N,t) where, N = the number of destination ports of the multicast group. t = the throughput, in frames per second, relative to the source stream. Dubray, K. Expires September 1997 [Page 8] INTERNET-DRAFT Multicast Benchmarking Terminology March 1997 3.2.4 Translational Throughput (TT) Definition: The maximum rate at which none of the frames offered in an transitional format to the SUT are dropped in the process of converting those frames to their appropriate, final format and subsequent correct delivery. Discussion: A popular technique in presenting frames to devices that may not support a protocol feature is to encapsulate, or tunnel, the packet containing the unsupported feature in a format that is supported by that device. This benchmark attempts to characterize the overhead behavior associated with that transitional process. Consideration may need to be given with respect to the impact of different frame formats on usable bandwidth. Measurement units: Frames per second. 3.3 Fairness. Definition: The ability of a SUT to fulfill the requirements of a flow without compromising the requirements, if any, of other flows. Discussion: Measurement units: Not applicable. 3.4 Forwarding Latency. This section presents terminology relating to the characterization of the forwarding latency of a DUT/SUT in a multicast environment. It extends the concept of latency presented in RFC 1242. Dubray, K. Expires September 1997 [Page 9] INTERNET-DRAFT Multicast Benchmarking Terminology March 1997 3.4.1 Multicast Latency. Definition: The set of individual latencies from a single input port on the DUT or SUT to all tested ports belonging to the destination multicast group. Discussion: This benchmark is based on the RFC 1242 definition of latency. While it is useful to collect latency between a pair of source and destination multicast ports, it may be insightful to collect the same type of measurements across a range of ports supporting that group flow. A variety of statistical exercises can be applied to the set of latencies measurements. Measurement units: Time units with enough precision to reflect measurement. 3.4.2 Min/Max Multicast Latency. Definition: The difference between the maximum latency measurement and the minimum latency measurement from the set of latencies produced by the Multicast Latency benchmark. Discussion: This statistic may yield some insight into how a particular implementation handles its multicast traffic. This may be useful to users of multicast synchronization types of applications. Measurement units: Time units with enough precision to reflect measurement. 3.5 Overhead This section presents terminology relating to the characterization of the overhead delays associated with explicit operations found in multicast environments. Dubray, K. Expires September 1997 [Page 10] INTERNET-DRAFT Multicast Benchmarking Terminology March 1997 3.5.1 Group Join Delay. Definition: The time duration it takes a DUT/SUT to start forwarding multicast packets from the time a successful IGMP group membership report has been issued to the DUT/SUT. Discussion: Many different factors can contribute to different results, such as the number or type of multicast-related protocols configured on the system under test. A consideration for the related methodology: possible need to differentiate a specifically-forwarded multicast frame from those sprayed by protocols implementing a flooding tactic to solicit prune feedback. Measurement units: Microseconds. 3.5.2 Group Leave Delay. Definition: The time duration it takes a DUT/SUT to cease forwarding multicast packets after a corresponding IGMP "Leave Group" message has been successfully offered to the DUT/SUT. Discussion: While it is important to understand how quickly a system can process multicast frames; it may be beneficial to understand how quickly that same system can stop the process as well. Measurement units: Microseconds. Issues: Methodology may need to consider protocol-specific timeout values. 3.6 Capacity This section offers terms relating to the identification of multicast group limits of a DUT/SUT. Dubray, K. Expires September 1997 [Page 12] INTERNET-DRAFT Multicast Benchmarking Terminology March 1997 3.6.1 Multicast Group Capacity. Definition: The maximum number of multicast groups a SUT/DUT can support while maintaining the ability to forward multicast frames to all multicast groups registered to that SUT/DUT. Discussion: Measurement units: Multicast groups. Issues: The related methodology may have to consider the impact of multicast sources per group on the ability of a SUT/DUT to "scale up" the number of supportable multicast groups. 4. Security Considerations Security issues are not addressed in this memo. 5. References [1] Bradner, S. Benchmarking Terminology for Network Interconnection Devices. RFC 1242. July, 1991. [2] Bradner, S., McQuaid, J. Benchmarking Methodology for Network Interconnect Devices. RFC 1944. May, 1996. [3] Craig, R. Terminology for Cell/Call Benchmarking. November, 1996. Work in progress. [4] Mandeville, R. Benchmarking Terminology for LAN Switching Devices. February, 1997. Work in progress. 5. Author's Address Kevin Dubray Bay Networks, Inc. 2 Federal Street Billerica, MA 01984 (508) 436-3862 kdubray@baynetworks.com or direct discussion to the Benchmarking Methodology Working Group: bmwg@harvard.edu Dubray, K. Expires September 1997 [Page 13]