Benchmarking Working Group D. Newman INTERNET-DRAFT Data Communications Expires in January 1998 H. Holzbaur, J. Hurd, and S. Platt National Software Testing Laboratories Benchmarking Terminology for Network Security Devices Status of This Memo This document is an Internet-Draft. Internet-Drafts are working documents of the Internet Engineering Task Force (IETF), its areas, and its working groups. Note that other groups may also distribute working documents as Internet-Drafts. Internet-Drafts are draft documents valid for a maximum of six months and may be updated, replaced, or obsoleted by other documents at any time. It is inappropriate to use Internet- Drafts as reference material or to cite them other than as "work in progress." To view the entire list of current Internet-Drafts, please check the "1id-abstracts.txt" listing contained in the Internet-Drafts Shadow Directories on ftp.is.co.za (Africa), ftp.nordu.net (Europe), munnari.oz.au (Pacific Rim), ds.internic.net (US East Coast), or ftp.isi.edu (US West Coast). This memo provides information for the Internet community. This memo does not specify an Internet standard of any kind. Distribution of this memo is unlimited. 1. Introduction Despite the rapid rise in deployment of network security devices such as firewalls and authentication/encryption products, there is no standard method for evaluating the performance of these devices. The lack of a standard is troubling for two reasons. First, hardware and software implementations vary widely, making it difficult to do direct performance comparisons. Second, a growing number of organizations are deploying these devices on internal networks that operate at relatively high data rates, while many network security devices are optimized for use over relatively low-speed wide-area connections. As a result, users are often unsure whether the products they buy will stand up to the relatively heavy loads found on internal networks. This document defines terms used in measuring the performance of network security devices. It extends the terminology already used for benchmarking routers and switches to network security devices. The primary metrics defined in this document are maximum forwarding rate and maximum number of connections. Depending on the outcome of discussions within the BMWG, we may also attempt to classify devices using various architectural considerations (proxy, packet filter) or offered load levels Newman et al [Page 1] INTERNET-DRAFT Terminology for Network Security Devices July 1997 (high, medium, low) as criteria. Additionally, new metrics may need to be defined to evaluate application-level issues. 2. Existing definitions This document uses the conceptual framework established in RFCs 1242 and 1944 and draft-ietf-bmwg-lanswitch-05.txt, which describes benchmarking of LAN switch performance. In addition to defining basic practices, these documents contain discussions of several terms relevant to benchmarking performance of network security devices. This document uses the definition format described in RFC 1242, Section 2. Readers should consult these documents before making use of this document. 3. Term definitions 3.1 Authentication Definition: The process of verifying that a client user or machine requesting a network resource is who he, she, or it claims to be, and vice versa. Discussion: Trust is a critical concept in network security. Obviously, any network resource (such as a file server or printer) with restricted access MUST require authentication before granting access. Authentication takes many forms, including but not limited to IP addresses; TCP or UDP port numbers; passwords; external token authentication cards; and pattern matching based on human characteristics such as signature, speech, or retina patterns. Authentication MAY work either by client machine (for example, by proving that a given IP source address really is that address, and not a rogue machine spoofing that address) or by user (by proving that the user really is who he or she claims to be). Servers SHOULD also authenticate themselves to clients. Measurement units: Not applicable Issues: See also: forwarding rate (3.9) user (3.25) virtual client (3.26) 3.2 Bidirectional traffic Definition: Newman et al [Page 2] INTERNET-DRAFT Terminology for Network Security Devices July 1997 Packets presented to a DUT/SUT such that the network interfaces of the DUT/SUT both receive and transmit traffic. Discussion: Traffic patterns offered to the DUT/SUT MUST be bidirectional or fully meshed. See forwarding rate (3.9) for a more complete discussion of issues with traffic patterns. Measurement units: Not applicable Issues: truncated binary exponential back-off algorithm See Also: forwarding rate (3.9) fully meshed traffic (3.10) unidirectional traffic (3.24) 3.3 Data source Definition: A station capable of generating traffic to the DUT/SUT. Discussion: One data source MAY emulate multiple users or stations. In addition, one data source MAY offer traffic to multiple network interfaces on the DUT/SUT. However, each virtual client MUST offer traffic to only one interface. Measurement units: Not applicable Issues: See also: user (3.25) virtual client (3.26) 3.4 Demilitarized zone (DMZ) Definition: A network segment or segments located between protected and external networks. DMZ networks are sometimes called perimeter networks. Discussion: As an extra security measure, networks are often designed such that protected and external segments are never directly connected. Instead, security devices (and possibly other public resources such as WWW or FTP servers) often reside in the so-called DMZ network. To connect protected, DMZ, and external networks with one device, the device MUST have at least three network interfaces. Newman et al [Page 3] INTERNET-DRAFT Terminology for Network Security Devices July 1997 Multiple devices MAY constitute the DMZ, in which case the devices connected the protected network with the DMZ and the DMZ with the external network MUST have two network interfaces. Measurement units: Not applicable Issues: Dual-homed Multihomed See also: external network (3.8) perimeter network (3.15) protected network (3.17) 3.5 Device under test (DUT) Definition: The network security device to which traffic is offered and response measured. Discussion: A single station, generally equipped with at least two network interfaces. Measurement units: Not applicable Issues: See also: system under test (SUT) (3.23) 3.6 Dual-homed Definition: A station with at least two network interfaces. Discussion: Dual-homed network security devices connect two segments with different network-layer addresses. Measurement units: Not applicable Issues: See also: multihomed (3.12) 3.7 Dynamic proxy Definition: Newman et al [Page 4] INTERNET-DRAFT Terminology for Network Security Devices July 1997 A proxy service that is set up and torn down in response to a client request, rather than existing on a static basis. Discussion: Proxy services (see section 3.18) typically are configured to "listen" on a given port number for client requests. However, some devices set up a proxy service only when a client requests the service. Measurement units: Not applicable Issues: rule sets See also: proxy (3.18) rule sets (3.20) 3.8 External network Definition: The segment or segments not protected by the network security DUT/SUT. Discussion: Network security devices are deployed between protected and unprotected segments. The external network is not protected by the DUT/SUT. Measurement units: Not applicable Issues: See also: demilitarized zone (DMZ) (3.4) protected network (3.17) 3.9 Forwarding rate Definition: The number of bits per second a DUT/SUT can transmit to the correct destination network interface in response to a specified offered load. Discussion: Network security devices are by definition session-oriented: They will only grant access to a desired resource once authentication occurs and a session has been established. Because application-layer sessions are always involved, unidirectional packet-per-second metrics are not meaningful in the context of testing network security devices. Instead, this Newman et al [Page 5] INTERNET-DRAFT Terminology for Network Security Devices July 1997 definition MUST measure application-layer performance once a session has been established. Forwarding rate refers to the number of bits per second observed on the output side of the network interface under test. Forwarding rate can be measured with different traffic orientations and distributions. When multiple network interfaces are measured, measurements MUST be observed from the interface with the highest forwarding rate. Measurement units: bits per second (bit/s) kilobits per second (kbit/s) Megabits per second (Mbit/s) Issues: truncated binary exponential back-off algorithm unidirectional vs. bidirectional See Also: authentication (3.1) maximum forwarding rate (3.11) offered load (3.13) unidirectional traffic (3.24) 3.10 Fully meshed traffic Definition: Packets forwarded simultaneously among all of a designated number of network interfaces of a DUT/SUT such that each of the interfaces under test will both forward packets to and receive packets from all of the other interfaces. Discussion: Fully meshed traffic is the most thorough method of exercising the transmitting and receiving capabilities of the DUT/SUT. Unlike past definitions for router or switch testing, it should be noted that fully meshed traffic in this context is not necessarily symmetrical. While all of a designated group of network interfaces MUST simultaneously send and receive traffic, the type and amount of traffic offered MAY differ on each interface. For example, a network security device may see more traffic from the protected network bound for the external network than the opposite (although the inverse could be true during an attack on the DUT/SUT). Measurement units: Not applicable Issues: Half duplex Full duplex See also: Newman et al [Page 6] INTERNET-DRAFT Terminology for Network Security Devices July 1997 bidirectional traffic (3.2) unidirectional traffic (3.24) 3.11 Maximum forwarding rate Definition: The highest forwarding rate of a network security device taken from a set of iterative measurements. Discussion: Maximum forwarding rate may degrade before maximum load is offered. Unlike benchmarks for evaluating router and switch performance, this definition MUST involve measurement of application-layer performance rather than network-layer packet-per-second metrics. Measurement units: Megabits per second kbytes per second bytes per second Issues: full duplex vs. half duplex truncated binary exponential back-off algorithm See also: bidirectional traffic (3.2) partially meshed traffic (3.24) unidirectional traffic 3.12 Multihomed Definition: A network security device with more than two network interfaces. Discussion: Multihoming is a way to connect three or more networks-protected, DMZ, and external-with a single network security device. However, this configuration is not mandatory if multiple network security devices are used. For example, one device could secure the connection between an external and DMZ network, while another could secure the connection between a DMZ and protected network; the two stations collectively form the SUT. Because of the differences in traffic patterns between dual-homed and multihomed devices, direct performance comparisons should be avoided. However, it is acceptable to compare results between a dual-homed device and a DUT/SUT in which only two network interfaces are used. Measurement units: Not applicable Newman et al [Page 7] INTERNET-DRAFT Terminology for Network Security Devices July 1997 Issues: truncated binary exponential back-off algorithm See also: bidirectional traffic (3.2) dual-homed (3.6) fully meshed traffic (3.10) 3.13 Offered load Definition: The number of bits per second that an external source can transmit to a DUT/SUT for forwarding to a specified network interface or interfaces. Discussion: The load that an external source actually applies to a DUT/SUT may be lower than the external source attempts to apply because of collisions on the wire. The transmission capabilities of the external source SHOULD be verified without the DUT/SUT by transmitting unidirectional traffic. Measurement units: bits per second kilobits per second (kbit/s) Megabits per second (Mbit/s) Issues: truncated binary exponential back-off algorithm See also: forwarding rate (3.9) maximum forwarding rate (3.11) 3.14 Packet filtering Definition: The process of controlling access by examining packets based on network-layer or transport-layer criteria. Discussion: Packet-filtering devices forward or deny packets based on information in each packet's header. A packet-filtering network security device uses a rule set (see section 3.20) to determine which traffic should be forwarded and which should be blocked. Packet filtering may be used in a dual-homed or multihomed device. Measurement units: Not applicable Issues: See also: dynamic proxy (3.7) Newman et al [Page 8] INTERNET-DRAFT Terminology for Network Security Devices July 1997 proxy (3.18) rule set (3.20) stateful inspection (3.22) 3.15 Perimeter network Definition: A network segment or segments located between protected and external networks. Perimeter networks are often called DMZ networks. Discussion: See the definition of DMZ (which see) for a discussion. Measurement units: Not applicable Issues: Dual-homed Multihomed See also: Demilitarized zone (DMZ) (3.4) external network (3.8) protected network (3.17) 3.16 Policy Definition: A document defining acceptable use of protected, DMZ, and external networks. Discussion: Security policies generally do not spell out specific configurations for network security devices; rather, they set general guidelines for what it and is not acceptable network behavior. The actual mechanism for controlling access is usually the rule set (see section 3.20) implemented in the DUT/SUT. Measurement units: Not applicable Issues: See also: Rule set (3.20) 3.17 Protected network Definition: A network segment or segments to which access is controlled by the DUT/SUT. Newman et al [Page 9] INTERNET-DRAFT Terminology for Network Security Devices July 1997 Discussion: Network security devices are intended to prevent unauthorized access either to or from the protected network. Depending on the configuration specified by the policy and rule set, the DUT/SUT may allow stations on the protected segment to act as clients for servers on either the DMZ or the external network, or both. Protected networks are often called "internal networks." That term is not used here because network security devices increasingly are deployed within an organization, where all segments are by definition internal. Measurement units: Not applicable Issues: See also: Demilitarized zone (DMZ) (3.4) external network (3.8) policy (3.16) rule set (3.20) 3.18 Proxy Definition: The process of requesting sessions with servers on behalf of clients. Discussion: Proxy-based network security devices never involve direct connections between client and server. Instead, two sessions are established: one between the client and the DUT/SUT, and another between the DUT/SUT and server. As with packet-filtering network security devices, proxy-based devices use a rule set (which see) to determine which traffic should be forwarded and which should be blocked. Measurement units: Not applicable Issues: See also: dynamic proxy (3.7) packet filtering (3.14) stateful inspection (3.22) 3.19 Rejected traffic Definition: Packets dropped as a result of the rule set of the DUT/SUT. Newman et al [Page 10] INTERNET-DRAFT Terminology for Network Security Devices July 1997 Discussion: Network security devices typically are configured to drop any traffic not explicitly permitted in the rule set (which see). Dropped packets MUST NOT be included in calculating the forwarding rate or maximum forwarding rate of the DUT/SUT. Measurement units: Not applicable Issues: See also: forwarding rate (3.9) maximum forwarding rate (3.11) policy (3.16) rule set (3.20) 3.20 Rule set Definition: The collection of definitions that determines which packets the DUT/SUT will forward and which it will reject. Discussion: Rule sets control access to and from the network interfaces of the DUT/SUT. By definition, rule sets MUST NOT apply equally to all network interfaces; otherwise there would be no need for the network security device. Therefore, a specific rule set MUST be applied to each network device used in the DUT/SUT. The order of rules within the rule set is critical. Network security devices generally scan rule sets in a "top down" fashion, which is to say that the device compares each packet received with each rule in the rule set until it finds a rule that applies to the packet. Once the device finds an applicable rule, it applies the actions defined in that rule (such as forwarding or rejecting the packet) and ignores all subsequent rules. For purposes of this document, the rule set MUST conclude with a rule denying all access except that which is permitted in the rule set. Measurement units: Not applicable Issues: See also: Demilitarized zone (DMZ) (3.4) external network (3.8) policy (3.17) protected network (3.18) rejected traffic (3.19) 3.21 Session Newman et al [Page 11] INTERNET-DRAFT Terminology for Network Security Devices July 1997 Definition: A logical connection established between two stations using a known protocol. For purposes of this document, a session MUST be conducted over either TCP (RFC 793) or UDP (RFC 768). Discussion: Because of the application-layer focus of many network security devices, sessions are a more useful metric than the packet-based measurements used in benchmarking routers and switches. Although network security device rule sets generally work on a per-packet basis, it is ultimately sessions that a network security device must handle. For example, the number of file transfer protocol (ftp) sessions a DUT/SUT can handle concurrently is a more meaningful measurement in benchmarking performance than the number of ftp "open" packets it can reject. Further, a stateful inspection device (which see) will not forward individual packets if those packets' headers conflict with state information maintained in the device's rule set. For purposes of this document, a session MUST be established using a known protocol. A traffic pattern is not considered a session until it successfully completes the establishment procedures defined by that protocol. Also for purposes of this document, a session constitutes the logical connection between two end-stations and not the intermediate connections that proxy-based network security devices may use. Issues: See also: policy (3.16) proxy (3.18) rule set (3.20) stateful inspection (3.22) 3.22 Stateful inspection Definition: The process of forwarding or rejecting traffic based on the contents of a state table maintained by the network security device. Discussion: Packet filtering and proxy devices are essentially static, in that they always forward or reject traffic based on the contents of the rule set. Devices using stateful inspection, in contrast, will only forward traffic if it corresponds with state information maintained by the device about each session. For example, a stateful inspection device will reject a packet on TCP port 21 (ftp DATA) if no ftp session has been established. Newman et al [Page 12] INTERNET-DRAFT Terminology for Network Security Devices July 1997 Measurement units: Not applicable Issues: See also: dynamic proxy (3.7) packet filter (3.14) proxy (3.18) 3.23 System under test (SUT) Definition: The collective set of network security devices to which traffic is offered as a single entity and response measured. Discussion: A system under test may comprise multiple network security devices. A typical configuration involves two or more devices, with at least one located between the protected network and DMZ and at least one other located between the DMZ and external network. Some devices may be active, such as firewalls or authentication products; other devices, such as systems for logging, may be passive. Measurement units: Not applicable Issues: See also: demilitarized zone (DMZ) (3.4) device under test (3.5) external network (3.8) protected network (3.17) 3.24 Unidirectional traffic Definition: Packets offered to the DUT/SUT such that the sending and receiving network interface or interfaces are mutually exclusive. Discussion: This definition is included mainly for purposes of completeness; it is not particularly meaningful in the context of network security device performance. As noted in the discussion of forwarding rate (see section 3.9), network security devices almost invariably involve sessions with bidirectional traffic flow. However, unidirectional traffic is appropriate for evaluating the maximum forwarding rate of data sources (absent the DUT/SUT), and for evaluating the maximum forwarding rate of certain connectionless protocols. Newman et al [Page 13] INTERNET-DRAFT Terminology for Network Security Devices July 1997 Measurement units: Not applicable Issues: half duplex vs. full duplex See also: bidirectional traffic (3.2) forwarding rate (3.9) maximum forwarding rate (3.11) 3.25 User Definition: The person or machine requesting access to resources protected by the DUT/SUT. Discussion: "User" is a problematic term in the context of security device performance testing, for several reasons. First, a user may in fact be a machine or machines requesting services through the DUT/SUT. Second, different "user" requests may require radically different amounts of DUT/SUT resources. Third, traffic profiles vary widely from one organization to another, making it difficult to characterize the load offered by a typical users. For these reasons, we prefer not to measure DUT/SUT performance in terms of users supported. Instead, we describe performance in terms of maximum forwarding rate and maximum number of sessions sustained. Measurement units: Not applicable Issues: See also: data source (3.3) virtual client (3.26) 3.26 Virtual client Definition: A subset of a data source that represents one individual user. Discussion: In offering traffic to the DUT/SUT it may be useful for one data source to emulate multiple users, machines, or networks. For purposes of this document, each emulated user should be considered a virtual client. One data source MAY offer traffic from multiple virtual clients to multiple network interfaces on the DUT/SUT. However, each virtual client MUST offer traffic to just one network interface. Newman et al [Page 14] INTERNET-DRAFT Terminology for Network Security Devices July 1997 Measurement units: Not applicable Issues: See also: data source (3.3) user (3.25) 4. Security considerations Security considerations are explicitly excluded from this memo. The authors plan to address security and management concerns in a separate proposal brought to the IETF's security directorate. 5. References Bradner, S., editor. "Benchmarking Terminology for Network Interconnection Devices." RFC 1242. Bradner, S., and McQuaid, J. "Benchmarking Methodology for Network Interconnect Devices." RFC 1944. Mandeville, B. "Benchmarking Terminology for LAN Switching Devices." ftp://ietf.org/internet-drafts/draft-ietf-bmwg- lanswitch-05.txt Newman, D., and Melson, B. "Can Firewalls Take the Heat?" Data Communications, November 21, 1995. http://www.data.com/Lab_Tests/Firewalls.html Newman, D., Holzbaur, H., and Bishop, K. "Firewalls: Don't Get Burned," Data Communications, March 21, 1997. http://www.data.com/lab_tests/firewalls97.html Ranum, M. "Firewall Performance Measurement Techniques: A Scientific Approach." http://www.clark.net/pub/mjr/pubs/fwperf/intro.htm Shannon, G. "Profile of Corporate Internet Application Traffic." http://www.milkyway.com/libr/prof.html 6. Acknowledgments The authors wish to thank the IETF Benchmarking Working Group for agreeing to review this document. Ted Doty (Network Systems), Shlomo Kramer (Check Point Software Technologies), Bob Mandeville (European Network Laboratories), Brent Melson (National Software Testing Laboratories), Marcus Ranum (Network Flight Recorder Inc.), Greg Shannon (Milkyway Networks), Rick Siebenaler (Cyberguard), and Greg Smith (Check Point Software Technologies) offered valuable contributions and critiques during this project. 7. Contact Information David Newman Data Communications magazine Newman et al [Page 15] INTERNET-DRAFT Terminology for Network Security Devices July 1997 1221 Avenue of the Americas, 41st Floor New York, NY 10020 USA 212-512-6182 voice 212-512-6833 fax dnewman@data.com Helen Holzbaur National Software Testing Laboratories Inc. 625 Ridge Pike Conshohocken, PA 19428 USA helen@nstl.com Jim Hurd National Software Testing Laboratories Inc. 625 Ridge Pike Conshohocken, PA 19428 USA jimh@nstl.com Steven Platt, PhD. National Software Testing Laboratories Inc. 625 Ridge Pike Conshohocken, PA 19428 USA steve@nstl.com Newman et al [Page 16]