Network Working Group Terry Martin Internet-Draft M2networx INC Expiration Date: B. Hickman Netcom Systems June 2000 Benchmarking Methodology for Firewalls Status of this Memo This document is an Internet-Draft and is in full conformance with all provisions of Section 10 of RFC2026. Internet-Drafts are working documents of the Internet Engineering Task Force (IETF), its areas, and its working groups. Note that other groups may also distribute working documents as Internet- Drafts. Internet-Drafts are draft documents valid for a maximum of six months and may be updated, replaced, or obsoleted by other documents at any time. It is inappropriate to use Internet-Drafts as reference material or to cite them other than as "work in progress." The list of current Internet-Drafts can be accessed at http://www.ietf.org/ietf/1id-abstracts.txt The list of Internet-Draft Shadow Directories can be accessed at http://www.ietf.org/shadow.html. Table of Contents 1. Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . 2 2. Requirements . . . . . . . . . . . . . . . . . . . . . . . . . . . 2 3. Scope . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 2 4. Test setup . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3 4.1 Test Considerations . . . . . . . . . . . . . . . . . . . . . 4 4.2 Virtual Client/Servers . . . . . . . . . . . . . . . . . . . 4 4.3 Test Traffic Requirements . . . . . . . . . . . . . . . . . . 4 4.4 DUT/SUT Traffic Flows . . . . . . . . . . . . . . . . . . . . 5 4.5 Multiple Client/Server Testing . . . . . . . . . . . . . . . 5 4.6 NAT(Network Address Translation) . . . . . . . . . . . . . . 6 4.7 Rule Sets . . . . . . . . . . . . . . . . . . . . . . . . . . 6 4.8 Web Caching . . . . . . . . . . . . . . . . . . . . . . . . . 6 5. Benchmarking Tests . . . . . . . . . . . . . . . . . . . . . . . . 7 5.1 Concurrent Connection Capacity . . . . . . . . . . . . . . . 7 5.2 Maximum Connection Rate . . . . . . . . . . . . . . . . . . . 9 5.3 Denial Of Service Handling . . . . . . . . . . . . . . . . . 10 5.3 Single Application Goodput . . . . . . . . . . . . . . . . . 12 5.4 FTP Goodput . . . . . . . . . . . . . . . . . . . . . . . . . 12 5.5 SMTP Goodput . . . . . . . . . . . . . . . . . . . . . . . . 13 5.6 HTTP Goodput . . . . . . . . . . . . . . . . . . . . . . . . 15 Martin, Hickman [Page 1] INTERNET-DRAFT Benchmarking Methodology for Firewalls June 2000 5.7 Mixed Application Goodput . . . . . . . . . . . . . . . . . . 17 5.8 Illegal Traffic Handling . . . . . . . . . . . . . . . . . . 18 5.9 Latency . . . . . . . . . . . . . . . . . . . . . . . . . . . 19 Appendices . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 19 A. File Transfer Protocol(FTP) . . . . . . . . . . . . . . . . . . . 19 A.1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . 19 A.2 Connection Establishment/Teardown . . . . . . . . . . . . . . 19 A.3 Object Format . . . . . . . . . . . . . . . . . . . . . . . . 20 B. Simple Mail Transfer Protocol(SMTP) . . . . . . . . . . . . . . . 20 B.1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . 20 B.2 Connection Establishment/Teardown . . . . . . . . . . . . . . 20 B.3 Object Format . . . . . . . . . . . . . . . . . . . . . . . . 21 C. HyperText Transfer Protocol(HTTP) . . . . . . . . . . . . . . . . 21 C.1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . 21 C.2 Version Considerations . . . . . . . . . . . . . . . . . . . . 21 C.3 Object Format . . . . . . . . . . . . . . . . . . . . . . . . 21 D. GoodPut Measurements . . . . . . . . . . . . . . . . . . . . . . . 22 E. References . . . . . . . . . . . . . . . . . . . . . . . . . . . . 23 1. Introduction This document is intended to provide methodology for the benchmarking of firewalls. It provides methodologies for benchmarking forwarding performance, connection performance, latency and filtering. In addition to defining the tests, this document also describes specific formats for reporting the results of the tests. A previous document, "Benchmarking Terminology for Firewall Performance" [1], defines many of the terms that are used in this document. The terminology document SHOULD be consulted before attempting to make use of this document. 2. Requirements The key words "MUST", "MUST NOT", "REQUIRED", "SHALL", "SHALL NOT", "SHOULD", "SHOULD NOT", "RECOMMENDED", "MAY", and "OPTIONAL" in this document are to be interpreted as described in RFC 2119. 3. Scope Firewalls can provide a single point of defense between two networks--it protects one network from the other. Usually, a firewall protects the company's private network from the public or shared networks to which it is connected. A firewall can be as simple as a device that filters different packets or as complex as a group of devices providing solutions that offers combined packet filtering and application-level proxy or network translation services. This RFC will focus on developing bench mark testing of systems from an application perspective and will be developed independent of any firewall implementation. These tests will evaluate the ability of firewall devices to control and manage applications services used Martin, Hickman [Page 2] INTERNET-DRAFT Benchmarking Methodology for Firewalls June 2000 by today's businesses such as applications like the World Wide Web, File transfer procedures and e-mail. Even through there are many different control methods of managing application level being implemented, this RFC does not condone or promote any aforementioned process or procedure. It's goal is to present a procedure that will evaluate firewall performance independent of their implementation. 4. Test Setup Test configurations defined in this document will be confined to dual-homed and tri-homed as shown in figure 1 and figure 2 respectively. Firewalls employing Dual-Homed configurations connect two networks. One interface of the firewall is attached to the unprotected network, typically the public network(i.e. - Internet). The other interface is connected to the protected network, typically the internal LAN. In the case of Dual-Homed configurations, servers which are made accessible to the public(Unprotected) network are attached to the private(Protected) network. +----------+ +----------+ | | | +----------+ | | | | Servers/ |----| | | |------| Servers/ | | Clients | | | | | | Clients | | | |-------| DUT/SUT |--------| | | +----------+ | | | | +----------+ | +----------+ | Protected | | Unprotected Network Network Figure 1(Dual-Homed) Tri-homed configurations employ a third segment called a DMZ. With tri-homed configurations, servers accessible to the public network are attached to the DMZ. Tri-Homed configurations offer additional security by separating server accessible to the public network from internal hosts. Martin, Hickman [Page 3] INTERNET-DRAFT Benchmarking Methodology for Firewalls June 2000 +----------+ +----------+ | | | +----------+ | | | | Clients |----| | | |------| Servers/ | | | | | | | | Clients | +----------+ |-------| DUT/SUT |--------| | | | | | | +----------+ | +----------+ | Protected | | | Unprotected Network | Network | | ----------------- | DMZ | | +-----------+ | | | Servers | | | +-----------+ Figure 2(Tri-Homed) 4.1 Test Considerations 4.2 Virtual Clients/Servers Since firewall testing may involve data sources which emulate multiple users or hosts, the methodology uses the terms virtual clients/servers. For these firewall tests, virtual clients/servers specify application layer entities which may not be associated with a unique physical interface. For example, four virtual clients may originate from the same data source. 4.3 Test Traffic Requirements While the function of a firewall is to enforce access control policies, the criteria by which those policies are defined vary depending on the implementation. Firewalls may use network layer and/or, in many cases, application-layer criteria to make access-control decisions. Therefore, the test equipment used to benchmark the DUT/SUT performance MUST consist of real clients and servers generating legitimate layer 7 conversations. Specifically, the tests defined in this document use HTTP, FTP, and SMTP sessions for benchmarking the performance of the DUT/SUT. See appendices for specific information regarding the transactions involved in establishing/tearing down connections as well as object formats for each of the aforementioned protocols. Martin, Hickman [Page 4] INTERNET-DRAFT Benchmarking Methodology for Firewalls June 2000 4.4 DUT/SUT Traffic Flows This section discusses the client -> server traffic flow requirements associated with the source -> destination interfaces of the firewall. Since the number of interfaces are not fixed, the traffic flows will be dependent upon the configuration used in benchmarking the DUT/SUT. Note that the term "traffic flows" is in the context of client requests since traffic between clients and servers are, by nature, bi-directional with transactions occurring between both entities. In the case of Dual-Homed configurations, traffic flows presented to the DUT/SUT SHOULD consist of client -> server requests associated with the following interfaces: Protected -> Unprotected Unprotected -> Protected In the case of Tri-Homed configurations, traffic flows presented to the DUT/SUT SHOULD consist of client -> server requests associated with the following interfaces: Protected -> Unprotected Protected -> DMZ Unprotected -> DMZ 4.5 Multiple Client/Server Testing The methodologies defined in this document MAY be performed with one or more clients targeting multiple servers for a given application. When performing tests with one or more virtual clients targeting multiple servers, each virtual client MUST initiate requests (Connection, file transfers, etc.) in a round-robin fashion. For example, if the test consisted of six virtual clients targeting three servers, the pattern would be as follows: Client Target Server(In order of request) #1 1 2 3 1... #2 2 3 1 2... #3 3 1 2 3... #4 1 2 3 1... #5 2 3 1 2... #6 3 1 2 3... Martin, Hickman [Page 5] INTERNET-DRAFT Benchmarking Methodology for Firewalls June 2000 4.6 NAT(Network Address Translation) Most firewalls come with Network Address Translation(NAT)networks built in which translates internal host IP addresses attached to the protected network to a virtual IP address for communicating across the unprotected network(Internet). Since this involves additional processing on the part of the DUT/SUT, its impact on performance should be measured. Therefore, tests defined in this document SHOULD first be ran with NAT disabled and then repeated with NAT enabled to determine the performance differentials. 4.7 Rule Sets Rule sets are a collection of access control policies that determines which packets the DUT/SUT will forward and which it will reject. The criteria by which these access control policies may be defined will vary depending on the capabilities of the product. Therefore, this document will not attempt to define the specifics of the rule sets, but instead define how the rule sets should be applied when testing the DUT/SUT. The firewall monitors the incoming traffic and checks to make sure that the traffic meets one of the defined rules before allowing it to be forwarded. It is RECOMMENDED that a rule be entered for each host(i.e. - Virtual client). Although many firewalls permit groups of IP addresses to be defined for a given rule, tests SHOULD be performed with as large a rule set as possible to better stress test the DUT/SUT. In addition, a rule MUST be defined which denies access to all traffic which was not previously defined in the rule set, provided such a rule can be defined in the DUT/SUT. 4.8 Web Caching Some firewalls include caching agents in order to reduce network load. When making a request through a caching agent, the caching agent attempts to service the response from its internal resources. The cache itself saves responses it receives, such as responses for HTTP GET requests. Therefore, caching SHOULD be disabled on the DUT/SUT prior to performing the tests. Martin, Hickman [Page 6] INTERNET-DRAFT Benchmarking Methodology for Firewalls June 2000 5. Benchmarking Tests The following tests offer objectives, procedures, and reporting formats for benchmarking firewalls. 5.1 Concurrent Connection Capacity 5.1.1 Objective To determine the maximum number of concurrent connections supported by the DUT/SUT, as defined in RFC2647[1]. This test will employ a step search algorithm to obtain the maximum number of concurrent FTP connections the DUT/SUT can maintain. 5.1.2 Setup Parameters The following parameters MUST be defined. Each variable is configured with the following considerations. Connection Attempt Rate - The rate at which new connection requests are attempted. The rate SHOULD be set lower than maximum rate at which the DUT/SUT can accept new connection requests. The connection attempt rate MUST be evenly divided among all of the participating virtual clients. Connection Step count - Defines the number of additional connections attempted for each iteration of the search algorithm. The step count SHOULD be a multiple of the number of virtual clients participating in the test. In addition, the number MUST be greater than or equal to the number of virtual clients participating in the test. 5.1.3 Procedure Each virtual client will attempt to establish FTP control connections by issuing an OPEN command to their target server(s) at a fixed rate. Each iteration will involve the virtual clients attempting to establish a fixed number of additional connections. This search algorithm will be repeated until either: - One or more of the additional connection attempts fail to complete - One or more of the previously established connections drop. The transactions between the client and server associated with establishing FTP connections are defined in Appendix A. In order to verify that work can be performed across that connection, the virtual client MUST issue a NOOP command. This command is in addition to the transactions define in Appendix A, and must be accompanied by the Command Successful reply from the target server in order to be considered a successful connection. Martin, Hickman [Page 7] INTERNET-DRAFT Benchmarking Methodology for Firewalls June 2000 After each iteration, all virtual clients MUST issue a NOP command across all of their open connections to verify that none of the previously opened connections were dropped by the DUT/SUT. The algorithm for the test is as follows: CONSTANT STEP = Connection Step Count; {# of additional connection attempts per iteration} MAX = ...; {maximum connections supported by test tool implementation} VARIABLE CURRENT := 0; {Highest passed value} STATUS = PASS; BEGIN RESET; {RESET all connections} DO BEGIN EstablishStepConnections; { See Appendix A.3} IF(AttemptsSucessful) THEN BEGIN VerifyAllConnections { Send NOP command over all open connections } IF(Received replies for all NO0P commands) THEN BEGIN { Maximum number of connections NOT found } CURRENT := CURRENT + Step Count END ELSE BEGIN { Maximum number of connections < Step Count } STATUS := FAIL END END ELSE BEGIN { Maximum number of connections < Step Count } STATUS := FAIL END END END WHILE(STATUS == PASS && [[CURRENT + Step Count] < MAX]) END END {Value of CURRENT equals number connections supported by DUT/SUT} 5.1.4 Measurements Whether all of the connection attempts were successfully completed. Test equipment MUST be able to track each connection to verify all required transaction between the virtual client and server completed successfully. This includes successful completion of both the command sequences and exchanging of any data across each of those connections. Martin, Hickman [Page 8] INTERNET-DRAFT Benchmarking Methodology for Firewalls June 2000 5.1.5 Reporting Format Maximum concurrent connections reported MUST be the aggregate number of connections completed for the last successful iteration. Report SHOULD also include: - Number of virtual clients and servers participating in the test - Whether NAT was enabled or disabled. 5.2 Maximum Connection Setup Rate 5.2.1 Objective To determine the maximum connection rate which the DUT/SUT can successfully establish connections. As with the Concurrent Connection Capacity test, FTP session will be used to determine this metric. 5.2.2 Setup Parameters The following parameters MUST be defined. Each variable is configured with the following considerations. Initial Attempt Rate - The rate at which the initial connection requests are attempted. The connection attempt rate MUST be evenly divided among all of the participating virtual clients. Rate Step Count - The rate at which connection attempts are either increased or decreased during the search algorithm. The rate step count MUST be evenly divided among all of the participating virtual clients. Number of Connections - Defines the number of connections that must be established. The number MUST be between the number of participating virtual clients and the maximum number supported by the implementation. It is RECOMMENDED not to exceed the concurrent connection capacity found in section 5.1. 5.2.3 Procedure An algorithm similar to the one used to determine the concurrent connection capacity will be used to determine the maximum connection rate. This test iterates the rate at which connections are attempted by the virtual clients to their associated server(s). The connection establishment rate might be determined for different numbers of connections but in each test run, the number MUST remain constant and SHOULD be equal to or less than the maximum connection capacity. Martin, Hickman [Page 9] INTERNET-DRAFT Benchmarking Methodology for Firewalls June 2000 As with the connection capacity test, NOOP commands will be generated when both establishing the connection and after all connections have been established to verify none of the previously established connections have dropped. 5.2.4 Measurements Min/Avg/Max connection times MUST be measured. The connection time SHALL begin when the client initiates the OPEN command and end when the client received a successful login acknowledgment from the server. Note that although the NOOP command is included as part of the procedure for establishing the connection, it MUST not be included as part of the connection establishment time. 5.2.5 Reporting Format The results for these tests SHOULD be reported in the form of a graph. The x coordinate SHOULD be the connection count. The y coordinate should be the connection establishment time. The graph SHOULD include the Minimum, Average and Maximum connection times. Report SHOULD also include: - Number of virtual clients and servers participating in the test - Whether NAT was enabled or disabled. 5.3 Denial Of Service Handling 5.3.1 Objective To determine the effect of a denial of service attack in so far as it's impacts on connection establishment rates through the DUT/SUT. The Denial Of Service Handling test should be ran after obtaining baseline measurements from section 5.2. The results can then be compared with the baseline measurements to obtain the performance differentials. 5.3.2 Setup Parameters The following parameters MUST be defined. Each variable is configured with the following considerations. Initial Attempt Rate - The rate at which the initial connection requests are attempted. The connection attempt rate MUST be evenly divided among all of the participating virtual clients. Rate Step Count - The rate at which connection attempts are either increased or decreased during the search algorithm. The rate step count MUST be evenly divided among all of the virtual clients participating in the test. Martin, Hickman [Page 10] INTERNET-DRAFT Benchmarking Methodology for Firewalls June 2000 Number of Connections - Defines the number of connections that must be established. The number MUST be between the number of participating clients and the maximum number supported by the implementation. It is RECOMMENDED not to exceed the concurrent connection capacity found in section 5.1. SYN Attack Rate - Defines the rate at which the server(s) are targeted with TCP SYN packets. 5.3.3 Procedure This test will use the same procedure as defined in the maximum connection setup rate, except that the test traffic will include TCP SYN packets targeting the server(s) IP address or NAT proxy address. TCP employs a "three-way handshake" when establishing a connection between two hosts. When a normal TCP connection starts, a destination host receives a SYN (synchronize/start)packet from a source host and sends back a SYN ACK (synchronize acknowledge). The destination host must then hear an ACK (acknowledge) of the SYN ACK before the connection is established. The TCP SYN attack exploits this design by having an attacking source host generate TCP SYN packets with random source addresses towards a victim host, thereby consuming that hosts resources. The tester originating the TCP SYN attack MUST be attached to the Unprotected network. In addition, the tester MUST not respond to the SYN ACK packets sent by target server in response to the SYN packet. 5.3.4 Measurements Min/Avg/Max connection times MUST be measured. The connection time SHALL begin when the client initiates the OPEN command and end when the client received a successful login acknowledgment from the server. Note that although the NOOP command is included as part of the procedure for establishing the connection, it MUST not be included as part of the connection establishment time. 5.2.5 Reporting Format The results for these tests SHOULD be reported in the form of a graph. The x coordinate SHOULD be the connection count. The y coordinate should be the connection establishment time. The graph SHOULD include the Minimum, Average and Maximum connection times. Report SHOULD also include: - Number of virtual clients and servers participating in the test - Whether NAT was enabled or disabled. - TCP SYN Attack Rate Martin, Hickman [Page 11] INTERNET-DRAFT Benchmarking Methodology for Firewalls June 2000 5.4 Single Application Goodput This section defined the procedures for baselining the Goodput of the DUT/SUT for FTP, HTTP and SMTP traffic. 5.4.1.1 FTP Goodput 5.4.1.2 Objective The File Transfer Protocol is a common application used by companies to transfer files from one device to another. Evaluating FTP Goodput will allow individuals to measure how much successful traffic has been forwarded by the DUT/SUT. 5.4.1.3 Setup Parameters The following parameters MUST be defined. Each variable is configured with the following considerations. Number of Connections - Defines the number of connections to be opened for transferring data objects. Number MUST be equal or greater than the number of virtual clients participating in the test. The number SHOULD be a multiple of the virtual client participating in the test. Connection Rate - Defines the rate at which connections are established. Number MUST be evenly divided among all of the virtual clients participating in the test. Object Size - Defines the number of bytes to be transferred across each connection. RECOMMENDED object sizes still need to be determined. 5.4.1.4 Procedure Each virtual client will establish a FTP connection to its respective server(s) at a user specified rate. The transaction involved in establishing the FTP connection should follow the procedure defined in Appendix A. After the login process has been completed, the virtual client will initiate a file transfer by issuing a "Get" command. The "Get" command will target a predefined file of object size bytes. Once the file transfer has completed, the virtual client will close the FTP connection by issuing the QUIT command. When testing multiple object sizes, all connections established for the previous file transfer MUST have been torn down before testing the next object size. Martin, Hickman [Page 12] INTERNET-DRAFT Benchmarking Methodology for Firewalls June 2000 5.4.2.2 Measurement The Goodput for each of the object sizes transferred MUST be measured. See appendix B for details in regards to measuring the Goodput of the DUT/SUT. Only file transfers which have been successfully acknowledged by the server are to be included in the Goodput measurements. The transaction time for each object transferred MUST be measured. The start time will begin when the time the "Get" commands is initiated and end at the time when the client receives an acknowledgment from the server that file transfer has completed. 5.4.2.3 Reporting Format Goodput analysis: Reporting SHOULD include a graph of the number of connections versus the measured Goodput in Mbps. Failure analysis: Reporting should include a graph of number of connections versus percent success. Transaction Processing analysis: Reporting should include a graph of number of virtual connections versus average transaction time. 6.4.3 SMTP Goodput Another application commonly in use today is the mail transfer. One the common transport mechanisms for mail messages is the Simple Mail Transfer Protocol(SMTP). As with the FTP Goodput, the SMTP Goodput will allow individuals to measure how much successful SMTP traffic has been forwarded by the DUT/SUT. 5.4.1.3 Setup Parameters The following parameters MUST be defined. Each variable is configured with the following considerations. Number of Connections - Defines the number of connections to be opened for transferring data objects. Number MUST be equal or greater than the number of virtual clients participating in the test. The number SHOULD be a multiple of the virtual client participating in the test. Connection Rate - Defines the rate at which connections are attempted. Number MUST be evenly divided among all of the virtual clients participating in the test. Martin, Hickman [Page 13] INTERNET-DRAFT Benchmarking Methodology for Firewalls June 2000 Message Size - Defines the number of bytes to be transferred across each connection. RECOMMENDED message sizes still need to be determined. 5.4.1.4 Procedure Each virtual client will establish a SMTP connection to its respective server(s) at a user specified rate. The transaction involved in establishing the SMTP connection should follow the procedure defined in Appendix B. After the greeting exchanges have been completed, the client will initiate the transfer of the message by initiating the MAIL command. The client will then send the predefined message. Once the message has been acknowledged as being received by the server, the virtual client will then close the connection. When testing multiple message sizes, all connections established for the previous test MUST be torn down before testing the next message size. This process is to be repeated for each of the defined message sizes. 5.4.2.2 Measurement The Goodput for each of the message sizes transferred MUST be measured. See appendix D for details in regards to measuring the Goodput of the DUT/SUT. Only messages which have been successfully acknowledged by the server are to be included in the Goodput measurements. The transaction time for each message transferred MUST be measured. The start time will begin when the time the "MAIL" command is initiated and end at the time when the client receives an acknowledgment from the server that the message has been received. 6.4.2.3 Reporting Format Goodput analysis: Reporting SHOULD include a graph of the number of connections versus the measured Goodput in Mbps. Failure analysis: Reporting should include a graph of number of connections versus percent success. Transaction Processing analysis: Reporting should include a graph of number of virtual connections versus average transaction time. Martin, Hickman [Page 14] INTERNET-DRAFT Benchmarking Methodology for Firewalls June 2000 6.4.3 HTTP Goodput Another common application is the World Wide Web (WWW) application that can access documents over the Internet. This application uses the Hypertext Transfer Control Protocol (HTTP) to copy information from one system to another. HTTP Goodput measurement is actually determined by evaluating the Forwarding rate of packets. This is based on measuring only data that has successfully been forwarded to the destination interface. When benchmarking the performance of the DUT/SUT, consideration of the HTTP version being used must be taken into account. Appendix C of this document discusses enhancements to the HTTP protocol which may impact performance results. 6.4.1.3 Setup Parameters The following parameters MUST be defined. Each variable is configured with the following considerations. Number of Connections - Defines the number of HTTP connections to be opened for transferring data objects. Number MUST be equal or greater than the number of virtual clients participating in the test. The number SHOULD be a multiple of the virtual client participating in the test. Connection Rate - Defines the rate at which connections are attempted. Number MUST be evenly divided among all of the virtual clients participating in the test. Object Size - Defines the number of bytes to be transferred across each connection. Need to determine the RECOMMENDED object sizes. 6.4.2.1 HTTP Procedure For the HTTP Goodput tests, it is RECOMMENDED to determine which version of HTTP the DUT/SUT has implemented and use the same version for the test. To determine the version of HTTP, the user documentation of the DUT/SUT SHOULD be consulted. Each client will attempt to establish HTTP connection's to their respective servers a user defined rate. The clients will attach to the servers using either the servers IP address or NAT proxy address. After the client has established the connection with the server, the client will initiate GET command(s) to retrieve predefined web page(s). Martin, Hickman [Page 15] INTERNET-DRAFT Benchmarking Methodology for Firewalls June 2000 When employing HTTP/1.0 in benchmarking the performance of the DUT/SUT, only one object will be retrieved for each of the defined object sizes. After the object has been transferred, the connection should then be torn down. When defining multiple objects, object transfers must be completed and the connections closed for all of the participating clients prior testing the next object size. This process is repeated until all of the defined objects are tested. When employing HTTP/1.1, all objects defined by the user will be requested with a GET request over the same connection. The connection should then be torn down after all of the objects have been transferred. 6.4.2.2 Measurement The Goodput for each of the objects sizes transferred MUST be measured. See appendix D for details in regards to measuring the Goodput of the DUT/SUT. Only objects which have been successfully acknowledged by the server are to be included in the Goodput measurements. The transaction times for each object transferred MUST measured. The transaction connection time starts when the connection is made and will end when the web pages is completely mapped on the virtual client (when the client sends an acknowledgment packet is sent from the client). 6.4.2.3 Reporting Format Goodput analysis: Reporting SHOULD include a graph of the number of connections versus the measured Goodput in Mbps. Failure analysis: Reporting should include a graph of number of connections versus percent success. Transaction Processing analysis: Reporting should include a graph of number of virtual connections versus average transaction time. Version Information Report MUST include the version of HTTP used for the test. In addition, if the HTTP/1.1 is used, the number of concurrent GET's allowable(Pipelining) SHOULD be reported. Martin, Hickman [Page 16] INTERNET-DRAFT Benchmarking Methodology for Firewalls June 2000 6.5 Concurrent Application Goodput To determine the Goodput of the DUT/SUT when offering a mix of FTP, SMTP and HTTP traffic flows. Real world traffic does not consist of a single protocol, but a mix of different applications. This test will allow an individual to determine how well the DUT/SUT handles a mix of applications by comparing the results to the individual baseline measurements. 6.5.1 Setup Parameters The following parameters MUST be defined. Each variable is configured with the following considerations. Number of Connections - Defines the aggregate number of connections to be opened for transferring data/message objects. Number MUST be equal to or greater than the number of virtual clients participating in the test. The number SHOULD be a multiple of the virtual client participating in the test. Connection Rate - Defines the rate at which connections attempts are opened. Number MUST be evenly divided among all of the virtual clients participating in the test. Object/Message Size - Defines the number of bytes to be transferred across each connection. RECOMMENDED message sizes still needs to be determined. At a minimum, at least one of the following parameters MUST be defined. In addition, the cumulative percentage all the defined percentages MUST equal 100%. FTP Percentage - Defines the percentage of traffic connections which are to consist of FTP file transfers. SMTP Percentage - Defines the percentage of traffic connections which are to consist of SMTP Message transfers. HTTP Percentage - Defines the percentage of traffic connections which are to consist of HTTP GET requests. 6.5.1 Procedure This test will run each of the single application Goodput tests, for which there is a defined percentage, concurrently. For each of the defined traffic types, the connection establishment, data/message transfer and teardown procedures will be the same as defined in the individual tests. Martin, Hickman [Page 17] INTERNET-DRAFT Benchmarking Methodology for Firewalls June 2000 6.5.2 Measurements As with the individual tests, the Goodput for each of the defined traffic types MUST be measured. See appendix D for details in regards to measuring the Goodput of the DUT/SUT. Only messages/data which have been successfully acknowledged as being transferred are to be included in the Goodput measurements. The transaction times for each of the defined applications MUST be measured. See the appropriate single application Goodput test for the specifics of measuring the transaction times for each of the defined traffic types. 6.5.3 Reporting Format Goodput analysis: Reporting SHOULD include a graph of the number of connections versus the measured Goodput in Mbps for each of the defined traffic types(FTP, SMTP, HTTP). Failure analysis: Reporting should include a graph of number of connections versus percent success for each of the defined traffic types. Transaction Processing analysis: Reporting should include a graph of number of virtual connections versus average transaction for each of the defined traffic types. 6.6 Illegal Traffic Handling To determine the behavior of the DUT/SUT when presented with a combination of both legal and Illegal traffic. 6.6.1 Procedure Still Needs to be determined 6.6.2 Measurements Still Needs to be determined 6.6.3 Reporting Format Still Needs to be determined Martin, Hickman [Page 18] INTERNET-DRAFT Benchmarking Methodology for Firewalls June 2000 6.7 Latency Determine the latency of application layer data through the DUT/SUT. 6.7.1 Procedure Still Needs to be determined 6.7.2 Measurements Still needs to be determined. 6.7.3 Reporting format Still needs to be determined. APPENDICES APPENDIX A: FTP(File Transfer Protocol) A.1 Introduction The FTP protocol was designed to be operated by interactive end users or application programs. The communication protocol to transport this service is TCP. The core functions of this application enable users to copy files between systems, view directory listings and perform house keeping chores - such as renaming, deleting and copying files. Unlike other protocols, FTP uses two connections. One connection, called the control connection, is used to pass commands between the client and the server. The other, called the data connection, is used to transfer the actual data(Files, directory lists, etc.). A.2 Connection Establishment/Teardown(Control) FTP control connections are established by issuing OPEN command targeting either the URL or a specific IP address. Since the methodology does not include DNS servers, OPEN commands should target specific IP address of target server. It is RECOMMENDED to perform the test using Anonymous FTP Login and should use the following syntax: User ID: Anonymous Password: will correspond to the System ID (client1_1@test.net through client 1_8@test.net) Once a successful login acknowledgment is received from the server, the client may then initiate a file transfer. After all transfer operations have been completed, the FTP connection may be closed by issuing a QUIT command. Martin, Hickman [Page 19] INTERNET-DRAFT Benchmarking Methodology for Firewalls June 2000 A.3 Data Connection The data connection is established each time the user requests a file transfer and torn down when the transfer is completed. FTP supports two modes of operation, namely normal mode and passive mode, which determine who initiates the data connection. In normal mode operation, the server initiates the data connection, targeting a predefined PortID specified in the PORT command. In passive mode, the client initiates the data connection, targeting the PortID returned in response to the PASV Command. It is RECOMMENDED to perform the tests in normal mode operation. File transfers are initiated by using the "Get" or "Put" command and specifying the desired filename. The tests defined in this document will use the "Get" command to initiate file transfers from the target server to the client. A.4 Object Format Need to define the object format. APPENDIX B: SMTP (Simple Mail Transfer Protocol) B.1 Introduction The SMTP defines a simple straight forward way to move messages between hosts. There are two roles in the SMTP protocol, one is the sender and one is the receiver. The sender acts like a client and establishes a TCP connection with the receiver which acts like a server. The transactions defined in this section will use the terms client and server in place of sender and receiver. B.2 Connection Establishment/Teardown Each connection involves a connection greeting between the sender(Client) and receiver(Server). After a connection is first established, both the server and the client will exchange greetings identifying themselves. The syntax used to identify each other's hostnames SHOULD be: "SMTPRcv_.com" "SMTPSender_.com" where and represent a unique virtual number for the client and server respectively. Martin, Hickman [Page 20] INTERNET-DRAFT Benchmarking Methodology for Firewalls June 2000 The basic transactions in moving mail between two hosts involve three basic steps which are outlined below. These are: 1) Client issuing a MAIL command identifying the message originator for that session. Syntax used to identify the originator SHOULD be as follows: connection1,2,3...@hostname 2) Client issues an RCPT command identifying the recipient of the message for that session. Syntax used to identify the recipient of the message SHOULD be as follows: reciever1,2,3...@hostname 3) Client issues a DATA command. After receiving the acknowledgment from the server, the client will then transfer the message which MUST include a line with a period to indicate to the server the end of the message. Once the end of message is received by the server, it will acknowledge the end of message. The client may initiate another message transfer or close the session by initiating the QUIT command. B.3 Message Format As Internet e-mail has evolved, SMTP extensions have been added to support both audio and video message transfers. For these firewall tests, messages SHOULD consist of plain text ASCII. APPENDIX C: HTTP(HyperText Transfer Protocol) C.1 Introduction As HTTP has evolved over the years, changes to the protocol have occurred to both fix problems of previous versions as well as improve performance. The most common versions in use today are HTTP/1.0 and HTTP/1.1 and are and are discussed below. C.2 Version Considerations HTTP/1.1 was approved by the WWW Consortium in July 1999 as an IETF Draft Standard. This is a formal recognition of the fact that all known technical issues have been resolved in the specification which was brought out in June 1997. HTTP/1.1 is also downward compatible with HTTP/1.0. Both protocols on the popular browsers in use today. The following is a list of features that are offered in HTTP 1.1 that are not in HTTP 1.0. Martin, Hickman [Page 21] INTERNET-DRAFT Benchmarking Methodology for Firewalls June 2000 - Persistent connections and pipelining Though both use TCP for data transfer, but differ in the way it is used, with the later version being more efficient. Once a connection is opened, it is not closed until the HTML document and all objects referred by it are downloaded. This technique is called persistent connection. By serving multiple requests on the same TCP segment, many control packets (which are not part of actual data transfer) are avoided. The technique of containing multiple requests and responses within the same TCP segment over a persistent connection is called pipelining. - Data compression HTTP/1.1 provides for compression of documents before file transfer. Since most other objects like images and binaries are already compressed, this feature applies only to HTML and plain text documents. - Range request and validation Bandwidth saving measure is the introduction of two new fields in an HTTP request header, viz. If-Modified-Since: and If-Unmodified- Since:. The significance of this feature is that if a browser identifies a file in its cache, it needn't reload it unless it has changed since the last time it was used. - Support for multiple hosts It is common for an ISP to host more than one Web site on a single server. In such a case, each domain requires its own IP address. C.3 Object Format Object SHOULD be an HTLM formatted object. Martin, Hickman [Page 22] INTERNET-DRAFT Benchmarking Methodology for Firewalls June 2000 Append D GOODPUT Measurements Methodology for determining the "Goodput" of the DUT still needs to be determined. Note that the methodology should be independent of the application which is initiating the transfer of the object (File, message, Web Page, etc.). Appendix E. References Newman, "Benchmarking Terminology for Firewall Devices", RFC 2647, February 1998. J. Postel, "Simple Mail Transfer Protocol", RFC 821, August 1982. R. Fielding, J. Gettys, J. Mogul, H Frystyk, T. Berners, "Hypertext Transfer Protocol -- HTTP/1.1", January 1997 J. Postel, J. Reynolds, "File Transfer Protocol(FTP)", October 1985 Martin, Hickman [Page 23] INTERNET-DRAFT Benchmarking Methodology for Firewalls June 2000