Internet Engineering Task Force Jan Novak Internet-Draft Benoit Claise Intended status: Informational Cisco Systems, Inc. Expires: January 1, 2010 July 13, 2009 IP Flow Information Accounting and Export Benchmarking Methodology draft-novak-bmwg-ipflow-meth-03.txt Abstract This document provides methodology and framework for quantifying performance of selective monitoring of IP flows on a network device and export of this information to a collector. It is based on the Architecture for IP Flow Information Export [RFC5470]. Status of this Memo This Internet-Draft is submitted to IETF in full conformance with the provisions of BCP 78 and BCP 79. Internet-Drafts are working documents of the Internet Engineering Task Force (IETF), its areas, and its working groups. Note that other groups may also distribute working documents as Internet- Drafts. Internet-Drafts are draft documents valid for a maximum of six months and may be updated, replaced, or obsoleted by other documents at any time. It is inappropriate to use Internet-Drafts as reference material or to cite them other than as "work in progress." The list of current Internet-Drafts can be accessed at http://www.ietf.org/ietf/1id-abstracts.txt. The list of Internet-Draft Shadow Directories can be accessed at http://www.ietf.org/shadow.html. This Internet-Draft will expire on January 1, 2010. Copyright Notice Copyright (c) 2009 IETF Trust and the persons identified as the document authors. All rights reserved. This document is subject to BCP 78 and the IETF Trust's Legal Provisions Relating to IETF Documents in effect on the date of publication of this document (http://trustee.ietf.org/license-info). Please review these documents carefully, as they describe your rights and restrictions with respect to this document. Novak Expires January 1, 2010 [Page 1] Internet-Draft Flow Monitoring Benchmarking July 2009 Conventions used in this document The key words "MUST", "MUST NOT", "REQUIRED", "SHALL", "SHALL NOT", "SHOULD", "SHOULD NOT", "RECOMMENDED", "MAY", and "OPTIONAL" in this document are to be interpreted as described in RFC 2119 [RFC2119]. Table of Contents 1. Introduction . . . . . . . . . . . . . . . . . . . . . . . . 2 2. Terminology . . . . . . . . . . . . . . . . . . . . . . . . . 4 2.1 Existing Terminology . . . . . . . . . . . . . . . . . . 4 2.2 New Terminology. . . . . . . . . . . . . . . . . . . . . 4 3. Flow Monitoring Performance Metric . . . . . . . . . . . . . 6 3.1 The Definition . . . . . . . . . . . . . . . . . . . . . . . 6 3.2 The Applicability. . . . . . . . . . . . . . . . . . . . . . 6 3.3 The Concept. . . . . . . . . . . . . . . . . . . . . . . . . 7 3.4 Setup Examples . . . . . . . . . . . . . . . . . . . . . . . 8 4. Test Set Up . . . . . . . . . . . . . . . . . . . . . . . . .11 4.1 Test Topology. . . . . . . . . . . . . . . . . . . . . .11 4.2 Base DUT Set Up. . . . . . . . . . . . . . . . . . . . .12 4.3 Flow Monitoring Configuration. . . . . . . . . . . . . .12 4.4 Collector. . . . . . . . . . . . . . . . . . . . . . . .14 4.5 Packet Sampling. . . . . . . . . . . . . . . . . . . . .15 4.6 Frame Formats. . . . . . . . . . . . . . . . . . . . . .16 4.7 Frame Sizes. . . . . . . . . . . . . . . . . . . . . . .16 4.8 DUT Load . . . . . . . . . . . . . . . . . . . . . . . .16 5. Flow Monitoring Throughput Measurement Methodology. . . . . .16 5.1 Normal Cache Mode . . . . . . . . . . . . . . . . . . . .17 5.2 Cache Overflow Mode . . . . . . . . . . . . . . . . . . .18 6. RFC2544 Measurements. . . . . . . . . . . . . . . . . . . . .20 6.1 Flow Monitoring Configuration . . . . . . . . . . . . . .20 6.2 Single Traffic Component. . . . . . . . . . . . . . . . .20 6.3 Two Traffic Components. . . . . . . . . . . . . . . . . .21 7. Flow Monitoring Accuracy . . . . . . . . . . . . . . . . . . 21 8. Evaluating Flow Monitoring Applicability . . . . . . . . . . 22 9. Acknowledgements . . . . . . . . . . . . . . . . . . . . . . 23 10.IANA Considerations. . . . . . . . . . . . . . . . . . . . . 23 11.Security Considerations . . . . . . . . . . . . . . . . . . .23 12.References. . . . . . . . . . . . . . . . . . . . . . . . . .23 12.1 Normative References. . . . . . . . . . . . . . . . . .23 12.2 Informative References. . . . . . . . . . . . . . . . .24 Novak Expires January 1, 2010 [Page 2] Internet-Draft Flow Monitoring Benchmarking July 2009 1. Introduction Monitoring of IP flows (Flow Monitoring) on network devices is a widely used application that has numerous uses in both service provider and enterprise segments as detailed in the Requirements for IP Flow Information Export [RFC3917]. Flow Monitoring is defined in the Architecture for IP Flow Information Export [RFC5470] and related IPFIX documents. What is the cost of enabling the IP Flow Monitoring and export to a collector is a basic question that this document tries to answer. This document goal is a series of methodology specifications for the monitoring of Flow Monitoring performance, in a way that is comparable amongst various implementations, various platforms, and vendors. The most impacting parameter in terms of performance is the rate at which IP flows are created and expired in the devices memory and exported to a collector. Therefore, this document focuses on a methodology on how to measure the maximum IP flow rate that a network device can sustain without impacting the forwarding plane, without losing any IP flow information, and without compromising the IP flow accuracy. Since Flow Monitoring will in most cases run on network devices forwarding packets, methodology for RFC2544 measurements in the presence of Flow Monitoring is also proposed here. [RFC2544], [RFC5180] and [MPLS] specify benchmarking of network devices forwarding IPv4, IPv6 and MPLS [RFC3031] traffic, respectively. This draft document also proposes the methodology of performing this kind of benchmarking in the presence of Flow Monitoring, but not necessarily restricted to IPv4, IPv6 and MPLS traffic types. The methodology stays the same for any traffic, the only restriction being what the actual Flow Monitoring implementation supports. The impact of Flow Monitoring on the network devices central processor unit utilisation is out of scope of this document to avoid white box testing even though it represents an interesting performance metric for the comparison of different measurement profiles. Novak Expires January 1, 2010 [Page 3] Internet-Draft Flow Monitoring Benchmarking July 2009 2. Terminology 2.1 Existing Terminology Device Under Test (DUT) [RFC2285, section 3.1.1] Flow [RFC5470, section 2] Flow Key [RFC5470, section 2] Flow Record [RFC5470, section 2] Observation Point [RFC5470, section 2] Metering Process [RFC5470, section 2] Exporting Process [RFC5470, section 2] Exporter [RFC5470, section 2] Collector [RFC5470, section 2] Control Information [RFC5470, section 2] Data Stream [RFC5470, section 2] Flow Expiration [RFC5470, section 5.1.1] Flow Export [RFC5470, section 5.1.2] Throughput [RFC1242, section 3.17] 2.2 New Terminology 2.2.1 Cache Definition: Memory area held and dedicated by the DUT to store Flow Record information 2.2.2 Cache Size Definition: The size of the cache in terms of how many entries of Flow Records the cache can hold Discussion: This term is typically represented as a configurable option in the particular Flow Monitoring implementation. It MUST be at least known in order to define the tests circumstances properly. Its highest value will depend on the memory available in the network device. Novak Expires January 1, 2010 [Page 4] Internet-Draft Flow Monitoring Benchmarking July 2009 Measurement units: Number of entries 2.2.3 Active Timeout Definition: The time interval time after which the Metering Process expires a Flow Record from the Cache while the packets belonging to that specific Flow are still seen. Discussion: This term is typically represented as a configurable option in the particular Flow Monitoring implementation. See section 5.1.1 of [RFC5470] for more detailed discussion. It MUST be known in order to define the tests circumstances properly. Measurement units: Seconds 2.2.4 Inactive Timeout Definition: The time interval after which the Metering Process expires a Flow Record from the Cache if no more packets belonging to that specific Flow are seen. Discussion: This term is typically represented as a configurable option in the particular Flow Monitoring implementation. See section 5.1.1 of [RFC5470] for more detailed discussion. It MUST be known in order to define the tests circumstances properly. Measurement units: Seconds 2.2.5 Flow Expiration Rate Definition: Number of Flow Records that expire from the Cache (as defined by the Flow Expiration term) from the Cache within a time interval Measurement units: Number of Flow Records per second Novak Expires January 1, 2010 [Page 5] Internet-Draft Flow Monitoring Benchmarking July 2009 3. Flow Monitoring Performance Metric 3.1 The Definition Flow Monitoring Throughput Definition: The maximum Flow Expiration Rate the DUT can sustain without losing any information about any offered Flow while exporting the Flow Record information to an external collecting device (Collector) and as measured at the Collector. The measured Flow Expiration Rate MUST include the Control Information. Measurement units: Number of Flow Records per second 3.2 The Applicability The Flow Monitoring Performance Metric is applicable to network devices which implement the RFC 5470 [RFC5470] architecture. These devices can be network packet forwarding devices or appliances which analyse the traffic but do not forward traffic (probes, sniffers, replicators). The Flow Monitoring Performance Metric is not applicable to the Collector since it does not implement the RFC 5470 architecture and the factors influencing performance can be quite different. Novak Expires January 1, 2010 [Page 6] Internet-Draft Flow Monitoring Benchmarking July 2009 3.3 The Concept Figure 1. The functional block diagram of the DUT +-------------------------+ | ^ | | ^ | | Flow Export | | ^ | | ^ | | +-------------+ | | | | | | | Flow | | | | Exporter | | | | | | | +-------------+ | | ^ | | ^ | | Flow Expiration | | ^ | | ^ | | +-------------+ | | | | | | | Flow | | | | Monitoring | | | | Cache | | | | | | | +-------------+ | | ^ | | ^ | | Flow Records | | ^ | | ^ | external | +-------------+ | generator | | | | traffic ---|---->| Forwarding |-----|----> | | plane | | | | | | | +-------------+ | | | | DUT | +-------------------------+ Traffic in the Figure 1 represents the test traffic sent to the DUT and forwarded by the DUT The Flow Monitoring enabled (see section 4.3) on the DUT uses the Flow Keys to create the Flow Records representing the live traffic forwarded by the DUT. The Flow Records are stored in the Flow Monitoring Cache and expired from there depending on the Cache configuration (mainly timeouts, number of Flow Records and the Cache size) and the traffic pattern (if the Flow Keys in the traffic change Novak Expires January 1, 2010 [Page 7] Internet-Draft Flow Monitoring Benchmarking July 2009 or not). If Flow Export is configured then Flow Records are exported from the DUT to the Collector (see Figure 2 and Figure 3 in section 4). 3.4 Set-up Examples 3.4.1 Set-up Example 1 - Active Timeout Flow Expiration The traffic generator sends 1000 packets per second in 100 streams, each stream defined by an unique destination IP address, each stream has then packet rate 10 packets per second. The packets are sent in a round robin fashion while incrementing the destination IP address. The configured Cache size is 1000 Flow Records so that Flow Expiration does not happen due to the Cache overflow. The configured Active Timeout is 100 seconds, the Inactive Timeout is 10 seconds. Flow Monitoring on the DUT uses as Flow Key the destination IP address. After first 100 packets sent, 100 Flow Records are created and placed in the Flow Monitoring Cache. The consequent packets will be counted against the already created Flow Records since the destination IP address (Flow Key) has already been seen by the DUT (provided the Flow Record did not expire yet - see below). A packet with destination IP address equal to A is sent every 0.1 second, so it means that the Flow Record is refreshed in the Cache every 0.1 second, while the Inactive Timeout is 10 seconds. In this case the Flow Records will not expire from the Cache until the Active Timeout, e.g. they will expire every 100 seconds and then the Flow Records will be created again. If the test measurement time is 50 seconds from the start of the traffic generator the measured Flow Expiration Rate is 0 since during this period no Flow Records expired from the Cache. If the test measurement time is 100 seconds from the start of the traffic generator measured Flow Expiration Rate is 1 Flow Record per second. If the test measurement time is 300 seconds from the start of the traffic generator measured Flow Expiration Rate is 2/3 of Flow Record per second since during the 300 seconds period we expired 2 times the same 100 of Flows. This set-up is not suitable for any benchmarking since the Flow Expiration happens only at discrete time points and the measured Flow Expiration Rate will depend on how many of those time points were captured within the measurement period. Novak Expires January 1, 2010 [Page 8] Internet-Draft Flow Monitoring Benchmarking July 2009 3.4.2 Set-up Example 2 - Inactive Timeout Flow Expiration The traffic generator sends 1000 packets per second in 10000 streams, each stream defined by an unique destination IP address, each stream has then packet rate 0.1 packets per second. The packets are sent in a round robin fashion while incrementing the destination IP address. The configured Cache size is 20000 Flow Records so that Flow Expiration does not happen due to the Cache overflow. The configured Active Timeout is 100 seconds, the Inactive Timeout is 5 seconds. Flow Monitoring on the DUT uses as Flow Key the destination IP address. After first 100 packets sent, 100 Flow Records are created and placed in the Flow Monitoring Cache. A packet with destination IP address equal to A is sent every 10 seconds, so it means that the Flow Record is refreshed in the Cache every 10 seconds, while the Inactive Timeout is 5 seconds. In this case the Flow Records will expire from the Cache due to the Inactive Timeout and when a new packet is sent with the same IP address A it will create a new Flow Record in the Cache. The measured Flow Expiration Rate in this case will be 1000 Flow Records per second since each and single sent packet will always create a new Flow Record. This setup can be used for benchmarking since the measured Flow Expiration Rate does not depend on the measurement period supposing the measurement period is longer than the Inactive Timeout. If the measurement period is smaller or equal to the Inactive Timeout the measured Flow Expiration Rate will be zero. 3.4.3 Set-up Example 3 - Cache Overflow Flow Expiration The traffic generator sends 10000 packets per second in 10000 streams, each stream defined by an unique destination IP address, each stream has then packet rate 1 packets per second. The packets are sent in a round robin fashion while incrementing the destination IP address. Flow Monitoring on the DUT uses as Flow Key the destination IP address. The configured Cache size is 1000 Flow Records so the Cache overflows after first 1000 packets sent (provided the Flow Records do not expire due to timeouts - see below). The configured Active Timeout is 100 seconds, the Inactive Timeout is 10 seconds. The Cache will be full after the traffic generator sends first 1000 packets which will happen in 0.1 second so the Flow Records can never expire from the Cache due to any of Active or Inactive Timeouts. Each packet sent after this initial period of 0.1 second will create one new Flow Record and cause another Flow Record to expire from the Novak Expires January 1, 2010 [Page 9] Internet-Draft Flow Monitoring Benchmarking July 2009 Cache. The measured Flow Expiration Rate in this case will be 10000 Flow Records per second since each and single sent packet will always create a new Flow Record and cause expiration of another Flow Record. To perform this kind of measurement the Active and Inactive Timeouts must always be larger than the time needed to fill up the Cache. This setup can be used for benchmarking since the measured Flow Expiration Rate does not depend on the measurement period supposing the measurement period is longer than amount of time needed to fill up the Cache. Otherwise the measured Flow Expiration Rate will be zero. 4. Test Set Up 4.1 Test Topology The test set-up is identical to the one used by [RFC2544], with just an addition of a Collector to analyse the Flow Export: Figure 2 Test topology with unidirectional traffic +-----------+ | | | Collector | | | |Flow Record| | analysis | | | +-----------+ | | Flow Export | | Export Interface +--------+ +-------------+ +----------+ | | | | | | | | | | | receiver | | sender |-------->| DUT |--------->| | | | | | | traffic | | | | | | analysis | +--------+ +-------------+ +----------+ Novak Expires January 1, 2010 [Page 10] Internet-Draft Flow Monitoring Benchmarking July 2009 Figure 3 Test topology with bidirectional traffic +-----------+ | | | Collector | | | |Flow Record| | analysis | | | +-----------+ | | Flow Export | | Export Interface +----------+ +-------------+ +----------+ | | | | | | | sender |------->| |-------->| receiver | | | | DUT | | | | | | | | | | receiver |<-------| |<--------| sender | | | | | | | +----------+ +-------------+ +----------+ The ideal way to implement the test is using one traffic generator (device providing the sender and receiver capabilities) with a sending port and a receiving port. This allows for an easy check if all the traffic sent by the sender was transmitted by the DUT and received at the receiver. If the effects of enabling Flow Monitoring on several interfaces are of concern or the media maximum speed is less that the DUT throughput, the topology can be expanded with several input and output ports. The traffic forwarding interfaces are only those between the DUT and sender and between the DUT and the receiver. The interface towards the Collector MUST NOT be used for the test traffic forwarding but only for the Flow Export traffic. 4.2 Base DUT Set Up The base DUT set up and the way the set-up is reported in the test results is fully specified in Section 7 of [RFC2544]. The base DUT configuration might include other features like packet filters or quality of service on the input and/or output interfaces if there is the need to study Flow Monitoring in the presence of those features. The Flow Monitoring measurement procedures do not change in this case. Consideration needs to be made when evaluating measurements results to take into account the possible change of packets rates offered to the DUT and Flow Monitoring after application of the features to the configuration. Novak Expires January 1, 2010 [Page 11] Internet-Draft Flow Monitoring Benchmarking July 2009 4.3 Flow Monitoring Configuration The DUT MUST support Flow Monitoring architecture as specified by [RFC5470]. The DUT SHOULD support IPFIX [RFC5101] for easier results comparison. If the measurements are to be performed with packet sampling the DUT MUST support packet sampling as specified in [RFC5476] and related documents. The DUT configuration and any existing Cache MUST be fully erased before application of any new configuration for the currently executed test. 4.3.1 Observation Points The DUT Observation Points configuration needs to be decided upon depending on the interest and scope of the testing as follows: a. ingress port/ports only b. egress port/ports only c. both ingress and egress Generally, the placement of Observation Points depends upon the position of the DUT in the deployed network and the purpose of Flow Monitoring deployment. See [RFC3917] for detailed discussion. The testing procedures are otherwise the same for all these possible configurations. To measure input and output simultaneously on one DUT port the topology in the Figure 2 can be used if the traffic sender and receiver allows for full duplex operation on one port. The only change is in the configurations of the sender and receiver. If the traffic generator does not allow for full duplex operation, then the topology in Figure 3 needs to be used with separate sending and receiving ports in both directions. 4.3.2 Metering Process Metering Process MUST be enabled in order to create the Cache in the DUT and configure the Cache related parameters. Cache Size available to the DUT operation MUST be known and taken into account when designing the test as specified in the section 5. Inactive and Active Timeouts MUST be known and taken into account when designing the test as specified in the section 5. Novak Expires January 1, 2010 [Page 12] Internet-Draft Flow Monitoring Benchmarking July 2009 4.3.3 Exporting Process Exporting Process MUST be configured in such a way that all Flow Record information from all configured Observation Points is exported. This makes sure that Flow Expiration Rate as measured from the Collector data includes all the Flow information handled by the DUT at the time of the measurement. The measurement time to be used to capture Flow Export data and to calculate Flow Expiration Rate is defined in sections 5.1 and 5.2. Various Flow Monitoring implementations might use different default values regarding the export of Control Information. The Flow Export corresponding to Control Information SHOULD be analysed and reported as a separate item on the test report. IPFIX documents [RFC5101] in section 10 and [RFC5470] in section 8.1 discuss the possibility to deploy various transport layer protocols to deliver Flow Export data from the DUT to the Collector. The chosen protocol SHOULD be included in the test report. Only measurements with same transport layer protocol MUST be compared. 4.3.4 Flow Records The Flow Record definition is very implementation specific. A Flow Monitoring implementation might allow only for fixed Flow Record definition, based on the most common IP parameters in the IPv4 or IPv6 headers - like source and destination IP addresses, IP protocol numbers or transport level port numbers. Another implementation might allow the user to actually define his own completely arbitrary Flow Record to monitor the traffic. The requirement for the tests defined in this document is only the need for a large number of Flow Records in the Cache. The Flow Keys needed to achieve that will typically be source and destinations IP addresses and transport level port numbers. Recommended full Flow Record: Flow Key fields Source IP address Destination IP address Transport layer source port Transport layer destination port IP protocol IP Type of Service or IP flow label Other fields Packet counter Byte counter If the Flow Monitoring allows for user defined Flow Records the minimal Flow Record configurations allowing to achieve large Novak Expires January 1, 2010 [Page 13] Internet-Draft Flow Monitoring Benchmarking July 2009 numbers of Cache entries for example are: Flow Key fields Source IP address Destination IP address Other fields Packet counter or: Flow Key fields Transport layer source port Transport layer destination port Other fields Packet counter 4.3.5 MPLS Test Specifics The Flow Record configuration for measurements with MPLS encapsulated traffic SHOULD contain MPLS label or any field which is part of the MPLS header. The DUT Cache SHOULD be checked prior the performance test to contain the correct MPLS related information. The captured export data at the Collector SHOULD be checked for the presence of MPLS labels or the monitored MPLS parameters. MPLS forwarding performance document [MPLS] specifies number of possible MPLS label operations to test. The Observation Points SHOULD be placed on all the DUT interfaces where the particular MPLS label operation takes place. The performance tests SHOULD be performed with only one MPLS label operation at the time. The DUT SHOULD be configured in such a way, that all the traffic is subject of the tested MPLS label operation. 4.4 Collector The Collector SHOULD support IPFIX [RFC5101] for easier results analysis. If proprietary Flow Export is deployed, the Collector MUST support it. The Collector MUST be capable to capture at the full rate the export packets sent from the DUT without losing any of them. It does not have to have any Flow Export decoding capabilities itself, just needs Novak Expires January 1, 2010 [Page 14] Internet-Draft Flow Monitoring Benchmarking July 2009 to provide the captured data in the hexadecimal format for an off- line analysis which can be done after each performed measurement. During the analysis the Flow Export data need to be decoded and the received Flow Records counted. The Collector SHOULD support Ethernet type of interface to connect to the DUT but any media which allow data capturing and analysis can be used. The capture buffer MUST be cleared at the beginning of each measurement. 4.5 Packet Sampling A Flow Monitoring implementation might provide the capability to analyse the Flows after packet sampling is performed. The possible procedures and ways of packet sampling are described in [RFC5476] and [RFC5475] and only those SHOULD be used for measurements. If the DUT is configured with one of the sampling techniques as specified in [RFC5475] the test report MUST include the sampling technique along with its parameters. The Collector MUST support the deployed techniques. The configured packet sampling technique on the DUT and its parameters SHOULD be verified in the Flow Export data as received on the Collector. Packet sampling will affect the measured Flow Expiration Rate. If systematic sampling (see section 6.5 of [RFC5476]) is in use, the Flow Expiration Rate can be derived from the packet rates (see section 5 of this document) using the configured sampling parameters. If random sampling is in use the Flow Expiration Rate can be derived from the traffic rates as obtained on the receiver side of the traffic generator, provided that packet loses can be excluded by monitoring the DUT statistics. If tests are performed with Flows containing more than one packet (see section 6.2 of this document) the sampling ratio SHOULD always be higher than the number of packets in the Flows (for small number of packets). This decreases the probability of erasing a whole Flow to a minimum and the measured Flow Expiration rates stay unaffected by sampling. If Flow accuracy analysis is performed the results will be always affected by packet sampling and the complete check of data cannot be performed (see section 7). This document does not intend to study the effects of packet sampling itself on the network devices but packet sampling can simply be applied as part of the Flow Monitoring configuration on the DUT and perform the measurements as specified in the later sections. Consideration needs to be made when evaluating measurements results Novak Expires January 1, 2010 [Page 15] Internet-Draft Flow Monitoring Benchmarking July 2009 to take into account the change of packet rates offered to the DUT and especially to Flow Monitoring after packet sampling is applied. 4.6 Frame Formats Flow Monitoring itself is not dependent in any way on the media used on the input and output ports, any media can be used as supported by the DUT and the test equipment. The most common media and frame formats for IPv4, IPv6 and MPLS traffic are specified within [RFC2544], [RFC5180] and [MPLS]. 4.7 Frame Sizes Frame sizes to use are specified in [RFC2544] section 9 for Ethernet type interfaces (64, 128, 256, 1024, 1280, 1518 bytes) and in [RFC5180] section 5 for Packet over Sonet interfaces (47, 64, 128, 256, 1024, 1280, 1518, 2048, 4096 bytes). 4.8 DUT Load The Flow Monitoring Throughput SHOULD be measured under different levels of traffic load through the DUT. The traffic creating the load MUST be designed in such a way that it represents only one Flow Record in the DUT Cache. The variance in Flow Monitoring Throughput as function of the traffic load should be noted for comparison purposes between two DUTs of similar architecture and capability. 5. Flow Monitoring Throughput Measurement Methodology Objective: To define and measure metric fully expressing the Flow Monitoring performance which can be used to compare different implementations. Metric definition: Flow Monitoring Throughput - see section 3. Flow Expiration Rate calculation: The Flow Expiration Rate is obtained from the data captured at the Collector (section 4.4) as the number of Flows seen divided by the measurement time as specified in the section 5.1 and 5.2 The Flow Expiration Rate the DUT can achieve without losing any Flow information depends on the combination of a number of Flow Monitoring parameters and traffic patterns offered to the DUT. This makes it a complex task to define and measure a single metric which would characterise the Flow Monitoring performance. There are two Flow Novak Expires January 1, 2010 [Page 16] Internet-Draft Flow Monitoring Benchmarking July 2009 Monitoring operational modes which need to be distinguished and measured separately while using the same metric. 5.1 Normal Cache Mode In the normal cache mode the DUT MUST hold less Flow Records in the Cache than the available Cache Size during the whole test. Test Feasibility: The DUT Cache status MUST be monitored during this test. If the number of Flow Records starts to grow during the test and approach the Cache Size testing MUST be stopped. The reported Flow Monitoring Throughput for normal cache mode is the highest possible Flow Expiration Rate at which the number of Flow Records in the Cache stays constant or marginally increases during the measurement. The reason for this test mode is, that the Flow Monitoring implementation might choose to use different Flow Expiration mechanism as opposed to the test mode specified in section 5.2 and the results for the two modes can be quite different. 5.1.1 Flow Monitoring Configuration Parameters Cache Size: Maximum available value on the network device Inactive Timeout Minimum possible value on the network device Active Timeout Higher or equal value to the Inactive Timeout Flow Keys Definition: Needs to allow for large numbers of unique Flow Records to be created in the Cache by incrementing values of one or several Flow Keys. The number of unique combinations of Flow Keys MUST be larger than the DUT Cache Size or larger than the product of the offered packet rate and the Inactive Timeout so that the Flow Records in the Cache never get updated before Flow Expiration and Flow Export. 5.1.2 Traffic Configuration Parameters Traffic Generation The traffic generator (sender) when sending the packets needs to increment the packet header values defined as Flow Keys values in the Flow Record with each sent packet. Each packet then represents one Flow Record in the DUT Cache and the packet rate equals to the Flow Expiration Rate. Novak Expires January 1, 2010 [Page 17] Internet-Draft Flow Monitoring Benchmarking July 2009 Maximum Packet Rate The maximum packet rate which can be used for the normal cache mode measurement is the Cache Size divided by the Inactive Timeout. If the Flow Monitoring implementation allows to configure Inactive Timeout 0 then any packet rate is possible to be used. This makes sure the Flow Records expire and get exported from the Cache before it becomes full. Test Duration The test duration MUST be longer than Inactive Timeout. 5.1.3 Measurement Time The measurement time to be used to calculate Flow Expiration Rate is equal to the time interval when traffic is offered to the DUT. The Collector MUST continue capturing export data at least for the Inactive Timeout period after the traffic generation is stopped. This makes sure that all the Flow Records created in the DUT Cache will be exported and analysed. 5.1.4 Procedure The measurement procedure is same as the Throughput measurement in the section 26.1 of [RFC2544] for the traffic sending side. The output analysis is not done on the DUT receiving side but the analysed quantity for the measurement of the maximum rate value is the Flow Record count as provided by the Flow Export data captured at the Collector as specified in the section 4.4. If the maximum packet rate (as specified in the section 4.1.2) does not cause the DUT to drop any Flow Record then the reported Flow Monitoring Throughput SHOULD be equal to the maximum packet rate. The limitation here is that the configurable values of Cache Size and Inactive Timeout do not allow to test with higher Flow Expiration Rates in normal Cache mode. 5.2 Cache Overflow Mode In the cache overflow mode the DUT MUST have the Cache fully occupied by Flow Records at any time during the measurement. 5.2.1 Flow Monitoring Configuration Parameters Cache Size Cache Size should be configured to a moderate size accordingly to the expected DUT position in the network. Its value may be limited by the fact that the number of unique Flow Keys sets which the traffic generator (sender) can provide should be multiple times larger than the Cache Size to make sure that the Flow Records in the Cache never get updated before Flow Expiration and Flow Export. Novak Expires January 1, 2010 [Page 18] Internet-Draft Flow Monitoring Benchmarking July 2009 Example Cache Sizes: Small office device - 5 to 10 000 entries Medium access device - 50 to 100 000 entries Backbone device - 500 000 and higher Inactive Timeout Maximum possible value on the network device. The value has to be larger than the Cache Size divided by the test packet rate offered to the DUT to make sure the Flow Records do not expire from the Cache before the number of Flow Records in the Cache reaches the Cache Size. Active Timeout Higher or equal value to the Inactive Timeout Flow Keys Definition: Needs to allow for large numbers of unique Flow Records to be created in the Cache by incrementing values of one or several Flow Keys. The number of unique combinations of Flow Keys values SHOULD be at least two times larger than the DUT Cache Size. This makes sure that any incoming packet will never refresh any already existing Flow Record in the Cache and will cause another Flow Record to expire from the Cache. This is sometimes called emergency Flow Expiration as opposed to the inactive or active timeout of a Flow Record. 5.2.2 Traffic Configuration Parameters Traffic Generation The traffic generator needs to increment the Flow Keys values with each sent packet, this way each packet represents one Flow in the DUT Cache and the packet rate equals to the Flow Expiration Rate. Test Duration The test duration MUST be longer than the time needed to fully fill up the Cache with Flow Records. 5.2.3 Measurement Time The measurement time to be used to calculate Flow Expiration Rate equal to the time interval when traffic is offered to the DUT minus the time needed to fully fill the Cache calculated as Cache Size divided by the test traffic packet rate during the measurement. The Collector MUST stop capturing export data immediately when the traffic generation is stopped. This makes sure that the Flow Records exported later from the fully occupied Cache do not poison the measured Flow Monitoring Throughput. 5.2.4 Procedure The measurement procedure is same as the Throughput measurement in the section 26.1 of [RFC2544] for the traffic sending side. The output analysis is not done on the DUT receiving side but the Novak Expires January 1, 2010 [Page 19] Internet-Draft Flow Monitoring Benchmarking March 2009 analysed quantity for the measurement of the maximum rate value is the Flow Record count as derived from the Flow Export data captured at the Collector as specified in the section 4.4. 6. RFC2544 Measurements Objective: Provide RFC2544 network device characteristics in the presence of Flow Monitoring on the DUT. The RFC2544 studies numerous characteristics of network devices. The purely forwarding characteristics without Flow Monitoring present on the DUT can significantly vary when Flow Monitoring starts to be deployed on the network device. The objective of the section is to a define controlled environment for Flow Monitoring during all the tests with all the parameters as specified below included in the test report. Metric definition: Metric as specified in [RFC2544] 6.1 Flow Monitoring Configuration Flow Monitoring Configuration (as detailed in the section 4.3) needs to be applied the same way as discussed in the section 5 depending on the desired Cache test mode. RFC2544 measurements which do not involve Throughput measurement can be set to be measured with fixed Flow Expiration Rate or at the value corresponding to the Flow Monitoring Throughput. RFC2544 Throughput measurement with Flow Monitoring enabled represents two dimensional measurement and it depends on the DUT capabilities and the test set-up if the maximum of both variables can be found at the same time as specified below. 6.2 Single Traffic Component Section 12 of [RFC2544] discusses the use of protocol source and destination addresses for defined tests. To perform all the RFC2544 type measurements with Flow Monitoring enabled the defined Flow Keys SHOULD contain IP source and destination address. The RFC2544 type measurements can be executed under these additional conditions: a. the test traffic does not use just single unique pair of source and destination address b. the traffic generator defines test traffic as follows 1) define the test exactly as specified in the section 5.1 or 5.2 Novak Expires January 1, 2010 [Page 20] Internet-Draft Flow Monitoring Benchmarking July 2009 2) allow for a parameter to say send N (where N is an integer number starting at 1 and incremented in small steps) packets with IP addresses A and B before changing both IP addresses to the next value This test traffic definition allows execution of the above defined IP Flow Monitoring tests with one defined Flow Expiration Rate while measuring the DUT RFC2544 characteristics. This set-up is the better option since it best simulates the live network traffic scenario with Flows containing more than just one packet. The initial packet rate at N equal to 1 defines the Flow Expiration Rate for the whole measurement procedure. The consequent increases of N do not change it, the time and Cache characteristics of the test traffic stay the same. The initial rate needs to be small enough to allow for the test traffic rates granular enough while allowing large enough value of Flow Expiration Rate for the measurement. The best procedure is to measure Flow Monitoring Throughput and RFC2544 Throughput independently first and then design the rates for the mutual test. 6.3 Two Traffic Components The test traffic set-up in the section 6.2 might be difficult to achieve with commercial traffic generators. The way around it is to define two traffic components in the test traffic - one to populate Flow Monitoring Cache and the second one to execute the RFC2544 tests. Flow Monitoring Test Traffic Component - the exact traffic definition as specified in the sections 5.1 and 5.2. RFC2544 Test Traffic Component - test traffic as specified by [RFC2544] but under the condition it MUST create just one Flow Record in the DUT Cache. In the particular set-up discussed here this would mean a traffic stream with just one pair of unique source and destination IP addresses (but could be avoided if Flow Keys were for example UDP/TCP source and destination ports and Flow Keys did not contain the addresses). The first traffic component will exercise the DUT in terms of Flow activity while the second traffic component will measure the RFC2544 characteristics. The traffic rates to be reported as Throughput are the sum of rates of both components. The RFC2544 metrics do not need any change. 7. Flow Monitoring Accuracy The pure Flow Monitoring tests in section 5 provide the capability to verify the Flow Monitoring accuracy in terms of the exported Flow Record data. Since every Flow Record created in the Cache is populated by just one packet, the full set of captured data on the Collector can be parsed (e.g. providing the exported Flow Record Novak Expires January 1, 2010 [Page 21] Internet-Draft Flow Monitoring Benchmarking July 2009 contents not only the Flow Record count) and each set of parameters from each Flow Record can be checked against the parameters as configured on the traffic generator and set in packets sent to the DUT. The exported Flow Record is considered accurate if: 1. all the Flow Record fields are present in each exported Flow Record 2. all the Flow Record fields values match the value ranges as set by the traffic generator (for example an IP address falls within the range of the IP addresses increments on the traffic generator) 3. all the possible Flow Record fields values as defined at the traffic generator have been found in the captured export data on the Collector. This check needs to be offset to potential detected packet loses during the test If Packet Sampling is deployed then only verifications in point 1 and 2 can be performed. This verification can be eased by collecting some Flow Monitoring statistics from the DUT. 8. Evaluating Flow Monitoring Applicability The test results as discussed in this document and obtained for certain DUTs allow for a preliminary analysis of a Flow Monitoring deployment based on Internet traffic analysis data provided by organisations like [CAIDA] or performed on the network in question. The data needed to make an estimate if a certain network device can manage the particular amount of live traffic with Flow Monitoring enabled is: Average packet size: 350 bytes Number of packets per IP Flow: 20 Expected data rate on the network device: 1 Gbit/s This results in: Expected packet rate: 357 000 pps being (1 Gbit/s divided by 350 bytes/packet) Flows per second: 18 000 being (packet rate 357 000 pps divided by 20 packets per IP Flow) Under constant load with the parameters above the network device will always run in the Cache Overflow mode (if the network is large enough the Flows will hardly ever repeat within the few seconds needed to fill up lets say 300 000 entries Cache (if the Flow Keys contain parameters like IP addresses or transport level protocols and ports). Novak Expires January 1, 2010 [Page 22] Internet-Draft Flow Monitoring Benchmarking July 2009 It needs to be kept in mind that the above is a very rough Flow activity estimate which cannot account for traffic anomalies like large number of for example DNS request packets which are typically small packets coming from many different sources and represent always just one packet Flow. 9. Acknowledgements This work could have been performed thanks to the patience and support of Cisco System Netflow development team, namely Paul Aitken, Paul Atkins and Andrew Johnson. Thanks belong also to Aamer Akhter for initiating this work and to Benoit Claise for careful reviews and presentations of the draft. 10. IANA Considerations This document requires no IANA considerations. 11. Security Considerations Documents of this type do not directly affect the security of the Internet or corporate networks as long as benchmarking is not performed on devices or systems connected to operating networks. Benchmarking activities as described in this memo are limited to technology characterization using controlled stimuli in a laboratory environment, with dedicated address space and the constraints specified in the sections above. The benchmarking network topology will be an independent test setup and MUST NOT be connected to devices that may forward the test traffic into a production network, or misroute traffic to the test management network. Further, benchmarking is performed on a "black-box" basis, relying solely on measurements observable external to the DUT. Special capabilities SHOULD NOT exist in the DUT specifically for benchmarking purposes. Any implications for network security arising from the DUT SHOULD be identical in the lab and in production networks. 12. References 12.1. Normative References [RFC2119] Bradner, S., "Key words for use in RFCs to Indicate Requirement Levels", BCP 14, RFC 2119, March 1997 [RFC2544] Bradner, S., "Benchmarking Methodology for Network Interconnect Devices", Informational, RFC 2544, March 1999 Novak Expires January 1, 2010 [Page 23] Internet-Draft Flow Monitoring Benchmarking July 2009 [RFC5470] Sadasivan, G., Brownlee, N., Claise, B., and J. Quittek, "Architecture Model for IP Flow Information Export", RFC 5470, July 2009 12.2. Informative References [RFC1242] Bradner, S., "Benchmarking Terminology for Network Interconnection Devices", RFC 1242, July 1991 [RFC2285] Mandeville R., "Benchmarking Terminology for LAN Switching Devices", Informational, RFC 2285, November 1998 [RFC3031] E. Rosen, A. Viswanathan, R. Callon, Multiprotocol Label Switching Architecture, Standards Track, RFC 3031, January 2001 [RFC5180] C. Popoviciu, A. Hamza, D. Dugatkin, G. Van de Velde, "IPv6 Benchmarking Methodology for Network Interconnect Devices, Informational, RFC 5180, May 2008 [RFC3917] Quittek j., "Requirements for IP Flow Information Export (IPFIX)", Informational, RFC 3917, October 2004. [RFC5101] Claise B., "Specification of the IP Flow Information Export (IPFIX) Protocol for the Exchange of IP Traffic Flow Information", Standards Track, RFC 5101, January 2008 [RFC5102] Quittek, J., Bryant, S., Claise, B., Aitken, P., and J. Meyer, "Information Model for IP Flow Information Export", RFC 5102, January 2008 [RFC5472] Zseby, T., Boschi, E., Brownlee, N., Claise, B., "IP Flow Information Export (IPFIX) Applicability", RFC 5472, July 2009 [RFC5477] T. Dietz, F. Dressler, G. Carle, B. Claise, "Information Model for Packet Sampling Exports", RFC 5477, July 2009 [RFC5476] Claise, B., Quittek, J., and A. Johnson, "Packet Sampling (PSAMP) Protocol Specifications", RFC 5476, July 2009 [RFC5474] D. Chiou, B. Claise, N. Duffield, A. Greenberg, M. Grossglauser, P. Marimuthu, J. Rexford, G. Sadasivan, "A Framework for Passive Packet Measurement" RFC 5474, July 2009 [RFC5475] T. Zseby, M. Molina, N. Duffield, F. Raspall, Sampling and Filtering Techniques for IP Packet Selection RFC 5475, July 2009 Novak Expires January 1, 2010 [Page 24] Internet-Draft Flow Monitoring Benchmarking July 2009 [PSAMP-MIB] Dietz, T., Claise, B. "Definitions of Managed Objects for Packet Sampling", Internet-Draft work in progress, June 2006 [MPLS] Akhter A. "MPLS Forwarding Benchmarking Methodology", Internet-Draft work in progress, November 2008 [CAIDA] Claffy, K., "The nature of the beast: recent traffic measurements from an Internet backbone", http://www.caida.org/publications/papers/1998/Inet98/ Inet98.html Author's Addresses Jan Novak (editor) Cisco System Edinburgh, United Kingdom Phone: +44 7740 925889 Email: janovak@cisco.com Benoit Claise Cisco Systems De Kleetlaan 6a b1 1831 Diegem Belgium Phone: +32 2 704 5622 EMail: bclaise@cisco.com Novak Expires January 1, 2010 [Page 25] Internet-Draft Flow Monitoring Benchmarking July 2009