Internet Draft Expiration: March 2003 B. Claise Document: draft-claise-netflow-9-01.txt Cisco Systems Category: Informational October 2002 Cisco Systems NetFlow Services Export Version 9 Status of this Memo This document is an Internet-Draft and is in full conformance with all provisions of Section 10 of RFC2026. Internet-Drafts are working documents of the Internet Engineering Task Force (IETF), its areas, and its working groups. Note that other groups may also distribute working documents as Internet- Drafts. Internet-Drafts are draft documents valid for a maximum of six months and may be updated, replaced, or obsolete by other documents at any time. It is inappropriate to use Internet-Drafts as reference material or to cite them other than as "work in progress." The list of current Internet-Drafts can be accessed at http://www.ietf.org/ietf/1id-abstracts.txt The list of Internet-Draft Shadow Directories can be accessed at http://www.ietf.org/shadow.html. Abstract This document discusses the Cisco Systems NetFlow services that provide network administrators with access to IP flows information. The NetFlow services create flow records that are then exported to a NetFlow collector. The exported flow records can be used for a variety of purposes including network management and planning, accounting, departmental chargebacks, Internet service provider (ISP) billing, data warehousing, data mining for marketing purposes, etc. This document focuses on the most recent evolution of the NetFlow flow record export format, which is known as version 9. The distinguishing feature of the NetFlow version 9 export format compared with previous formats, is that it is template based. The templates (collections of fields along with the description and structure) provide a flexible and extensible design to the flow- record export format. This facilitates future enhancements to NetFlow services without requiring changes to the basic flow record export Claise Expires û March 2003 [Page 1] Cisco Systems NetFlow Services Export Version 9 October 2002 format. Another advantage is that only the required fields are exported within the flow record, which minimizes the consumed export bandwidth. Conventions used in this document The key words "MUST", "MUST NOT", "REQUIRED", "SHALL", "SHALL NOT", "SHOULD", "SHOULD NOT", "RECOMMENDED", "MAY", and "OPTIONAL" in this document are to be interpreted as described in RFC 2119. Table of Contents 1. Introduction...................................................2 1.1 Overview...................................................2 1.2 Applications...............................................3 2. Terminology....................................................5 3. NetFlow High-Level Picture on the Exporter.....................8 3.1 The NetFlow Process on the Exporter........................8 3.2 Flow Expiration............................................8 3.3 Transport Protocol.........................................8 4. Packet Layout..................................................9 5. Export Packet Format..........................................10 5.1 Header Format.............................................10 5.2 Template FlowSet Format...................................11 5.3 Data FlowSet Format.......................................13 6. Options.......................................................14 6.1 Options Template FlowSet..................................14 6.2 Options Data FlowSet......................................16 7. Template Management...........................................18 8. Field Type Definitions........................................19 9. The Collector's Side..........................................21 10. Examples.....................................................22 10.1 Packet Header Example....................................23 10.2 Template FlowSet Example.................................23 10.3 Data FlowSet Example.....................................24 10.4 Option Template FlowSet Example..........................25 10.5 Option Data FlowSet Example..............................25 11. References...................................................26 12. Authors......................................................26 13. Acknowledgments..............................................26 1. Introduction 1.1 Overview NetFlow services data can be used for a variety of purposes. A non-exhaustive list is available in the next section. This paper discusses the most recent evolution of the NetFlow flow-record Claise Informational [Page 2] Cisco Systems NetFlow Services Export Version 9 October 2002 format, which is known as version 9. The distinguishing feature of the NetFlow version 9 format, compared with previous formats, is that it is template based. A template is a collection of fields with the corresponding description of their structure and semantics. This approach gives the following advantages: - Because the template mechanism is flexible, it allows the export of the required fields alone from the IP Flows to the NetFlow collector. This helps to reduce the exported flow data volume and possible memory savings at exporter and collector. Network load can also be reduced by sending only the required information. - Using the template mechanism, new fields can be added to NetFlow flow records without changing the structure of the export record format. With previous NetFlow versions, adding a new field in the flow record implies a new version of the export protocol format and a new version of the NetFlow collector that supports the parsing of this new export protocol format. - Templates that are sent to the collector contain the structural information about the exported flow records fields. Therefore, if the collector does not understand the semantics of new fields, it can still interpret the flow record. 1.2 Applications NetFlow services data enables several key customer applications: Accounting and Billing NetFlow services data provides fine-grained metering (for example, flow records include such details as IP addresses, packet and byte counts, timestamps, Type of Service (TOS), application ports, etc.) for highly flexible and detailed resource usage accounting. ISPs can use this information to migrate from single fee, flat-rate billing to more flexible charging mechanisms based on time of day, bandwidth usage, application usage, quality of service, etc. Enterprise customers can use this information for departmental chargeback or cost allocation for resource usage. Network Planning NetFlow services data captured over a long period of time allows the possibility to track and anticipate network growth and plan Claise Informational [Page 3] Cisco Systems NetFlow Services Export Version 9 October 2002 upgrades to increase the number of routing devices, ports, or higher-bandwidth interfaces. NetFlow services data optimizes both strategic network planning (peering, backbone upgrade planning, and routing policy planning) as well as tactical network engineering decisions (upgrading the router/link capacity). This helps to minimize the total cost of network operations while maximizing network performance, capacity, and reliability. Peering Agreements NetFlow services data enables ISP peering partners to measure the volume and characteristics of traffic exchanged with other ISP peers. Traffic Engineering NetFlow services data provides traffic engineering details for a set of prefixes, that can be used in network optimization for load balancing traffic across alternate paths or by forwarding traffic of a certain set of prefixes on a preferred route. Network Monitoring NetFlow services data enables extensive near real-time network monitoring capabilities. NetFlow services data analysis can be used to display traffic patterns associated with routing devices and switches on an individual, or network-wide basis. This can display traffic or application-based views and therefore provide proactive problem detection, efficient troubleshooting, and rapid problem resolution. Application Monitoring and Profiling NetFlow services data enables content and service providers to view detailed, time-based and application-based usage of a network. This information allows planning and allocation of network and application resources (such as Web server, gaming, or multimedia). User Monitoring and Profiling NetFlow services data provides a detailed understanding of customer or end-user usage of network and application resources. This information can then be used to efficiently plan and allocate access, backbone and application resources as well as to detect and resolve potential security and policy violations. Security Analysis Claise Informational [Page 4] Cisco Systems NetFlow Services Export Version 9 October 2002 NetFlow services data provides details on source and destination addresses, along with the start time of Flows and application ports. This data can be used to analyze network security and identify attacks. NetFlow Data Warehousing and Mining NetFlow services data (or derived information) can be stored for later retrieval and analysis to support proactive marketing and customer service programs. An example of this would be to determine which applications and services are being used by internal and external users and then target them for improved service such as advertising and so on. This is especially useful for ISPs because NetFlow Services data enables them to create better service packaging. 2. Terminology Various terms used in this document are described below: IP Flow or Flow A Flow is defined as a set of IP packets passing an Observation Point in the network during a certain time interval. All packets that belong to a particular Flow have a set of common properties derived from the data contained in the packet and from the packet treatment at the Observation Point. Flow Record A Flow Record provides information about an IP Flow that exists on the Exporter. The Flow Records are also referred to as NetFlow services data or NetFlow data in this document. Exporter A device (for example, a router) with NetFlow services enabled. The Exporter monitors packets entering an Observation Point and creates Flows out of these packets. The information from these Flows are exported in the form of Flow Records to the Collector. NetFlow Collector The NetFlow Collector receives Flow Records from one or more Exporters. It processes the received Export Packet, i.e. parses, stores the Flow Record information. Flow records can be optionally aggregated before being stored on the hard disk. The NetFlow Collector is also referred to as the Collector in this Claise Informational [Page 5] Cisco Systems NetFlow Services Export Version 9 October 2002 document. Observation Point A location in the network where IP packets can be observed. For example, one or a set of interfaces of the Exporter. An Observation Domain is associated with every Observation Point. Observation Domain: The set of Observation Points, which is the largest aggregatable set of Flow information at the Exporter is termed an Observation Domain. Each Observation Domain presents itself a unique ID to the Collector for identifying the Export Packets it generates. For example, a router line card, composed of several interfaces with each interface being an Observation Point. Export Packet A packet originating at the Exporter, which carries the Flow Records of the Exporter and whose destination is the NetFlow Collector. Export Packet: +--------+------------------------------------------------------+ | Packet | +-----------------+ +------------------+ +---------+ | | Header | | FlowSet | | FlowSet | | FlowSet | | | | +-----------------+ +------------------+ +---------+ | +--------+------------------------------------------------------+ Packet Header The first part of an Export Packet, which provides basic information about the packet such as the NetFlow version, number of records contained within the packet, sequence numbering, etc. FlowSet FlowSet is a generic term for a collection of records that have similar structure. In an Export Packet, one or more FlowSets follow the Packet Header. There are three different types of FlowSets: Template FlowSet, Data FlowSet and Option FlowSet. An Export Packet contains one or more FlowSets, and the three FlowSet types can be mixed within the same Export Packet. Template Record A Template Record is used to define the structure and interpretation Claise Informational [Page 6] Cisco Systems NetFlow Services Export Version 9 October 2002 of fields in a data record. Data records that correspond to a template MAY appear in the same and/or subsequent Export Packets. The template information is not necessarily carried in every Export Packet. As such, the Collector MUST store the "Template Record" in order to interpret the corresponding data records that are received in subsequent data packets. Template FlowSet A Template FlowSet is a collection of one or more Template Records that have been grouped together in an Export Packet. Template ID A unique number that distinguishes a Template Record from all other Template Records produced by the same Observation Domain. A NetFlow Collector that receives Export Packets from several Observations Domains from the same Exporter MUST be aware that uniqueness of Template ID is not guaranteed across Observation Domains. For this reason, the NetFlow Collector MUST store the address of the Exporter that produced the Template ID, along with the Observation Domain, in order to enforce uniqueness. Data FlowSet A Data FlowSet is a collection of one or more Flow Records that are grouped together in an Export Packet. A Data FlowSet contains records that belong to the same Template ID. Each Data FlowSet references a previously transmitted Template ID, which can be used to parse the data contained within the Flow Records. Options FlowSet An Options FlowSet is a collection of one or more Options Templates that have been grouped together in an Export Packet. Options Template A template that describes the format of the Flow measurement parameters (for example, the sampling algorithm used, sampling interval) done at the Exporter. Option Templates are identified by a well-known Template ID. Options Data Record The data record that contains values of the Flow measurement parameters corresponding to an Option Template. FlowSet ID Claise Informational [Page 7] Cisco Systems NetFlow Services Export Version 9 October 2002 An ID used to distinguish the different FlowSets. FlowSet IDs between 0 and 255 are reserved. Template FlowSet and Option Template FlowSet use fixed FlowSet ID of 0 and 1, respectively. Data FlowSets have a FlowSet ID greater than 255. 3. NetFlow High-Level Picture on the Exporter 3.1 The NetFlow Process on the Exporter The description of the NetFlow process (for example, sampled NetFlow, full NetFlow, aggregation), that is, the way in which Flows are created from the observed IP packets is beyond the scope of this document. 3.2 Flow Expiration A Flow is considered to be inactive if no packets belonging to this Flow have been observed at the Observation Point for a given timeout interval. A Flow can be exported under the following conditions: 1. If the Exporter can detect the end of a Flow, it SHOULD export the Flow Records at the end of the Flow. For example, a Flow generated by TCP [3] type of traffic where the FIN or RST bits indicate the end of the Flow. 2. If the Flow has been inactive for a certain period of time. This inactivity timeout SHOULD be configurable, with a minimum value of 0 for a immediate expiration. For example, a Flow generated by UDP [2] type of traffic. 3. For long-lasting Flows, the Exporter SHOULD export the Flow Records on a regular basis. This periodicity SHOULD be Configurable. 4. If the Exporter experiences internal constraints, a Flow MAY be forced to expire prematurely (for example, counters wrapping or low memory). 3.3 Transport Protocol To achieve efficiency in terms of processing at the Exporter while handling high volumes of Export Packet, the NetFlow Export Packet is encapsulated into UDP [2] datagrams for export to the NetFlow Claise Informational [Page 8] Cisco Systems NetFlow Services Export Version 9 October 2002 Collector. However, NetFlow version 9 has been designed to be transport protocol independent. Hence, it can also operate over congestion-aware protocols such as TCP [3] or SCTP [4]. Note that the Exporter can export to multiple Collectors, using independent transport protocols. 4. Packet Layout An Export Packet consists of a Packet Header followed by one or More FlowSets. The FlowSets can be any of the possible three types: Template, Data, or Option. Export Packet: +--------+------------------------------------------+ | | +----------+ +---------+ +---------+ | | Packet | | Template | | Data | | Option | | | Header | | FlowSet | | FlowSet | | FlowSet | ... | | | +----------+ +---------+ +---------+ | +--------+------------------------------------------+ The possible combinations that can occur in an Export Packet are: - An Export Packet that consists of interleaved Template, Data, and Options FlowSets. Export Packet: +--------+-------------------------------------------------------+ | | +----------+ +---------+ +----------+ +---------+ | | Packet | | Template | | Data | ... | Options | | Data | | | Header | | FlowSet | | FlowSet | ... | FlowSet | | FlowSet | | | | +----------+ +---------+ +----------+ +---------+ | +--------+-------------------------------------------------------+ - An Export Packet consisting entirely of Data FlowSets. Once the appropriate Template IDs have been defined and transmitted to the NetFlow Collector device, the majority of Export Packets will consist solely of Data FlowSets. Export Packet: +--------+----------------------------------------------+ | | +---------+ +---------+ +---------+ | | Packet | | Data | ... | Data | ... | Data | | Claise Informational [Page 9] Cisco Systems NetFlow Services Export Version 9 October 2002 | Header | | FlowSet | ... | FlowSet | ... | FlowSet | | | | +---------+ +---------+ +---------+ | +--------+----------------------------------------------+ - An Export Packet consisting entirely of Template and Options FlowSets. The Exporter MAY transmit a packet containing Template FlowSets, ahead of time to help ensure that the NetFlow Collector has the correct template information before receiving the first data FlowSet. Export Packet: +--------+------------------------------------------------+ | | +----------+ +----------+ +---------+ | | Packet | | Template | ... | Template | ... | Options | | | Header | | FlowSet | ... | FlowSet | ... | FlowSet | | | | +----------+ +----------+ +---------+ | +--------+------------------------------------------------+ A Template FlowSet provides a description of the fields that would be present in future Data FlowSets. These Data FlowSets MAY occur later within the same Export Packet or in subsequent Export Packets. The format of the Template, Data, and Options FlowSets will be discussed later in this document. 5. Export Packet Format 5.1 Header Format Note that the Packet Header format has been kept similar to the one developed by the different versions of NetFlow defined by Cisco Systems, for backward compatibility. 0 1 2 3 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ | Version Number | Count | +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ | sysUpTime | +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ | UNIX Secs | +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ | Sequence Number | Claise Informational [Page 10] Cisco Systems NetFlow Services Export Version 9 October 2002 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ | Source ID | +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ Packet Header Field Descriptions Version Version of Flow Record format exported in this packet. The value of this field is 0x0009 for the current version. Count Count is the total number of record(s) in the Export Packet, which is the sum total of Option FlowSet record(s), Template FlowSet record(s) and Data FlowSet record(s). SysUpTime Time in milliseconds since this device was first booted. Refer to [1]. Unix Secs Seconds since 0000 UTC 1970. Sequence Number Incremental sequence counter of all Export Packets sent from the current Observation Domain by the Exporter. This value will be cumulative, and can be used to identify whether any Export Packets have been missed. Source ID The Source ID field is a 32-bit value that identifies the Observation Domain. NetFlow Collectors SHOULD use the combination of the source IP address and the Source ID field to separate different export streams originating from the same Exporter. 5.2 Template FlowSet Format One of the key elements in the NetFlow format is the Template FlowSet. Templates greatly enhance the flexibility of the Flow Record format, because they allow the NetFlow Collector to process Flow Records without necessarily knowing the interpretation of all Claise Informational [Page 11] Cisco Systems NetFlow Services Export Version 9 October 2002 the data in the Flow Record. The format of the Template FlowSet is as follows: 0 1 2 3 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ | FlowSet ID = 0 | Length | +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ | Template ID 1 | Field Count | +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ | Field Type 1 | Field Length 1 | +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ | Field Type 2 | Field Length 2 | +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ | ... | ... | +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ | Field Type N | Field Length N | +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ | Template ID 2 | Field Count | +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ | Field Type 1 | Field Length 1 | +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ | Field Type 2 | Field Length 2 | +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ | ... | ... | +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ | Field Type M | Field Length M | +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ Template FlowSet Field Descriptions FlowSet ID FlowSet ID value of 0 is reserved for Template FlowSet. Length Total length of this FlowSet. Because an individual Template FlowSet MAY contain multiple Template Records, the Length value MUST be used to determine the position of the next FlowSet Record, which could be any type of FlowSet. Length is the sum total of lengths of FlowSet ID, the Length itself, and all Template Records within this FlowSet Template ID. Claise Informational [Page 12] Cisco Systems NetFlow Services Export Version 9 October 2002 Template ID Each of the newly generated Template Records is given a unique Template ID. This uniqueness is local to the Observation Domain that generated the Template ID. Template IDs 0-255 are reserved for Template FlowSets, Option Flowsets and other reserved FlowSet that would be created in the future. Template IDs of Data FlowSets are numbered from 256 up to 65535. Field Count Number of fields in this Template Record. Because a Template FlowSet usually contains multiple Template Records, this field allows the Collector to determine the end of the current Template Record and the start of the next. Field Type A numeric value that represents the type of the field. Refer to the section on ôField Type Definitionsö. Field Length The length of the corresponding Field Type, in bytes. Refer to the section on ôField Type Definitionsö. 5.3 Data FlowSet Format The format of the Data FlowSet is as follows: 0 1 2 3 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ | FlowSet ID = Template ID | Length | +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ | Record 1 - Field Value 1 | Record 1 - Field Value 2 | +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ | Record 1 - Field Value 3 | ... | +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ | Record 2 - Field Value 1 | Record 2 - Field Value 2 | +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ | Record 2 - Field Value 3 | ... | +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ | Record 3 - Field Value 1 | ... | +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ Claise Informational [Page 13] Cisco Systems NetFlow Services Export Version 9 October 2002 | ... | Padding | +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ Data FlowSet Field Descriptions FlowSet ID = Template ID Each Data FlowSet is associated with a FlowSet ID. The FlowSet ID maps to a (previously generated) Template ID. The Collector MUST use the FlowSet ID to find the corresponding Template Record and decode the Flow Records from the FlowSet. Length The length of this FlowSet. Length is the sum total of lengths of FlowSet ID, Length itself, all Flow Records within this FlowSet, Template ID, and the padding bytes if any. Record N - Field Value N The remainder of the Data FlowSet is a collection of Flow Records each containing a set of field types and values. The Type and Length of the fields have been previously defined in the Template Record referenced by the FlowSet ID or Template ID. Padding Padding SHOULD be inserted so that subsequent FlowSet starts at 4 byte aligned boundary. It is important to note that the Length field includes the padding bits. Interpretation of the Data FlowSet format can be done only if the Template FlowSet corresponding to the Template ID is available at the Collector. 6. Options 6.1 Options Template FlowSet The Options Template (and its corresponding Options Data Record) is used to supply information about the NetFlow process configuration or NetFlow process specific data, rather than supplying information about IP Flows. For example, the sample rate of a specific interface, if sampling is supported, along with the sampling method used. Claise Informational [Page 14] Cisco Systems NetFlow Services Export Version 9 October 2002 The format of the Options Template FlowSet is detailed below: 0 1 2 3 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ | FlowSet ID = 1 | Length | +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ | Template ID | Option Scope Length | +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ | Option Length | Scope 1 Field Type | +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ | Scope 1 Field Length | ... | +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ | Scope N Field Length | Option 1 Field Type | +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ | Option 1 Field Length | ... | +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ | Option N Field Length | Padding | +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ Options Template Field Definitions FlowSet ID = 1 A FlowSet ID value of 1 is reserved for Option Template. Length Total length of this FlowSet. Each Options Templates MAY contain multiple Template IDs. Thus, the Length value MUST be used to determine the position of the next FlowSet record, which could be either a Template FlowSet or Data FlowSet. Length is the sum total of lengths of FlowSet ID, the Length itself, and all Template Records within this FlowSet Template ID. Template ID Template ID of this Option Template. This value is greater than 255. Option Scope Length The length in bytes of any Scope fields contained in the Options Template (The use of "Scope" is described below). Claise Informational [Page 15] Cisco Systems NetFlow Services Export Version 9 October 2002 Options Length The length (in bytes) of any options field definitions contained in this Options Template. Scope 1 Field Type The relevant portion of the Exporter/NetFlow process to which the Options Record refers. Currently defined values are: 0x0001 System 0x0002 Interface 0x0003 Line Card 0x0004 Cache 0x0005 Template For example, the NetFlow process can be implemented on a per-interface basis, so if the options record were reporting on how the NetFlow process is configured, the SCOPE for the report would be 0x0002 (Interface). The associated interface ID would then be carried in the associated Option Data FlowSet. Scope 1 Field Length The length (in bytes) of the scope field, as it would appear in an Options Record. Option 1 Field Type A numeric value that represents the type of the field that would appear in the Options record. Refer to the Field Type Definitions section. Option 1 Field Length The length (in bytes) of the Scope field. Padding Padding SHOULD be inserted so that subsequent FlowSet starts at a 4 bytes aligned boundary. It is important to note that the Length field includes the padding bits. 6.2 Options Data FlowSet The Option Data Records are sent in Data FlowSets, on a regular basis, but not with every Flow Record. How frequently these Option Data Records are exported is configurable. See the ôTemplates Claise Informational [Page 16] Cisco Systems NetFlow Services Export Version 9 October 2002 Managementö section for more details. The Options Data format is as follows: 0 1 2 3 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ | FlowSet ID = Template ID | Length | +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ | Record 1 - Scope 1 Value |Record 1 - Option Field 1 Value| +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ |Record 1 - Option Field 2 Value| ... | +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ | Record 2 - Scope 1 Value |Record 2 - Option Field 1 Value| +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ |Record 2 - Option Field 2 Value| ... | +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ | Record 3 - Scope 1 Value |Record 3 - Option Field 1 Value| +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ |Record 3 - Option Field 2 Value| ... | +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ | ... | Padding | +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ Options Data FlowSet Field Descriptions FlowSet ID = Template ID Each group of Options Data Records within an Option Data FlowSet is preceded by a FlowSet ID. The FlowSet ID maps to a (previously generated) Template ID corresponding to this Options Template. The Collector MUST use the FlowSet ID to map the appropriate type and length to any field values that follow. Length The length of this FlowSet. Length is the sum total of lengths of FlowSet ID, Length itself, all the Option Data Records within this FlowSet, and the padding bytes if any. Record N - Option Field N Value The remainder of the Option Data FlowSet is a collection of Flow Records each containing a set of field types and Claise Informational [Page 17] Cisco Systems NetFlow Services Export Version 9 October 2002 values. The Type and Length of the fields have been previously defined in the Option Template Record referenced by the FlowSet ID or Template ID. Padding Padding SHOULD be inserted so that subsequent FlowSet starts at 4 byte aligned boundary. It is important to note that the Length field includes the padding bits. Interpretation of the Options Data FlowSet format can be done only if the Options Template FlowSet corresponding the Template ID is available at the Collector. 7. Template Management The Template IDs must remain constant for the life of the NetFlow process and the Exporter. If the Exporter or the NetFlow process restarts due to some reason, all information about Templates would be lost. New Template IDs would be created. Template IDs are thus not guaranteed to be consistent across an Exporter or NetFlow process restart. A newly created template is assigned an unused Template ID from the Exporter. If the template configuration is changed, then the current Template ID is abandoned and not reused anymore until the NetFlow process or Exporter restarts. If a configured template on the Exporter is deleted, and re- configured with exactly the same parameters, the same Template ID COULD be reused. The Exporter sends the Template FlowSet and Option Template FlowSet under the following conditions: 1. After a NetFlow process restart, the Exporter MUST NOT send any Data FlowSet without having the corresponding Template FlowSet and the required Option Template FlowSet sent out in a previous packet or in the same export packet. It MAY transmit the Template FlowSet and Option Template FlowSet, without any Data FlowSets, ahead of time to help ensuring that the Collector will have the correct template information before receiving the first data. Claise Informational [Page 18] Cisco Systems NetFlow Services Export Version 9 October 2002 2. In the event of a configuration changes, the Exporter SHOULD send the incremental changes at an accelerated rate. In such a case, it MAY transmit the changed Template FlowSet and Option Template FlowSet, without any data, ahead of time to help ensure that the Collector will have the correct template information before receiving the first data. 3. On a regular basis, the Exporter MUST send all the Template FlowSets and Options Template FlowSets to refresh the Collector. Template IDs have a limited lifetime at the Collector and MUST be periodically refreshed. Two approaches are taken to make sure that Templates get refreshed at the Collector: * every N number of Export Packets. * on regular basis, so every N number of minutes. Both options MUST be user configurable. When one of these expiry condition is met, the Exporter MUST send the Template FlowSet and Option Template. 8. Field Type Definitions The following table describes all the field type definitions that an Exporter MAY support. The fields are a selection of Packet Header fields, lookup results (for example the AS numbers or the subnet masks), properties of the packet itself such as length. Field Type Value Length Description (bytes) counter with length IN_BYTES 1 N N x 8 bits for bytes associated with an IP Flow counter with length IN_PKTS 2 N N x 8 bits for packets associated with an IP Flow FLOWS 3 4 Number of Flows that were aggregated PROT 4 1 IP protocol byte TOS 5 1 Type of service byte Claise Informational [Page 19] Cisco Systems NetFlow Services Export Version 9 October 2002 TCP_FLAGS 6 1 TCP Flags (cumulative OR of TCP flags) TCP/UDP source port number L4_SRC_PORT 7 2 (e.g., FTP, Telnet, etc... ,or equivalent) IP_SRC_ADDR 8 N Source IP Address IPv4 have N=4 IPv6 have N=16 SRC_MASK 9 1 source route mask bits INPUT_SNMP 10 2 Input interface index TCP/UDP destination port L4_DST_PORT 11 2 number (e.g., FTP, Telnet, etc... ,or equivalent) IP_DST_ADDR 12 N Destination IP Address IPv4 have N=4 IPv6 have N=16 DST_MASK 13 1 destination route mask bits OUTPUT_SNMP 14 2 Output interface index IP_NEXT_HOP 15 N Next hop router's IP address IPv4 have N=4 IPv6 have N=16 SRC_AS 16 4 Source BGP Autonomous System number DST_AS 17 4 Destination BGP Autonomous System number BGP_NEXT_HOP 18 N Next-hop router's IP in the BGP domain IPv4 have N=4 IPv6 have N=16 Claise Informational [Page 20] Cisco Systems NetFlow Services Export Version 9 October 2002 MUL_DPKTS 19 4 Packet count for IP multicast MUL_DOCTETS 20 4 Octet (byte) count for IP multicast SysUptime at which the LAST_SWITCHED 21 4 last packet of this Flow was switched SysUptime at which the FIRST_SWITCHED 22 4 first packet of this Flow was switched PKTS 24 8 64-bit counter for packets associated with an IP Flow TOTAL_BYTES_EXP 40 4 Number of Bytes exported by the Observation Domain TOTAL_EXP_PKTS_SENT 41 4 Number of Packets exported by the Observation Domain TOTAL_FLOWS_EXP 42 4 Number of Flows exported by the Observation Domain The value field is an numeric identifier for the field type. When extensibility is required, the new field types will be added to the above list. The new field types have to be updated on the Exporter and Collector. However, the NetFlow export format itself would remain unchanged. Refer to the latest documentation at http://www.cisco.com for the newly updated list. 9. The Collector's Side The Collector will receive template definitions from the Exporter, normally before receiving Flow Records. The Flow Records can then be decoded and stored locally on the devices. If the template definitions have not been received at the time a Flow Record is received, the Collector SHOULD store the Flow Record and decode it after the template definition is received. A Collector device MUST Claise Informational [Page 21] Cisco Systems NetFlow Services Export Version 9 October 2002 NOT assume that the Data FlowSet and the associated Template IDs are exported in the same Export Packet. The Collector MUST NOT assume that one and only one Template FlowSet is present in an Export Packet. The life of a template at the Collector is limited to a fixed refresh timeout. Templates not refreshed from the Exporter within the timeout are expired at the Collector. The Collector MUST NOT attempt to decode the Flow Records with an expired Template. At any given time the Collector SHOULD maintain the following for all the current Templates and Options Templates: Note that the Observation Domain is characterized by the Source ID field from the Export Packet. Keep in mind that the Template IDs are unique per Exporter and per Observation Domain. If a new Template definition is received on the Collector (for example in the case of an Exporter restart) it MUST immediately override the existing Template definition. 10. Examples Let's consider the example of an Export Packet composed of a Template FlowSet, of a Data FlowSet (which contains three Flow Records), of one Option Template and of one Option Data FlowSet (which contains 2 Records) Export Packet: +--------+---------------------------------------. . . | | +--------------+ +------------------+ | Packet | | Template | | Data | | Header | | FlowSet | | FlowSet | . . . | | | (1 Template) | | (3 Flow Records) | | | +--------------+ +------------------+ +--------+---------------------------------------. . . . . .+-------------------------------------------+ +------------------+ +------------------+ | | Option | | Option | | Claise Informational [Page 22] Cisco Systems NetFlow Services Export Version 9 October 2002 . . .| Template FlowSet | | Data FlowSet | | | (1 Template) | | (2 Records) | | +------------------+ +------------------+ | . . .-------------------------------------------+ 10.1 Packet Header Example The Packet Header is composed of: 0 1 2 3 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ | Version = 0x0009 | Count = 7 | +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ | sysUpTime | +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ | UNIX Secs | +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ | Sequence Number | +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ | Source ID | +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 10.2 Template FlowSet Example We want to report the following Field Types: - The source IP address (IPv4), so the length is 4 - The destination IP address (IPv4), so the length is 4 - The next-hop IP address (IPv4), so the length is 4 - The number of bytes of the Flow - The number of packets of the Flow Therefore, the Template FlowSet will be composed of the following: 0 1 2 3 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ | FlowSet ID = 0 | Length = 28 bytes | +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ | Template ID 256 | Field Count = 5 | +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ | IP_SRC_ADDR = 0x0008 | Field Length = 4 | Claise Informational [Page 23] Cisco Systems NetFlow Services Export Version 9 October 2002 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ | IP_DST_ADDR = 0x000C | Field Length = 4 | +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ | IP_NEXT_HOP = 0x000F | Field Length = 4 | +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ | IN_PKTS = 0x0002 | Field Length = 4 | +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ | IN_BYTES = 0x0001 | Field Length = 4 | +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 10.3 Data FlowSet Example In this example, we report the following three Flow records: Src IP addr. | Dst IP addr. | Next Hop addr. | Packet | Bytes | | | Number | Number --------------------------------------------------------------- 198.168.1.12 | 10.5.12.254 | 192.168.1.1 | 5009 | 5344385 192.168.1.27 | 10.5.12.23 | 192.168.1.1 | 748 | 388934 192.168.1.56 | 10.5.12.65 | 192.168.1.1 | 5 | 6534 0 1 2 3 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ | FlowSet ID = 256 | Length = 64 | +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ | 198.168.1.12 | +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ | 10.5.12.254 | +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ | 192.168.1.1 | +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ | 5009 | +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ | 5344385 | +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ | 192.168.1.27 | +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ | 10.5.12.23 | +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ | 192.168.1.1 | +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ | 748 | Claise Informational [Page 24] Cisco Systems NetFlow Services Export Version 9 October 2002 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ | 388934 | +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ | 192.168.1.56 | +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ | 10.5.12.65 | +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ | 192.168.1.1 | +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ | 5 | +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ | 6534 | +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ Note that padding was not necessary in this example. 10.4 Option Template FlowSet Example Per line card (the Exporter being composed of two Line Cards), we want to report the following Field Types: - Total number of Export Packets - Total number of exported Flows The format of the Options Template FlowSet is as follows: 0 1 2 3 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ | FlowSet ID = 1 | Length = 24 | +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ | Template ID 257 | Option Scope Length = 4 | +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ | Option Length = 8 | Scope 1 Field Type = 0x0003 | +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ | Scope 1 Field Length = 2 | TOTAL_EXP_PKTS_SENT = 41 | +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ | Field Length = 4 | TOTAL_FLOWS_EXP = 42 | +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ | Field Length = 4 | Padding | +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 10.5 Option Data FlowSet Example Claise Informational [Page 25] Cisco Systems NetFlow Services Export Version 9 October 2002 In this example, we report the following two records: Line Card ID | Export Packet| Export Flow ------------------------------------------ Line Card 1 | 345 | 10201 Line Card 2 | 690 | 20402 0 1 2 3 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ | FlowSet ID = 257 | Length = 14 | +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ | 1 | 345 | +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ | 10201 | 2 | +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ | 2 | 690 | +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ | 20402 | Padding | +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 11. References [1] J. Case et al, "Management Information Base for Version 2 of the Simple Network Management Protocol (SNMPv2)" RFC 1907, January 1996 [2] J. Postel, "User Datagram Protocol" RFC 768, August 1980 [3] "TRANSMISSION CONTROL PROTOCOL DARPA INTERNET PROGRAM PROTOCOL SPECIFICATION", RFC 793, September 1981 [4] R. Stewart et al, "Stream Control Transmission Protocol" RFC 2960, October 2000 12. Authors This document was jointly written by Vamsidhar Valluri , Martin Djernaes , Ganesh Sadasivan gsadasiv@cisco.com and Benoit Claise bclaise@cisco.com. 13. Acknowledgments Claise Informational [Page 26] Cisco Systems NetFlow Services Export Version 9 October 2002 I would like to thank Pritam Shah for his valuable technical feedback. Authors Addresses Benoit Claise Cisco Systems De Kleetlaan 6a b1 1831 Diegem Belgium Phone: +32 2 704 5622 Email: bclaise@cisco.com Ganesh Sadasivan Cisco Systems, Inc. 3750 Cisco Way San Jose, CA 95134 USA Phone: +1 (408) 527-0251 Email: gsadasiv@cisco.com Vamsi Valluri Cisco Systems, Inc. 510 McCarthy Blvd. San Jose, CA 95035 USA Phone: +1 (408) 525-1835 Email: vvalluri@cisco.com Martin Djernaes Cisco Systems, Inc. 510 McCarthy Blvd. San Jose, CA 95035 USA Phone: +1 (408) 853-1676 Email: djernaes@cisco.com Claise Informational [Page 27]