Reducing redundancy in IPFIX and PSAMP reports Network Working Group E. Boschi Internet-Draft Hitachi Europe Expires: August 30, 2006 L. Mark Fraunhofer FOKUS February 2006 Reducing redundancy in IPFIX and PSAMP reports draft-boschi-ipfix-reducing-redundancy-00.txt Status of this Memo By submitting this Internet-Draft, each author represents that any applicable patent or other IPR claims of which he or she is aware have been or will be disclosed, and any of which he or she becomes aware will be disclosed, in accordance with Section 6 of BCP 79. Internet-Drafts are working documents of the Internet Engineering Task Force (IETF), its areas, and its working groups. Note that other groups may also distribute working documents as Internet-Drafts. Internet-Drafts are draft documents valid for a maximum of six months and may be updated, replaced, or obsoleted by other documents at any time. It is inappropriate to use Internet-Drafts as reference material or to cite them other than as "work in progress." The list of current Internet-Drafts can be accessed at http://www.ietf.org/ietf/1id-abstracts.txt. The list of Internet-Draft Shadow Directories can be accessed at http://www.ietf.org/shadow.html. This Internet-Draft will expire on August 30, 2006. Copyright Notice Copyright (C) The Internet Society (2006). Boschi, Mark Expires August 2006 [Page 1] Reducing redundancy in IPFIX and PSAMP reports Abstract This document describes a bandwidth saving methodology for exporting flow or packet information using the IP Flow Information Export (IPFIX) protocol. Being the PSAMP protocol based on IPFIX, these considerations are valid for PSAMP exports as well. The main idea is to separate the export of information common to several flows (or packets) and the specific flow (or packet) information, using two different records. The association between the records is kept using unique Identifiers. Table of Contents 1. Introduction············································2 2. Terminology·············································3 3. OPEN POINTS·············································3 4. Techniques for bandwidth saving information export······3 4.1 Single and multiple associations······················4 4.2 Export of Per-Packet Information······················5 4.3 Using Scopes··········································6 4.4 CommonPropertiesID Management·························7 5. Examples················································7 5.1 Multiple CommonPropertiesIDs··························7 5.2 Per-Packet Information Export························10 6. Export and evaluation considerations···················12 7. IANA Consideration·····································13 8. Security Considerations································13 9. Normative References···································13 10. Author's Addresses·····································13 11. Intellectual Property Statement························13 12. Copyright Statement····································14 13. Disclaimer·············································14 1. Introduction In [IPFIX-PROTO] the IPFIX working group has defined a protocol to export IP flow information. The main purpose of the protocol is to exchange information about IP traffic flows in combination with related measurement data. The export of that information is done via flow records. In this scope a flow is defined by a set of key attributes (e.g. source and destination IP address, source and destination port, etc.). When exporting flow records sharing a set of common attributes, these attributes have to be repeated for each data record even though the values remain the same. The elimination of redundant data from the export stream can result in a significant reduction of the transferred data. This draft proposes to use option data records to export common properties, whose value remains constant, and to send these properties only once. Measured properties vary over time and are sent multiple times in regular data records. The linkage of the flow records with the common properties exported via option data records is done via a CommonPropertiesID. The proposed method can be applied also to export per packet information (e.g. for Passive One-Way Delay measurements or PSAMP exports). In this case a flow is a collection of packets that share a set of common attributes. Boschi, Mark Expires August 2006 [Page 2] Reducing redundancy in IPFIX and PSAMP reports The proposed method does not need any changes to the IPFIX protocol. 2. Terminology Collecting Process The collecting process receives records of flow or packet information. The data is stored for later processing (by the calculation process) Exporting Process The exporting processes send flow and packet records to the collecting processes. The records are generated by the measurement process. Filtering Filtering selects a subset of packets by applying deterministic functions on parts of the packet content like header fields or parts of the payload. A filtering process needs to process the packet (look at packet header and/or payload) in order to make the selection decision. Measurement Device A measurement device has access to at least one observation point. It is hosting at least one measurement process and one export process. Metering/Measurement Process The measurement process generates records of packet and flow information. Packets passing the observation point are captured, time stamped, filtered and classified. The measurement process calculates the packet Ids. Passive One-Way-Delay Measurement Abbreviated: POWD Measurement CommonPropertiesID An identifier of a set of common properties that is unique within an Observation Domain. This ID can be used to link to information reported in separate records. The key words "MUST", "MUST NOT", "REQUIRED", "SHALL", "SHALL NOT","SHOULD", "SHOULD NOT", "RECOMMENDED", "NOT RECOMMENDED", "MAY", and "OPTIONAL" in this document are to be interpreted as described in RFC 2119. 3. OPEN POINTS [1] The current definition (and name) of FlowID needs to be generalised as well. It is currently more an identifier for a flow than a generic index. This draft proposes a new IE, CommonPropertiesID; its definition should be included in [IPFIX- INFO]. Throughout this draft flowID has been replaced with CommonPropertiesID. 4. Techniques for bandwidth saving information export The IPFIX protocol is template based. Templates define the structure of data to be exported, describing data fields together with their type and meaning. IPFIX specifies two types of records to export data: data records and option data records. Boschi, Mark Expires August 2006 [Page 3] Reducing redundancy in IPFIX and PSAMP reports These records are defined via template records and option template records. To export information more efficiently we group different flow records depending on their common properties. We define Common Properties as a collection of attributes shared by a set of different flow records. Specific Properties are the remaining flow or packet attributes, not shared with the same set of flows. Since a packet can be considered as a special kind of flow, these considerations remain valid for packets too. We separate the information export using two different records. Common Properties are exported via option data records and are sent only once. These properties represent values common to several flows. The associated Specific Properties are subsequently sent using data records. The association is kept using a particular index that uniquely identifies the Common Properties, the CommonPropertiesID. Figure 1 shows the relation between template and data sets for common and specific properties. The Common Properties option template defines the common attributes of a flow or a group of flows (e.g. IP source and destination address) and the CommonPropertiesID. The CommonPropertiesID is a unique identifier for a set of attributes; this field allows the linkage between those attributes and flow data records. Subsequent option data records of this template export the Common Properties values. The assignment of flow records to common attributes could be alternatively provided by the TemplateID, as explained in Section 4.3. In this case the CommonPropertiesID can be omitted. The format for the information related to single flows or packets is defined in the Specific Properties template. This information is flow/packet specific and normally not shared between many flows/packets. +---------------------+ +---------------------+ | Option Template Set | | Template Set | | | | | Description of | Common Properties | | Specific Properties | exported data +----------+----------+ +----------+----------+ ...........|............................|......................... | | +----------v----------+ +----------v----------+ Exported data | Option Data Set | | Data Set | with references | <------+ | by means of flow | Common Properties | | Specific Properties | or template +---------------------+ +---------------------+ identifiers Figure 1: Template Set and Data Set dependencies The Common Properties option data records SHOULD be sent prior to the corresponding Specific Properties data records. 4.1 Single and multiple associations Several flow records often share a set of common properties. Repeating the information about these common properties for every record introduces a huge amount of redundancy. Let's consider a set of properties "A", e.g. common sourceAddressA and sourcePortA and another set of properties "B", e.g. destinationAddressB and destinationPortB. Figure 2 shows how Boschi, Mark Expires August 2006 [Page 4] Reducing redundancy in IPFIX and PSAMP reports this information is repeated with classical IPFIX export in several records. In Figure 3 we separate Common Properties from Specific flow (or packet) Properties. In order to maintain the relation between these sets of properties we introduce indices (idxA and idxB) for the Common Properties that are unique for all Common Property entries. The purpose of the indices is to serve as "primary key" that identifies rows of the Common Properties. More details about these indices will be given in section 4.3. The rows are then referenced by the Specific Properties by using the appropriate value for the Common Properties identifier. A flow record can refer to one or more Common Properties sets, achieving thus, in the latter case, a better export efficiency. +--------------------+---------------------------------+ | properties A | | +--------------------+---------------------------------+ | properties A | | +--------------------+---------------------------------+ | properties B | | +--------------------+-------------------+-------------+ | properties A | properties B | | +--------------------+-------------------+-------------+ Figure 2: Common and Specific Properties exported in the same record Specific Properties +------+----------------------+ Common Properties > idxA | | +--------------+-----+ +------+----------------------+ | properties A |idxA < > idxA | | +--------------+-----+ +------+----------------------+ | properties B |idxB <--------> idxB | | +--------------+-----+ +------+------+---------------+ > idxA | idxB | | +------+------+---------------+ Figure 3: Common and Specific Properties exported separately 4.2 Export of Per-Packet Information A particular case is the export of per packet information (e.g. for Passive One-Way Delay measurements or PSAMP exports). In this case a single packet could be considered a special case of a flow and consequently per-packet information could be exported using flow records. Doing this though would introduce additional overhead, since packets belonging to the same flow share common attributes (e.g. source address, destination address, etc). Exporting these Common Properties on a per-packet basis, would be redundant. Figure 4 depicts three packets belonging to flow A and one packet belonging to flow B, respectively. It shows export records containing packet information plus flow information (source and destination address). Undoubtedly, flow information (or Common Properties, as defined in this draft) introduces a Boschi, Mark Expires August 2006 [Page 5] Reducing redundancy in IPFIX and PSAMP reports huge amount of redundancy, as it is repeated for every packet in every record. In Figure 5 we separate Common Properties from Specific Properties, i.e. flow from packet information. In order to maintain the relation between Specific (Packet) Properties and Common (Flow) Properties we introduce indices (idxA and idxB), much in the same way as explained in the previous sections. The linkage of one packet and flow B (flow properties B, idxB) is explicitly drawn. One-packet flows +-------------------+----------------------+ | flow properties A | | +-------------------+----------------------+ | flow properties A | | +-------------------+----------------------+ | flow properties B | | +-------------------+----------------------+ | flow properties A | | +-------------------+----------------------+ Figure 4: Flow and packet information represented in one-packet flows Specific (Packet) Properties +------+--------------------+ Common (Flow) Properties > idxA | | +------------------+-----+ +------+--------------------+ |flow properties A |idxA < > idxA | | +------------------+-----+ +------+--------------------+ |flow properties B |idxB <-------> idxB | | +------------------+-----+ +------+--------------------+ > idxB | | +------+--------------------+ Figure 5: Flow information and packet information 4.3 Using Scopes Common Properties are sent via IPFIX option records. IPFIX option records contain one or more scope fields. The Flow Properties record can contain the CommonPropertiesID and/or the TemplateID as scope fields. There are three options: 1) Use CommonPropertiesID as scope The common properties are valid for all data records containing that CommonPropertiesID. This solution limits the number of different flows that can be exported at the same time in the same observation domain to 2**32 (using 32 bits CommonPropertiesIDs) 2) Use CommonPropertiesID and TemplateID as scope The common properties are valid only for data records referring to the template specified by the TemplateID and containing that CommonPropertiesID. This allows the export up to 2**32 flows per template. The solution is to be chosen when the number of flows to be exported is expected to be very high (and beyond the limit posed by solution 1) Boschi, Mark Expires August 2006 [Page 6] Reducing redundancy in IPFIX and PSAMP reports 3) Use TemplateID as scope The common properties are valid for all data records of the specified template. In this case CommonPropertiesIDs are not needed but the exporting process requires a templateID per flow. In the general case this solution is not scalable but can be suitable for certain applications (e.g. flow aggregation). 4.4 CommonPropertiesID Management The management of CommonPropertiesIDs is very similar to the management of TemplateIDs described in [IPFIX-PROTO]. The Exporting Process maintains the CommonPropertiesIDs for the exporter's Observation Domains. Like templateIDs, a CommonPropertiesID MUST be unique per Observation Domain (source identifier in the IPFIX header). Different Observation Domains from the same exporter may use the same CommonPropertiesID value to refer to different flows. There are no constraints regarding the order of the CommonPropertiesID allocation. When limiting the scope to special templates, the CommonPropertiesIDs have to be unique per Observation Domain and template. Using 32 bit CommonPropertiesIDs allows the export of 2**32 active flows in parallel. CommonPropertiesIDs have a certain lifetime inside which they cannot be reused. After that time a CommonPropertiesID can be assigned to another flow. CommonPropertiesID whose lifetime has expired from longer SHOULD be preferred. The lifetime MUST be configurable. The collecting process associates a lifetime with each CommonPropertiesID. The lifetime MUST be configurable. The mapping of data records to flow properties uses the most recent flow definition of the specified CommonPropertiesID. If there is no flow definition of that CommonPropertiesID or the lifetime of the flow definition has been expired, no mapping is possible. In this case the collecting process SHOULD log an error. When IPFIX uses an unreliable transport protocol to export the option data records containing the flow properties and the CommonPropertiesIDs these records MUST be re-sent at regular intervals, whose frequency MUST be configurable. When using a connection oriented transport protocol the flow properties have to be re-sent after a connection re- establishment in prior to the corresponding Packet Properties data records. 5. Examples 5.1 Multiple CommonPropertiesIDs In this section we show how flow information can be exported efficiently using the method described in this draft. Let's suppose we have to export the following flow information: Flow | srcAddr | srcPort | dstAddr | dstPort | nPackets | nBytes ---------------------------------------------------------------- A |10.0.0.1 | 1932 |10.0.1.2 | 80 | 30 | 6000 B | -- | -- |10.0.1.2 | 80 | 50 | 9500 C |10.0.0.1 | 1932 | -- | -- | 60 | 8000 Boschi, Mark Expires August 2006 [Page 7] Reducing redundancy in IPFIX and PSAMP reports Figure 6 shows the option templates, containing the Common Properties together with the CommonPropertiesID, which is the scope. In the first Common Properties template we export the following Information Elements: - the source IPv4 Address, sourceIPv4Address [IPFIX-INFO], with a type of 8 and a length of 4 octets - the source Port, transportSourcePort [IPFIX-INFO], with a type of 7 and a length of 2 octets - the Common Properties ID, with a type [TO BE ASSIGNED, in this Draft referenced to as XX] and a length of 4 octets. The second option template contains the following Information Elements: - the destination IPv4 Address, destinationIPv4Address [IPFIX-INFO], with a type of 12 and a length of 4 octets - the destination port, transportDestinationPort [IPFIX-INFO] with a type of 11, and a length of 2 octets - the Common Properties ID, with a type [TO BE ASSIGNED, in this Draft referenced to as XX] and a length of 4 octets. The flow identifier, which is represented by the CommonPropertiesId Information Element [NOTE: to be included in IPFIX-INFO], is used in both cases as the Scope Field. +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ | Set ID = 3 | Length = 24 octets | +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ | Template ID = 256 | Field Count = 3 | +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ | Scope Field count = 1 |0| CommonPropertiesID = XX | +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ | Scope 1 Field Length = 4 |0| sourceIPv4Address = 8 | +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ | Field Length = 4 |0| transportSourcePort = 7 | +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ | Field Length = 2 | (Padding) | +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ | Set ID = 3 | Length = 24 octets | +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ | Template ID = 257 | Field Count = 3 | +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ | Scope Field count = 1 |0| CommonPropertiesID = XX | +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ | Scope 1 Field Length = 4 |0| destinationIPv4Address = 12| +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ | Field Length = 4 |0|transportDestinationPort = 11| +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ | Field Length = 2 | (Padding) | +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ Figure 6: Example Common Properties Templates Boschi, Mark Expires August 2006 [Page 8] Reducing redundancy in IPFIX and PSAMP reports Considering the values given at the beginning of this section we will export the following option data records: CommonPropertiesID | sourceAddress | sourcePort ---------------------------------------------------- 1 | 10.0.0.1 | 1932 and CommonPropertiesID | dstAddress | dstPort ------------------------------------------------ 2 | 10.0.1.2 | 80 The Specific Properties template consists of the information not contained in the Option Templates, i.e. flow specific information. Additionally, this template contains the CommonPropertiesID. In data records, the value of this field will contain one of the unique indices of the option records exported before. Figure 7 displays the template with the Specific Properties for Flow A (above) and Flow B (below). Note that for flow A two Information Elements of the same type are exported (CommonPropertiesID). In this example we export the following Information Elements: - CommonPropertiesID with a length of 4 octets - The number of packets of the Flow: inPacketDeltaCount in [IPFIX-INFO], with a length of 4 octets - The number of octets of the Flow: inOctetDeltaCount in [IPFIX-INFO], with a length of 4 octets +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ | Set ID = 2 | Length = 24 octets | +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ | Template ID = 259 | Field Count = 4 | +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ |0| CommonPropertiesID = XX | Field Length = 4 | +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ |0| CommonPropertiesID = XX | Field Length = 4 | +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ |0| inPacketDeltaCount = 2 | Field Length = 4 | +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ |0| inOctetDeltaCount = 1 | Field Length = 4 | +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ | Set ID = 2 | Length = 20 octets | +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ | Template ID = 258 | Field Count = 4 | +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ |0| CommonPropertiesID = XX | Field Length = 4 | +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ |0| inPacketDeltaCount = 2 | Field Length = 4 | +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ |0| inOctetDeltaCount = 1 | Field Length = 4 | +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ Boschi, Mark Expires August 2006 [Page 9] Reducing redundancy in IPFIX and PSAMP reports Figure 7: Example Specific Properties Template Considering the values given at the beginning of this section, the data records will look like in the following table. The first row represent the Specific Properties of Flow B (CommonPropertiesID = 2), while the second row those of flow C (CommonPropertiesID = 1). CommonPropertiesID | inPacketDeltaCount | inOctetDelataCount --------------------------------------------------------------- 2 | 50 | 9500 | | 1 | 60 | 8000 The Specific Properties for Flow A will report both CommonPropertiesID values. The data record for Flow A will look like: 0 1 2 3 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ | Set ID = 256 | Length = 16 | +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ | 1 | +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ | 2 | +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ | 30 | 6000 | +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ Figure 8: Specific Properties for Flow A 5.2 Per-Packet Information Export [NOTE: this example uses the FlowID IE to associate Common Properties to Specific Properties. In case of per-packet information export, this is an alternative solution. FlowID management is the same as the CommonPropertiesID management described in Section 4.4] To demonstrate how to use IPFIX efficiently to export per-packet information, this section proposes how to use the IPFIX protocol for exporting flow information (Common Properties) and per-packet information (Specific Properties, in this case related to a long-lived flow) for OWD computation. In order to acquire a One-Way path delay information, two measurement points with synchronized clocks must exist, one at each end of the path under examination. Both measurement points will capture packets, assign them timestamps and generate an identifier for a packet passing that point. Each measurement point will export its measurement data to a collecting process where the data are correlated based on the packet identifiers and timestamps and then the delay is calculated as a difference of two timestamps of a packet pair. The templates that would be needed for exporting measurement data of this kind are illustrated in the figures below. Figure 9 shows the option template containing the information concerning flows using the FlowID as scope. Boschi, Mark Expires August 2006 [Page 10] Reducing redundancy in IPFIX and PSAMP reports In the Flow Properties template we export the following Information Elements: - the source IPv4 Address, sourceIPv4Address [IPFIX-INFO], with a type of 8 and a length of 4 octets - the destination IPv4 Address, destinationIPv4Address [IPFIX-INFO], with a type of 12 and a length of 4 octets - the Class of Service field, ClassOfServiceIPv4 [IPFIX- INFO], with a type of 5 and a length of 1 octet - source and destination ports, transportSourcePort and transportDestinationPort [IPFIX-INFO]with a type of 7 and 11 respectively, and a length of 2 octets each The flow identifier, which is represented by the FlowId Information Element [IPFIX-INFO], is used as the Scope Field. +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ | Set ID = 3 | Length = 40 octets | +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ | Template ID = 256 | Field Count = 7 | +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ | Scope Field count = 1 |0| FlowID = 148 | +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ | Scope 1 Field Length = 4 |0| sourceIPv4Address = 8 | +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ | Field Length = 4 |0| destinationIPv4Address = 12 | +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ | Field Length = 4 |0| classOfServiceIPv4 = 5 | +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ | Field Length = 1 |0| protocolIdentifier = 4 | +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ | Field Length = 1 |0| transportSourcePort = 7 | +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ | Field Length = 2 |0|transportDestinationPort = 11| +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ | Field Length = 2 | (Padding) | +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ Figure 9: Example Flow Properties Template For passive One-Way-Delay measurement, the Packet Properties template consists of at least Timestamp and Packet ID. Additionally, this template contains a flow identifier field. In packet records, the value of this field will contain one of the unique indices of the flow records exported before. Figure 10 displays the template with the packet properties. In this example we export the following Information Elements: - FlowID [IPFIX-INFO] with a length of 4 octets - packetTimestamp, packetID, and packetLength. Since packetID, packetLength and flowID are not (yet) IETF- defined information elements, we export them as enterprise- specific IEs. The three IEs have respectively a type of 220, 221, and 222 and a length of 8, 4, and 4 octets. Boschi, Mark Expires August 2006 [Page 11] Reducing redundancy in IPFIX and PSAMP reports +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ | Set ID = 2 | Length = 36 octets | +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ | Template ID = 257 | Field Count = 4 | +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ |0| FlowID = 148 | Field Length = 4 | +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ |1| packetTimestamp = 220 | Field Length = 8 | +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ | Enterprise number | +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ |1| packetID = 221 | Field Length = 4 | +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ | Enterprise number | +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ |1| packetLength = 222 | Field Length = 4 | +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ | Enterprise number | +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ Figure 10: Example Packet Properties Template The delay is derived by a calculation step: at the collection point packet records of two measurement points are gathered and correlated by means of the packet ID. The resulting delay data records are exported in a similar manner as the packet data have been. Especially, the linkage between delay data and flow information is represented with the discussed flow identifier fields. The OWD properties contain the Packet Pair ID (which is the packet ID matching that of the two contributing packet records), a timestamp (which is the timestamp of the packet passing the reference monitor point) in order to reconstruct a time series, the calculated delay value, and finally a flow identifier. 6. Export and evaluation considerations The main advantage of the methods proposed in this draft is the reduced amount of measurement data that has to be transferred from the exporter to the collector. In addition there is less storage capacity needed at the collector side. On the other hand there is some extra processing power needed on the exporter side to manage Common Properties information and to assign them to flow records. The collector has to process records of two templates instead of just one but has to read and write less data. Additional effort is needed when post processing the measurement data, because now the correlation of flow records with Common Properties information is needed. In the example in section 5.2 using IPFIX to export the measurement data for each received packet 30 bytes have to be transferred (sourceAddressV4=4, destinationAddressV4=4, classOfServiceV4=1, protocolIdentifier=1, transportSourcePort=2, transportDestionationPort=2, packetTimestamp=8, packetID=4, packetLength=4). Disregarding the IPFIX protocol overhead a flow of 1000 packets produces 28000 bytes of measurement data. Using the proposed optimization each packet produces an export of only 20 bytes (packetTimestamp=8, packetID=4, packetLength=4, flowID=4). The export of the flow information produces 16 bytes (sourceAddressV4=4, destinationAddressV4=4, classOfServiceV4=1, protocolIdentifier=1, transportSourcePort=2, transportDestionationPort=2, flowID =4). For a flow of 1000 Boschi, Mark Expires August 2006 [Page 12] Reducing redundancy in IPFIX and PSAMP reports packet this sums up to 16016 bytes. This is a decrease of more than 40 percent. Note that IPFIX allows the reduction of the size of exported Information Elements. This can be applied also to the CommonPropertiesID and would result in a further reduction of bytes. 7. IANA Consideration This document does not imply any IANA action. 8. Security Considerations For the proposed use of the IPFIX protocol for bandwidth-saving export the security considerations as for the IPFIX protocol apply. 9. Normative References [IPFIX-PROTO] Benoit Claise et Al.: IPFIX Protocol Specification, IETF draft work in progress , September 2005 [IPFIX-INFO] J. Quittek, S.Bryant, B.Claise, J. Meyer: Information Model for IP Flow Information Export Internet-draft work in progress , September 2005 [PSAMP-PROTO] Benoit Claise: PSAMP Protocol Specification, Internet Draft , October 2005 10. Author's Addresses Elisa Boschi Hitachi Europe SAS Immeuble Le Theleme 1503 Route des Dolines 06560 Valbonne, France Phone: +33 4 89874180 Email: elisa.boschi@hitachi-eu.com Lutz Mark Fraunhofer Institute for Open Communication Systems Kaiserin-Augusta-Allee 31 10589 Berlin Germany Phone: +49-30-34 63 7306 Fax: +49-30-34 53 8306 Email: mark@fokus.fraunhofer.de 11. Intellectual Property Statement The IETF takes no position regarding the validity or scope of any Intellectual Property Rights or other rights that might be claimed to pertain to the implementation or use of the technology described in this document or the extent to which any license under such rights might or might not be available; nor does it represent that it has made any independent effort to identify any such rights. Information on the procedures with respect to rights in RFC documents can be found in BCP 78 and BCP 79. Boschi, Mark Expires August 2006 [Page 13] Reducing redundancy in IPFIX and PSAMP reports Copies of IPR disclosures made to the IETF Secretariat and any assurances of licenses to be made available, or the result of an attempt made to obtain a general license or permission for the use of such proprietary rights by implementers or users of this specification can be obtained from the IETF on-line IPR repository at http://www.ietf.org/ipr. The IETF invites any interested party to bring to its attention any copyrights, patents or patent applications, or other proprietary rights that may cover technology that may be required to implement this standard. Please address the information to the IETF at ietf-ipr@ietf.org. 12. Copyright Statement Copyright (C) The Internet Society (2006). This document is subject to the rights, licenses and restrictions contained in BCP 78, and except as set forth therein, the authors retain all their rights. 13. Disclaimer This document and the information contained herein are provided on an "AS IS" basis and THE CONTRIBUTOR, THE ORGANIZATION HE/SHE REPRESENTS OR IS SPONSORED BY (IF ANY), THE INTERNET SOCIETY AND THE INTERNET ENGINEERING TASK FORCE DISCLAIM ALL WARRANTIES, EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO ANY WARRANTY THAT THE USE OF THE INFORMATION HEREIN WILL NOT INFRINGE ANY RIGHTS OR ANY IMPLIED WARRANTIES OF MERCHANTABILITY OR FITNESS FOR A PARTICULAR PURPOSE. Boschi, Mark Expires August 2006 [Page 14]