IPFIX Working Group A. Kobayashi Internet-Draft K. Ishibashi Intended status: Informational T. Kondoh Expires: December 25, 2007 NTT PF Lab. D. Matsubara Hitachi June 23, 2007 Reference Model for IPFIX Mediators draft-kobayashi-ipfix-mediator-model-00.txt Status of this Memo By submitting this Internet-Draft, each author represents that any applicable patent or other IPR claims of which he or she is aware have been or will be disclosed, and any of which he or she becomes aware will be disclosed, in accordance with Section 6 of BCP 79. Internet-Drafts are working documents of the Internet Engineering Task Force (IETF), its areas, and its working groups. Note that other groups may also distribute working documents as Internet- Drafts. Internet-Drafts are draft documents valid for a maximum of six months and may be updated, replaced, or obsoleted by other documents at any time. It is inappropriate to use Internet-Drafts as reference material or to cite them other than as "work in progress." The list of current Internet-Drafts can be accessed at http://www.ietf.org/ietf/1id-abstracts.txt. The list of Internet-Draft Shadow Directories can be accessed at http://www.ietf.org/shadow.html. This Internet-Draft will expire on December 25, 2007. Copyright Notice Copyright (C) The IETF Trust (2007). Kobayashi, et al. Expires December 25, 2007 [Page 1] Internet-Draft Reference Model for IPFIX Mediators June 2007 Abstract An IPFIX Mediator is an intermediate node between IPFIX devices and traffic collectors. This node acts as an IPFIX proxy, IPFIX firewall, and IPFIX concentrator. An IPFIX Mediator mediates the IPFIX protocol using several functions. That enables the traffic monitoring system to become a high-capacity system and accommodate a variety of traffic monitoring methods. This document describes each function that is provided by IPFIX Mediators and the method of handling the Flow Records of each function. In addition, this describes the model of the solution scenario using IPFIX Mediator. Table of Contents 1. Introduction . . . . . . . . . . . . . . . . . . . . . . . . . 3 2. Terminology . . . . . . . . . . . . . . . . . . . . . . . . . 5 3. Internal Components Model . . . . . . . . . . . . . . . . . . 8 3.1. Collecting Process . . . . . . . . . . . . . . . . . . . . 8 3.2. Metering Process . . . . . . . . . . . . . . . . . . . . . 9 3.2.1. Selection Process . . . . . . . . . . . . . . . . . . 9 3.2.2. Aggregation Process . . . . . . . . . . . . . . . . . 9 3.2.3. Modification Process . . . . . . . . . . . . . . . . . 10 3.3. Exporting Process . . . . . . . . . . . . . . . . . . . . 12 3.4. Storing Process . . . . . . . . . . . . . . . . . . . . . 12 4. IPFIX Protocol Considerations . . . . . . . . . . . . . . . . 14 4.1. Export Time Issue . . . . . . . . . . . . . . . . . . . . 14 4.2. Observation Domain ID Management . . . . . . . . . . . . . 14 4.3. Template Management . . . . . . . . . . . . . . . . . . . 14 4.4. Transport Session Management . . . . . . . . . . . . . . . 15 4.5. Option Template Management . . . . . . . . . . . . . . . . 15 4.6. Reporting of Exporter Information . . . . . . . . . . . . 15 5. Solution Scenarios with IPFIX Mediators . . . . . . . . . . . 16 5.1. Flexible Aggregation . . . . . . . . . . . . . . . . . . . 16 5.2. Distributed Aggregation . . . . . . . . . . . . . . . . . 16 5.3. Duplication of Flow Records . . . . . . . . . . . . . . . 17 5.4. Distribution of Flow Records . . . . . . . . . . . . . . . 18 5.5. Extraction of Suspicious Flow . . . . . . . . . . . . . . 19 6. Mediator Option Template Presentation . . . . . . . . . . . . 20 6.1. Exporter Information Option Template . . . . . . . . . . . 20 6.2. Usage of Scope Field . . . . . . . . . . . . . . . . . . . 22 7. Security Considerations . . . . . . . . . . . . . . . . . . . 24 8. IANA Considerations . . . . . . . . . . . . . . . . . . . . . 25 9. References . . . . . . . . . . . . . . . . . . . . . . . . . . 26 9.1. Normative References . . . . . . . . . . . . . . . . . . . 26 9.2. Informative References . . . . . . . . . . . . . . . . . . 26 Authors' Addresses . . . . . . . . . . . . . . . . . . . . . . . . 27 Intellectual Property and Copyright Statements . . . . . . . . . . 28 Kobayashi, et al. Expires December 25, 2007 [Page 2] Internet-Draft Reference Model for IPFIX Mediators June 2007 1. Introduction Several problems regarding traffic monitoring have occurred in several networks, as follows. o Scalability of collector on large-scale networks As the sizes of networks become larger, the number of Flow Records becomes greater. Large numbers of Flow Records have been burdening management networks and the collecting process. Maintaining scalability is difficult as a particular network grows. Generally, network operations need to monitor overall wide-scale traffic behavior and investigate detailed traffic information when traffic incidents happen. Meeting these requirements seems to be difficult for a single collector node because of large numbers of Flow Records. o Handling multifaceted network environment On the other hand, networks such as IPv4, IPv6, and VPN on MPLS have recently become more multifaceted. These sorts of Flow Records need to be analyzed separately from a different perspective. However, handling them separately without improving the capability of the Collector is difficult. o Exchanging traffic information between different networks Traffic information is considered to be necessary for ISP network providers as well as each customer. To that end, the network provider needs to export specified Flow Records while avoiding privacy violations. In addition, exchanging traffic information between the associated networks could be necessary when traffic incidents happen. In that case, the Flow Records should be checked according to the service provider's policy before exporting. Therefore, Flow Records exported from Exporter directly without modification can not be fed each customer or associated network operators without modification. IPFIX Mediators enable us to overcome these problems by preprocessing Flow Records. By defining IPFIX Mediators, we can take increasing advantage of an extensive template format, and handle Flow Records in accordance with our preference. This document describes some solutions using IPFIX Mediators that solve these problems and components that are needed in the internal process. The IPFIX Mediator, which is located between one or more Exporting Processes and one or more Collecting Processes, has a main function for handling Flow Records. That function stores original Flow Kobayashi, et al. Expires December 25, 2007 [Page 3] Internet-Draft Reference Model for IPFIX Mediators June 2007 Records and mediates them. In addition, renewed Flow Records are generated and distributed to an appropriate Collector or traffic analyzer in accordance with flow content. The internal model of a Mediator is composed of Collecting Processes, Metering Processes, and Exporting Processes. IPFIX Mediator acts as a Collector by receiving Flow Records, and it acts as an Exporter by sending Flow Records. This dual-role architecture enables cascading Mediators and building a combination of several solutions. This document describes a model of a solution scenario by using IPFIX Mediator and its key component. The key words "MUST", "MUST NOT", "REQUIRED", "SHALL","SHALL NOT", "SHOULD", "SHOULD NOT", "RECOMMENDED", "MAY", and "OPTIONAL" in this document are to be interpreted as described in [RFC2119]. Kobayashi, et al. Expires December 25, 2007 [Page 4] Internet-Draft Reference Model for IPFIX Mediators June 2007 2. Terminology The definitions of basic IPFIX and PSAMP terms are identical with those in [I-D.ietf-psamp-framework], [RFC3917], [I-D.ietf-ipfix-protocol], [I-D.ietf-ipfix-info], and [I-D.ietf-ipfix-architecture]. Other than the above terminology, the following terminology related to IPFIX Mediator is used in this document. Therefore, the terms defined in the IPFIX terminology are capitalized in this document. IPFIX Mediator An IPFIX Mediator hosts at least one pair of Exporting Process and Collecting Process. An IPFIX Mediator may have the Metering Process and Storing Process as optional. An IPFIX Proxy, an IPFIX Firewall and an IPFIX concentrator are one node of IPFIX Mediators. Original Exporter An Original Exporter hosts Observation Points where IP packets can be observed. IPFIX Proxy An IPFIX Proxy acts as a proxy for Original Exporter, which hosts Observation Points. An IPFIX Proxy may receive Flow Records from one or multiple Exporting Processes, and send them to one or multiple Collecting Processes. An IPFIX Proxy does not send the information about an Original Exporter to the Collector to act as an Original Exporter. IPFIX Firewall An IPFIX Firewall exports the Flow Records to a different network domain at the edge of the self-network domain. From the Collector's point of view, an IPFIX Firewall acts as an Original Exporter, just like an IPFIX Proxy. In addition, an IPFIX Firewall reviews whether received Flow Records are passed forward to the Collector to hide the network topology or privacy information. Metering Process The Metering Process in IPFIX Mediators can be considered the partial Metering Process separated from the Metering Process in the Original Exporter. The Metering Process in IPFIX Mediators consists of a set of subprocesses that includes the Selection Kobayashi, et al. Expires December 25, 2007 [Page 5] Internet-Draft Reference Model for IPFIX Mediators June 2007 Process, the Aggregation Process and the Modification Process. The Metering Process generates the final Flow Records that should be exported. Selection Process The Selection Process in an IPFIX Mediator is similar to that of PSAMP Devices, which is described in [I-D.ietf-psamp-framework]. However, the Selection Process in an IPFIX Mediator differs from the following functions. The Selection Process in PSAMP Devices has two types of selection functions: Filtering and Sampling. In addition, the filtering function has two types of filtering methods: field-match filtering and hash-based selection. The Selection Process in an IPFIX Mediator has only the field-match filtering functions. This filtering function selects Flow Records based on Flow Record content. The Selection Process is one of the subprocesses in the Metering Process. Aggregation Process The Aggregation Process creates aggregated Flow Records from inputted Flow Records, in accordance with aggregation rules that are described in [I-D.dressler-ipfix-aggregation]. The Aggregation Process is one of the subprocesses in the Metering Process. Modification Process The Modification Process carries out the addition, deletion, and modification of the Information Elements included in inputted Flow Records. The Modification Process adds Information Elements like derived packet properties that canot be extracted in the Original Exporter. Information Elements related to derived packet properties are described in [I-D.ietf-ipfix-info]. In addition, the Modification Process modifies the values of the specific Information Element. For example, the modification of values is like anonymizing some Information Elements to avoid violating privacy. The Modification Process deletes some Information Elements included in inputted Flow Records. The Modification Process is one of the subprocesses in the Metering Process. Storing Process The Storing Process selects specified Information Elements according to the storing rules, and then stores inputted Flow Records in a storage system such as a database or flat-file system. Information Elements are specified in storing rules. In addition, the Observation Domain ID and Export Time in the header Kobayashi, et al. Expires December 25, 2007 [Page 6] Internet-Draft Reference Model for IPFIX Mediators June 2007 of IPFIX messages are specified in the storing rules. Distribute Function The Distribute Function distributes final Flow Records based on the Flow content. The final Flow Records are handled by the Exporting Process to export them to Collector. Each classified Flow Record is exported to each Collector. Observation Domain ID An IPFIX Mediator doesn't host the Observation Point and Observation Domain. Though, the Observation Domain ID in IPFIX header sent by IPFIX Mediator also indicates the largest set of Observation Points in the Original Exporter, but this value does not indicate the physical entity of the Original Exporter. If inputted Flow Records are aggregated in the Metering Process, the Observation Domain ID value in IPFIX header SHOULD be 0. Transport Session Information In SCTP, the Transport Session Information is the SCTP association. In TCP and UDP, the Transport Session Information corresponds to a 5 tuple {exporter IP address, collector IP address, exporter transport port, collector transport port, transport protocol}. In IPFIX Mediator, the Collecting Process manages this information. Kobayashi, et al. Expires December 25, 2007 [Page 7] Internet-Draft Reference Model for IPFIX Mediators June 2007 3. Internal Components Model The following figure indicates four components (Collecting Process, Metering Process, Exporting Process, and Storing Process) within the IPFIX Mediator that are referred to in [RFC3917]. The Metering Process can have one or multiple subprocesses. These subprocesses are the Selection Process, Aggregation Process, and Modification Process that can be connected to each other in any sequence defined by the user. The Metering Process and Storing Process are options. +--------------------------------------------------------------+ | IPFIX Mediator | | .----------. .---------. | | | | .-------------------------------. | | | | |Collecting| | Metering Process | |Exporting| | | |Process | |.-------. .-------. .-------.| |Process | | | | |-->|sub |->|sub |->|sub |-->| | | IPFIX | ||process| |process| |process|| | |IPFIX --> | | ||#1 | |#2 | |#3 || | |--> | | | |'-------' '-------' '-------'| | | | | | | '-------------------------------' | | | | | |------------------------------------>| | | | | | .-------. | | | | | |->|Storing| | | | | | | |Process| | | | | | | '-------' | | | | '----------' '---------' | +--------------------------------------------------------------+ Figure A: Key components within IPFIX Mediator. Each process is associated with a common identifier in the IPFIX Mediator. This method is similar to PSAMP associations in [I-D.ietf-psamp-sample-tech]. 3.1. Collecting Process This process receives Flow Records from the previous Exporter. The instance of this process is created according to the IPFIX session. This process has functions that are described in [I-D.ietf-ipfix-protocol]. The Collecting Process also forwards received Flow Records with IPFIX header information and Transport Session Information to multiple Metering Processes or Storing Processes. In other words, Flow Records can be duplicated by forwarding several Metering Processes. In addition, the Collecting Process can directly forward Flow Records to Exporting Process. Kobayashi, et al. Expires December 25, 2007 [Page 8] Internet-Draft Reference Model for IPFIX Mediators June 2007 3.2. Metering Process This process generates the renewed Flow Records from inputted Flow Records with received IPFIX header information, such as "Export Time" and "Observation Domain ID". This process hosts the several subprocesses. The processing order of these functions, which could be located by user definitions, would lead to different renewed Flow Records. 3.2.1. Selection Process This process decides whether each Flow Record passes through to the next process. Theprocess has a filtering function and selects Flow Records that are matched under given conditions. Prior to receiving Flow Records, this process has instruction pattern data that are defined by the user, which specifies how the Flow Records are treated by this process. If the value of some Information Elements in the Flow Record match the instruction pattern, this process selects Flow Records with all fields and forwards these Flow Records to the next process. For example, this process selects the Flow Records that are included in the specified destination IP address. 3.2.2. Aggregation Process This process gathers Flow Records within a given time interval and then distinguishes Flow Records that have common properties. If values of a given key field are the same, that means these Flow Records have common properties. This process merges Flow Records that have a common property and creates an aggregated Flow Record. Therefore, for example, aggregated Flow Records have an aggregation counter that indicates the number of packets. These functions are defined in accordance with the IPFIX aggregation rule in [I-D.dressler-ipfix-aggregation]. The process has instructions that are user defined prior to receiving Flow Records. The process indicates the Information Elements that should become aggregated flow keys and other Information Elements that should be kept or discarded. In addition, these instruction rules include Information Elements that should be added to aggregated Flow Records. The aggregated Flow Records may need to complement information that is discarded during the aggregation process. They help the Collector to analyze aggregated Flow Records. For example, these Information Elements correspond to "averageActiveTime", "synCount", and "flowCount" elements, as follows. o averageActiveTime This Information Element indicates average time in milliseconds of Kobayashi, et al. Expires December 25, 2007 [Page 9] Internet-Draft Reference Model for IPFIX Mediators June 2007 difference between flow start time and end time of each flow included in the aggregated flow. This Information Element is created from the flow time stamp Information Elements. There are "flowStartSeconds", "flowEndSeconds", "flowStartMilliSeconds", "flowEndMilliSeconds", "flowStartSysUpTime", and "flowEndSysUpTime". Moreover, "minimumActiveTime" and "maxmumActiveTime" might be considered in addition to the element. o synCount This Information Elements the number of Flow Records that have "tcpControlBits" which the SYN bit sets to 1 in an aggregated flow. Using this element, we can determine the number of SYN packets throughout the network. Moreover, "ackCount", "finCount", "pshCount", "urgCount", and "rstCount" might be considered in addition to this element. o flowCount This Information Element is the number of Flow Records included in the aggregated flow. 3.2.3. Modification Process This process modifies the received Flow Records. This process can add the new Information Elements, delete included Information Elements, or modify the value of included Information Elements, as follows. If this process modify the original template, it SHOULD revise the received "flowKeyIndicator". Addition of new Information Element This function adds specified Information Elements into the inputted Flow Records. The values of Information Elements are extracted by searching some database based on the inputted content of Flow Records. The added Information Elements and used Information Elements are configured according to instructions by the user to obtain the value. The method to obtain the value from some Information Elements is outside the scope of this document. The IPFIX Mediator instead of the Original Exporter adds a derived packet property parameter, which is useful for the traffic monitoring technique. Doing that can compensate for the inability of some Exporters to add a derived packet property parameter. Hereby, the Collector does not need to recognize the difference between implementations of routers from several vendors. For example, the addition of "bgpNextHop{IPv4|IPv6}Address" and "bgpCommunity" Information Elements is useful for making a traffic Kobayashi, et al. Expires December 25, 2007 [Page 10] Internet-Draft Reference Model for IPFIX Mediators June 2007 matrix that covers the whole network domain. "bgpNextHop{IPv4| IPv6}Address" can indicate the egress router of some network domain. In addition, "bgpCommunity" can indicate the same group of destination or source IP addresses. This value can be given by looking for the BGP route database based on the destination or source IP address. In addition, "mplsVpnRouteDistinguisher", which can not be extracted from the core router in MPLS networks, indicates the customer's identification. We can monitor the traffic behavior per customer by adding "mplsVpnRouteDistinguisher" to the Flow Records. This value can be given by looking for the BGP route database based on the "mplsTopLabelStackSection" and "mplsTopLabel{IPv4|IPv6}Address". Deletion of Information Element This function deletes specified Information Elements according to the instructions that are configured by the user, which indicate whether an Information Element should be removed. Hiding network topology information and private information by using this function is possible. In the case of exchanging Flow Records with different network domains or customers, this function can avoid making a vulnerability by deleting unnecessary Information Elements. By deleting unnecessary Information Elements, this function can hide the network topology and another customer's information. In particular, "ipNextHopIP{v4|v6}Address", "bgpNextHopIP{v4| v6}Address", and "bgp{Next|Prev}AdjacentAsNumber" correspond to network topology information. In addition, MPLS-related Information Elements, such as "mplsLabelStackSection", that are useless for customers might be removed in the case of feeding the Flow Records to VPN customers. Modification of the value of Information Element This function modifies the value of specified Information Elements according to instructions configured by the user. For example, this function enables us to overwrite private information with zeros or the maximum value. In particular, IP address and port number is sensitive private information. In the case of monitoring traffic trends and traffic engineering, these Information Elements are not essential factors for those purposes. In that case, modification anonymizes the relevant Information Elements to prevent a violation of privacy. If modification can anonymize some Information Elements, it might need to report which Information Elements are anonymized. For example, "anonymizationIndicator" indicates which Information Elements have Kobayashi, et al. Expires December 25, 2007 [Page 11] Internet-Draft Reference Model for IPFIX Mediators June 2007 been anonymized as a bitmap, just like the "flowKeyIndicator". The anonymization method is outside the scope of this document. 3.3. Exporting Process This process forwards Flow Records to the next Collector. These processes manage the reporting template and make an IPFIX datagram. In addition, this process has the Distribution Function as an option. If this function is enabled, this process distributes Flow Records based on Flow content and then exports each classified Flow Record to each Collector. The Exporting Process distributes Flow Records on the basis of the peering AS, as shown in the following figure. Each classified Flow Record is exported to a dedicated Collector on the basis of the Peering AS. +-------------------------------------------+ | IPFIX Mediator .----------. | | |Exporting | | | |Process | | | .----------. .---------. | | | PeerAS#100 Flow -->|Collecting|->|Metering |----->/------------> Collector#A Records |Process | |Process | | | | | PeerAS#200 | '----------' '---------' | /------------> Collector#B | | | | | PeerAS#300 | | /------------> Collector#C | | | | | PeerAS#400 | | /------------> Collector#D | | | | | '----------' | +-------------------------------------------+ Figure B: Exporting each classified Flow Record to dedicated Collector. 3.4. Storing Process This process selects specified Information Elements using the storing instruction from the inputted Flow Records. Prior to receiving Flow Records, this process has instructions that are configured by the user, which indicate whether each field should be stored or discarded. The field modifier indicates "keep" or "discard", which is similar to the instruction of the aggregation process. The field modifier specifies how these Information Elements are treated by the process. This field modifier is applied to Information Elements within Flow Record and IPFIX header, such as "Observation Domain ID" and "Export time". This header information MAY be used when IPFIX Kobayashi, et al. Expires December 25, 2007 [Page 12] Internet-Draft Reference Model for IPFIX Mediators June 2007 datagrams are made of past Flow Records. When another node retrieves past Flow Records, we can consider several specifications. One solution is that another node gets a specified flat file from a Mediator and decodes that flat file by itself. Other solutions are that another node sends out the query command to the IPFIX Mediator through XML-RPC, SNMP, or NETCONF, and then the IPFIX Mediator exports the specified past Flow Records. This function is outside the scope of this document. Kobayashi, et al. Expires December 25, 2007 [Page 13] Internet-Draft Reference Model for IPFIX Mediators June 2007 4. IPFIX Protocol Considerations This section describes IPFIX protocol considerations with regard to IPFIX Mediator. 4.1. Export Time Issue If the Exporting Process writes the "Export Time" of the IPFIX message when an IPFIX message leaves, an IPFIX Mediator needs to compensate for the delta time Information Elements contained in each Flow Record. An IPFIX Proxy SHOULD reuse the "Export Time" of received IPFIX messages from the Original Exporter. 4.2. Observation Domain ID Management To comply with the IPFIX protocol the Observation Domain ID value is RECOMMENDED to be assigned uniquely per IPFIX Mediator. In addition, Observation Domain ID SHOULD be 0 when IPFIX Mediator exports aggregated Flow Records and cannot manage the specific Observation Domain ID of the Original Exporter. If an IPFIX Proxy relays an IPFIX datagram from a transport session to a transport session, IPFIX Proxy does not need to overwrite the Observation Domain ID with another value. If an IPFIX Proxy relays an IPFIX datagram from multiple transport sessions to a single ransport session, IPFIX Proxy needs to overwrite the Observation Domain ID. In that case, IPFIX Proxy assigns the Observation Domain ID based on received Transport Session Information and the original Observation Domain ID. The renewed Observation Domain ID SHOULD be managed using the received Transport Session Information and original Observation Domain ID. This linkage information is available for overwriting the scope field of the option template. 4.3. Template Management The template ID of a generated template SHOULD be unique on the basis of the Observation Domain ID assigned by an IPFIX Mediator. The template ID needs to be unique on the basis of IPFIX Mediator when the Observation Domain ID is 0. If the IPFIX Mediator overwrites the received template ID to relay a received template or modified template, the renewed template ID SHOULD be managed using received Transport Session Information and received Observation Domain ID. This linkage information is available for overwriting scope field of an option template and template handling. If IPFIX Mediator receives a "template withdraw message", it SHOULD modify this message to indicate relevant templates, and send "template withdraw message". Kobayashi, et al. Expires December 25, 2007 [Page 14] Internet-Draft Reference Model for IPFIX Mediators June 2007 4.4. Transport Session Management Each session of the Collecting Process and Exporting Process should operate independently. Even if one session is reset, the status of the other session is kept current. However, templates for resetting collecting session SHOULD be withdrawn for the exporting session. 4.5. Option Template Management IPFIX Mediator MUST check whether the scope field is applicable, if received Data Records associated Options templates are exported. If an IPFIX Mediator rewrites the Observation Domain ID or template ID, these values included in scope fields SHOULD be rewritten before exporting. Instead of exporting the Options Template Records and associated Data Records, Information Elements exported using the Options template Record from the Original Exporter, such as sampling rate or sampling method, could be merged in a Flow Record in an IPFIX Mediator. In that case, IPFIX Mediator MUST modify the relevant Template Record. Several sorts of received statistics Options Template Records and associated Data Records could be exported in different ways as other templates. In IPFIX Proxy, the Data Record associated by statistics Options Template Records can be exported after merging its counter. In addition, statistics Options Template Records and associated Data Records can be exported by indicating the source of the statistics data as a scope field instead of merging the counter. This method is described in Section 6. The user policy determines whether IPFIX Mediator and the above methods should export Option Templates Records and associated Data Records. 4.6. Reporting of Exporter Information Reporting of Exporter Information, such as Exporter IP address, is useful to identify the Original Exporter. There are various methods as follows. An IPFIX Mediator can directly merge Exporter Information into Flow Records or use Options Templates described in Section 6. If an IPFIX Mediator received fields related to the Exporter information, IPFIX Mediator SHOULD NOT rewrite its own previous Exporter information. The IPFIX Mediator can append its own previous Exporter Information instead of rewriting. In the Collecting Process, the order of the Exporter information means the Original Exporter and the route of IPFIX Mediator. These methods defined by user policy determine whether IPFIX Mediator should report that information has been exported Kobayashi, et al. Expires December 25, 2007 [Page 15] Internet-Draft Reference Model for IPFIX Mediators June 2007 5. Solution Scenarios with IPFIX Mediators 5.1. Flexible Aggregation An IPFIX Mediator can aggregate Flow Records in the same manner as that of IPFIX concentrator and reduce the number of Flow Records received by a traffic collector. The following figure indicates a cascade connection of IPFIX Mediators. If a Collector measures a traffic matrix to obtain traffic demand, the Collector needs Flow Records of the whole network domain, but does not need detailed Flow Records. In the first step, a Mediator receives Flow Records from IPFIX Devices and then creates aggregated low-level Flow Records. For example, this step is prefix mask aggregation. Next, the Mediator receives aggregated Flow Records and aggregates them further. For example, the second step is the aggregation of the BGP next-hop address and exporter address. After this, the collector receives high-level aggregated Flow Records and then stores them. This method enables step-by-step aggregation of Flow Records without overloading a single node. .--------. .--------. |IPFIX | |IPFIX | |router#1|---->|Mediator|---. | | |*1 | | '--------' '--------' | .--------. .---------. '--->|IPFIX | |Traffic | .--------. .--------. .--->|Mediator|---->|Collector| |IPFIX | |IPFIX | | |*2 | | | |router#2|---->|Mediator|---' '--------' '---------' | | |*1 | '--------' '--------' Figure C: Flexible Aggregation with cascading IPFIX Mediators. 5.2. Distributed Aggregation When the network is used globally, the distances between PoPs become longer, and the maintenance of a dedicated management network is very expensive. Therefore, the huge number of Flow Records has burdened the management networks of global ISPs. If we place Mediators at each PoP, the number of Flow Records exported from each PoP can be reduced. Mediators can minimize the number of Flow Records exported to the Collector. If the Collector needs detailed information, it can retrieve Flow Records from Mediators that store original Flow Records. Kobayashi, et al. Expires December 25, 2007 [Page 16] Internet-Draft Reference Model for IPFIX Mediators June 2007 A management network of a global ISP is shown in the following figure. The Mediators are located at each PoP of the network, and they collect Flow Records from routers in each PoP domain. The Mediator reduces the number of Flow Records by aggregating or filtering, so this system reduces the load of a management network. POP#Asia .--------. .--------. | .---------. .--------. | |----->|IPFIX | |IPFIX | |------->|Mediator |----. |router |---'----->|#1 | | |#1 |-' '---------' | '--------' | | POP#America | .--------. | .--------. | .---------. | .---------. .--------. | |----->|IPFIX | '---->|Traffic | |IPFIX | |------->|Mediator |--------->|Collector| |router |---'----->|#2 | .---->| | |#4 |-' '---------' | '---------' '--------' | | POP#Europe | .--------. | .--------. | .---------. | .--------. | |----->|IPFIX | | |IPFIX | |------->|Mediator |----' |router |---'----->|#3 | |#7 |-' '---------' '--------' Figure D: Traffic monitoring architecture in global network. 5.3. Duplication of Flow Records An IPFIX Mediator duplicates Flow Records to achieve redundant storage or utilizes them for several purposes. The pair of Collecting Process and Metering Processes is similar to the pair of the Observation Point and Metering Process. The Collecting Process duplicates Flow records by forwarding them to the multi-Metering Process. Kobayashi, et al. Expires December 25, 2007 [Page 17] Internet-Draft Reference Model for IPFIX Mediators June 2007 Several departments in an ISP want to use the same traffic information for each intended purpose. For example, the network design department measures the traffic matrix to obtain traffic demand, and the customer service division uses traffic information for performing accounting services for each customer while the network operation center uses traffic information for trouble shooting analysis. That case is shown in the following figure. An IPFIX Mediator distributes Flow Records to several Collectors that have the appropriate aggregated granularity. In addition, when a NOC conducts troubleshooting, past Flow Records from Mediators can be retrieved. Measurement traffic matrix. .--------. .---------. |IPFIX | |Traffic | |router#1|----. .---->|Collector| | | | | |#1 | '--------' | | '---------' | | Using Accounting info. .--------. | .---------. | .---------. |IPFIX | '---->|IPFIX |----' |Traffic | |router#2|--------->|Mediator |--------->|Collector| | | .---->| |----. |#2 | '--------' | '---------' | '---------' | | Using Trouble shooting. .--------. | | .---------. |IPFIX | | | |Traffic | |router#1|----' '---->|Collector| | | |#3 | '--------' '---------' Figure E: Duplication of Flow Records for several purposes. 5.4. Distribution of Flow Records An IPFIX Mediator distributes Flow Records based on Flow Record content. This function enables load balancing of Collector and sorting Flow Records without extra Collector functions. If the Flow Records are used as accounting information, this solution is useful. Kobayashi, et al. Expires December 25, 2007 [Page 18] Internet-Draft Reference Model for IPFIX Mediators June 2007 When we disclose traffic information to each customer, security or the privacy policy should be considered. In that case, IPFIX Mediator hides private information about each customer. For example, Mediator distributes traffic information based on RD (Route Distinguisher), egress IF, peering AS number, or BGP next hop, which identify the customer. In the following figure, the IPFIX Mediator distributes Flow Records based on RD. The system securely allows each customer to access only their own records. .--------. .---------. |IPFIX | |Traffic | |router#1|----. .---->|Collector|<===> Customer#A | | | | |#1 | '--------' | | '---------' | RD=100:1 | .---------. | .--------. '---->|IPFIX |----' .---------. |IPFIX | |Mediator | RD=100:2 |Traffic | |router#2|--------->| |--------->|Collector|<===> Customer#B | | | | |#2 | '--------' .---->| |----. '---------' | '---------' | | RD=100:3 .--------. | | .---------. |IPFIX | | | |Traffic | |router#1|----' '---->|Collector|<===> Customer#C | | |#3 | '--------' '---------' Figure F: Distribution of Flow Records for each customer. 5.5. Extraction of Suspicious Flow An IPFIX Mediator performs filtering based on Flow Record content. If the filter conditions are set depending on the suspicious flow as follows, the Collector receives the specified suspicious flow and detects an anomalous flow by simply monitoring the traffic volume of each suspicious flow. o TCP Flow Records whose "tcpControlBits" value is set to "null" o TCP Flow Records whose "tcpControlBits" value is set to the SYN bit only and the packet counter is only 1. o ICMP Flow Records whose length is too long. Kobayashi, et al. Expires December 25, 2007 [Page 19] Internet-Draft Reference Model for IPFIX Mediators June 2007 6. Mediator Option Template Presentation This section describes Option Templates that are used by IPFIX Mediators. 6.1. Exporter Information Option Template Each IPFIX Mediator and final destination Collector needs to know the Original Exporter and route of IPFIX Mediators. Therefore, each IPFIX Mediator informs the next Collector about previous Exporter information, which is the Exporter Information Option Template that specified the Original Exporter and the route of the IPFIX Mediator. The final destination Collector can recognize them by receiving this template. This template is composed of the following Information Elements. o exporter{IPv4|IPv6}Address o collector{IPv4|IPv6}Address o exporterTransportPort o collectorTransportPort o collectorTransportProtocol o observationDomainId The Observation Domain ID of the Original Exporter or IPFIX Mediator is identified by specifying exporter/collector Information Elements, such as "collector{IPv4|IPv6}Address", "collectorTransportPort", "collectorTransportProtocol", ,"exporter{IPv4|IPv6}Address", "exporterTransportPort", and "observationDomainId". The set of "observationDomainId" and "templateId" or "observationDomainId" might be used as a scope field. Not all Information Elements are necessary. For example, the exporter{IPv4|IPv6}Address is necessary to inform the next Collector the Original Exporter which created Flow Records. If the IPFIX Mediator receives this template, it SHOULD not overwrite each field. The IPFIX Mediator appends its own previous Exporter information onto received Data Records specified by the Exporter Option Template and sends that information to the Collector. In this manner, the route is maintained until the final destination Collector. Kobayashi, et al. Expires December 25, 2007 [Page 20] Internet-Draft Reference Model for IPFIX Mediators June 2007 The following example describes the cascade connection of IPFIX Mediators. Each Mediator informs the next Collector about previous Exporter information. Session#a Session#b Session#c Router --------> Mediator#1 --------> Mediator#2 -------->Collector IP:1.1.1.1 IP:2.2.2.2 IP:3.3.3.3 SrcPort:10 DstPort:20 DstPort:40 ODID:10 SrcPort:30 SrcPort:50 ODID:0 ODID:0 Figure G: Cascade connection of IPFIX Mediators. Mediator#1 or Mediator#2 sends a Data Record specified by the Exporter Option Template. The Data records are shown in Session#b or Session#c, as follows. Session#b Data Record: Field Count = 7 Scope Count = 1 templateId = XXX exporterIPv4Address = 1.1.1.1 collectorIPv4Address = 2.2.2.2 collectorTransportProtocol = 16 exporterTransportPort = 10 collectorTransportPort = 20 observationDomainId = 10 Session#c Data Record: Field Count = 13 Scope Count = 1 templateId = XXX exporterIPv4Address = 1.1.1.1 collectorIPv4Address = 2.2.2.2 collectorTransportProtocol = 16 exporterTransportPort = 10 collectorTransportPort = 20 observationDomainId = 10 exporterIPv4Address = 2.2.2.2 collectorIPv4Address = 3.3.3.3 collectorTransportProtocol = 16 exporterTransportPort = 30 collectorTransportPort = 40 observationDomainId = 0 Kobayashi, et al. Expires December 25, 2007 [Page 21] Internet-Draft Reference Model for IPFIX Mediators June 2007 6.2. Usage of Scope Field An IPFIX Mediator needs to send forward a Options Template Records and associated Data Records from the Original Exporter. However, IPFIX Mediator can not export an original Option Template Records and associated Data Records without modification because changing a session from an Exporting Process to a Collecting Process causes the scope fields to become a useless value. When an IPFIX Mediator relays the Options Template Records that included Observation Domain ID as a scope field and associated Data Records, an IPFIX Mediator uses the Exporter Information Option Template. The Options Template Records that were created from an Original Exporter can use the entire fields of the Exporter Information Option template as multiple scope fields. The Options Template Records that were created from anIPFIX Mediator can uses the some fields of the Exporter Information Option template as multiple scope fields. An IPFIX Mediator needs to modify the associated Data Records according to the modified Options Template Record. However, if each node uses another field except for the Observation Domain ID as the scope, the scope field should be considered on a case-by-case basis. The following example describes the cascade connection of IPFIX Mediators. Router#1 and Mediator#1 export the Metering Process Statistics Option Template. Session#a Session#b Session#c Router --------> Mediator#1 --------> Mediator#2 -------->Collector IP:1.1.1.1 IP:2.2.2.2 IP:3.3.3.3 SrcPort:10 DstPort:20 DstPort:40 ODID:10 SrcPort:30 SrcPort:50 ODID:0 ODID:0 Figure H: Cascade connection of IPFIX Mediators. Mediator#2 exports each Option Template and its Data Record with a suitable scope. Session#c Metering Process Statistics Data Records from the Original Exporter: Field Count = 15 Scope Count = 12 exporterIPv4Address = 1.1.1.1 collectorIPv4Address = 2.2.2.2 collectorTransportProtocol = 16 exporterTransportPort = 10 collectorTransportPort = 20 observationDomainId = 10 Kobayashi, et al. Expires December 25, 2007 [Page 22] Internet-Draft Reference Model for IPFIX Mediators June 2007 exporterIPv4Address = 2.2.2.2 collectorIPv4Address = 3.3.3.3 collectorTransportProtocol = 16 exporterTransportPort = 30 collectorTransportPort = 40 observationDomainId = 0 exportedMessageTotalCount exportedFlowTotalCount exportedOctetTotalCount Kobayashi, et al. Expires December 25, 2007 [Page 23] Internet-Draft Reference Model for IPFIX Mediators June 2007 7. Security Considerations The IPFIX concentrator uses the IPFIX protocol. Security considerations about flow information are described in [I-D.ietf-ipfix-protocol]. Kobayashi, et al. Expires December 25, 2007 [Page 24] Internet-Draft Reference Model for IPFIX Mediators June 2007 8. IANA Considerations This document has no actions for IANA. Kobayashi, et al. Expires December 25, 2007 [Page 25] Internet-Draft Reference Model for IPFIX Mediators June 2007 9. References 9.1. Normative References [RFC2119] Bradner, S., "Key words for use in RFCs to Indicate Requirement Levels", BCP 14, RFC 2119, March 1997. 9.2. Informative References [I-D.dressler-ipfix-aggregation] Dressler, F., Sommer, C., and G. Munz, "IPFIX Aggregation", draft-dressler-ipfix-aggregation-03.txt (work in progress) , June 2006. [I-D.ietf-ipfix-architecture] Sadasivan, G., Brownlee, N., Claise, B., and J. Quittek, "Architecture for IP Flow Information Export", draft-ietf-ipfix-architecture-12.txt(work in progress) , September 2006. [I-D.ietf-ipfix-info] Quittek, J., Bryant, S., Claise, B., and J. Meyer, "Information Model for IP Flow Information Export", draft-ietf-ipfix-info-13.txt(work in progress) , June 2006. [I-D.ietf-ipfix-protocol] Claise, B., "IPFIX Protocol Specification", draft-ietf-ipfix-protocol-23 (work in progress) , October 2006. [I-D.ietf-psamp-framework] Duffield, N., "A Framework for Packet Selection and Reporting", draft-ietf-psamp-framework-10.txt , January 2005. [I-D.ietf-psamp-sample-tech] Zseby, T., Molina, M., Duffield, N., Niccolini, S., and F. Raspall, "Sampling and Filtering Techniques for IP Packet Selection", draft-ietf-psamp-sample-tech-07.txt , July 2005. [RFC3917] Quittek, J., Zseby, T., Claise, B., and S. Zander, "Requirements for IP Flow Information Export(IPFIX)", October 2004. Kobayashi, et al. Expires December 25, 2007 [Page 26] Internet-Draft Reference Model for IPFIX Mediators June 2007 Authors' Addresses Atsushi Kobayashi NTT Information Sharing Platform Laboratories 3-9-11 Midori-cho Musashino-shi, Tokyo 180-8585 Japan Phone: +81-422-59-3978 Email: akoba@nttv6.net Keisuke Ishibashi NTT Information Sharing Platform Laboratories 3-9-11 Midori-cho Musashino-shi, Tokyo 180-8585 Japan Phone: +81-422-59-3407 Email: ishibashi.keisuke@lab.ntt.co.jp Kondoh Tsuyoshi NTT Information Sharing Platform Laboratories 3-9-11 Midori-cho Musashino-shi, Tokyo 180-8585 Japan Phone: +81-422-59-2419 Email: kondoh.tsuyoshi@lab.ntt.co.jp Daisuke Matsubara Hitachi, Ltd., Central Reseach Laboratory 1-280 Higashi-koigakubo Kokubunji-shi, Tokyo 185-8601 Japan Phone: +81-42-323-1111 Email: d-matuba@crl.hitachi.co.jp Kobayashi, et al. Expires December 25, 2007 [Page 27] Internet-Draft Reference Model for IPFIX Mediators June 2007 Full Copyright Statement Copyright (C) The IETF Trust (2007). This document is subject to the rights, licenses and restrictions contained in BCP 78, and except as set forth therein, the authors retain all their rights. This document and the information contained herein are provided on an "AS IS" basis and THE CONTRIBUTOR, THE ORGANIZATION HE/SHE REPRESENTS OR IS SPONSORED BY (IF ANY), THE INTERNET SOCIETY, THE IETF TRUST AND THE INTERNET ENGINEERING TASK FORCE DISCLAIM ALL WARRANTIES, EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO ANY WARRANTY THAT THE USE OF THE INFORMATION HEREIN WILL NOT INFRINGE ANY RIGHTS OR ANY IMPLIED WARRANTIES OF MERCHANTABILITY OR FITNESS FOR A PARTICULAR PURPOSE. Intellectual Property The IETF takes no position regarding the validity or scope of any Intellectual Property Rights or other rights that might be claimed to pertain to the implementation or use of the technology described in this document or the extent to which any license under such rights might or might not be available; nor does it represent that it has made any independent effort to identify any such rights. Information on the procedures with respect to rights in RFC documents can be found in BCP 78 and BCP 79. Copies of IPR disclosures made to the IETF Secretariat and any assurances of licenses to be made available, or the result of an attempt made to obtain a general license or permission for the use of such proprietary rights by implementers or users of this specification can be obtained from the IETF on-line IPR repository at http://www.ietf.org/ipr. The IETF invites any interested party to bring to its attention any copyrights, patents or patent applications, or other proprietary rights that may cover technology that may be required to implement this standard. Please address the information to the IETF at ietf-ipr@ietf.org. Acknowledgment Funding for the RFC Editor function is provided by the IETF Administrative Support Activity (IASA). Kobayashi, et al. Expires December 25, 2007 [Page 28]