Internet Engineering Task Force L. Peluso Internet-Draft University of Napoli Intended status: Standards Track T. Zseby Expires: September 7, 2010 Fraunhofer Institute FOKUS S. D'Antonio CINI Consortium/University of Napoli "Parthenope" M. Molina DANTE March 06, 2010 Flow Selection Techniques draft-ietf-ipfix-flow-selection-tech-01.txt Abstract Flow selection is the process of selecting a subset of flows from all flows observed at an observation point. The objective of flow selection is to reduce the effor for post-processing flow data and for transfering flow records. The flow selection process can be enabled at different stages of the measurement process. It can be applied directly after classification or at recording/exporting time by limiting the number of flows to be stored and/or exported to the collecting process. This document describes motivations for flow selection and presents flow selection techniques. It furthermore provides an information model for configuring flow selection techniques and discusses what information about a flow selection process is beneficial to be exported by adopting a suitable information model. Requirements Language The key words "MUST", "MUST NOT", "REQUIRED", "SHALL", "SHALL NOT", "SHOULD", "SHOULD NOT", "RECOMMENDED", "MAY", and "OPTIONAL" in this document are to be interpreted as described in RFC 2119 [RFC2119]. Status of this Memo This Internet-Draft is submitted to IETF in full conformance with the provisions of BCP 78 and BCP 79. Internet-Drafts are working documents of the Internet Engineering Task Force (IETF), its areas, and its working groups. Note that other groups may also distribute working documents as Internet- Drafts. Internet-Drafts are draft documents valid for a maximum of six months Peluso, et al. Expires September 7, 2010 [Page 1] Internet-Draft Flow Selection Techniques March 2010 and may be updated, replaced, or obsoleted by other documents at any time. It is inappropriate to use Internet-Drafts as reference material or to cite them other than as "work in progress." The list of current Internet-Drafts can be accessed at http://www.ietf.org/ietf/1id-abstracts.txt. The list of Internet-Draft Shadow Directories can be accessed at http://www.ietf.org/shadow.html. This Internet-Draft will expire on September 7, 2010. Copyright Notice Copyright (c) 2010 IETF Trust and the persons identified as the document authors. All rights reserved. This document is subject to BCP 78 and the IETF Trust's Legal Provisions Relating to IETF Documents (http://trustee.ietf.org/license-info) in effect on the date of publication of this document. Please review these documents carefully, as they describe your rights and restrictions with respect to this document. Code Components extracted from this document must include Simplified BSD License text as described in Section 4.e of the Trust Legal Provisions and are provided without warranty as described in the BSD License. This document may contain material from IETF Documents or IETF Contributions published or made publicly available before November 10, 2008. The person(s) controlling the copyright in some of this material may not have granted the IETF Trust the right to allow modifications of such material outside the IETF Standards Process. Without obtaining an adequate license from the person(s) controlling the copyright in such materials, this document may not be modified outside the IETF Standards Process, and derivative works of it may not be created outside the IETF Standards Process, except to format it for publication as an RFC or to translate it into languages other than English. Peluso, et al. Expires September 7, 2010 [Page 2] Internet-Draft Flow Selection Techniques March 2010 Table of Contents 1. Introduction . . . . . . . . . . . . . . . . . . . . . . . . . 4 2. Scope . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4 3. Terminology . . . . . . . . . . . . . . . . . . . . . . . . . 5 4. Position of the Flow Selection Process . . . . . . . . . . . . 5 5. Flow selection techniques . . . . . . . . . . . . . . . . . . 7 5.1. Flow selection based on flow record content . . . . . . . 7 5.2. Flow selection based on flow record arrival time or sequence . . . . . . . . . . . . . . . . . . . . . . . . . 7 5.3. Flow selection on external events . . . . . . . . . . . . 7 6. Reporting of Flow Selection Information . . . . . . . . . . . 7 6.1. Flow selection in the metering process . . . . . . . . . . 8 6.2. Flow selection in the flow recording process . . . . . . . 8 6.3. Flow selection in the exporting process . . . . . . . . . 9 7. Information model for flow selection information exporting . . 11 7.1. Meter process related (TBD1-TBD2) . . . . . . . . . . . . 12 7.1.1. FsMeter_UnmeasPacketCount . . . . . . . . . . . . . . 13 7.1.2. FsMeter_UnmeasBytesCount . . . . . . . . . . . . . . . 13 7.2. Flow recording process related (TBD3-TBD8) . . . . . . . . 13 7.2.1. FsFrec_PacketInDroppedRecsCount . . . . . . . . . . . 14 7.2.2. FsFrec_ByteInDroppedRecsCount . . . . . . . . . . . . 14 7.2.3. FsFrec_FrecDroppedCount . . . . . . . . . . . . . . . 14 7.2.4. FsFrec_UnexportedFrecCount . . . . . . . . . . . . . . 15 7.2.5. FsFrec_UnexportedPacketInFrecCount . . . . . . . . . . 15 7.2.6. FsFrec_UnexportedBytesInFrecCount . . . . . . . . . . 15 7.3. Flow exporting process related (TBD9-TBD14) . . . . . . . 16 7.3.1. FsExp_PacketInDroppedRecsCount . . . . . . . . . . . . 16 7.3.2. FsExp_ByteInDroppedRecsCount . . . . . . . . . . . . . 16 7.3.3. FsExp_FrecDroppedCount . . . . . . . . . . . . . . . . 17 7.3.4. FsExp_UnexportedCount . . . . . . . . . . . . . . . . 17 7.3.5. FsExp_UnexportedPacketCount . . . . . . . . . . . . . 17 7.3.6. FsExp_UnexportedByteInExpCount . . . . . . . . . . . . 18 8. Requirements put on implementations . . . . . . . . . . . . . 18 9. Information Model for Configuration of Flow Selection Techniques . . . . . . . . . . . . . . . . . . . . . . . . . . 19 9.1. selectorMethod . . . . . . . . . . . . . . . . . . . . . . 19 9.2. flowMaxAdmitFlowRecords . . . . . . . . . . . . . . . . . 20 9.3. flowRecordBytesSize . . . . . . . . . . . . . . . . . . . 20 9.4. flowRecordPacketsSize . . . . . . . . . . . . . . . . . . 21 9.5. flowInactivityTime . . . . . . . . . . . . . . . . . . . . 21 10. Security Considerations . . . . . . . . . . . . . . . . . . . 22 11. IANA Considerations . . . . . . . . . . . . . . . . . . . . . 22 12. References . . . . . . . . . . . . . . . . . . . . . . . . . . 22 12.1. Normative References . . . . . . . . . . . . . . . . . . . 22 12.2. Informative References . . . . . . . . . . . . . . . . . . 22 Authors' Addresses . . . . . . . . . . . . . . . . . . . . . . . . 23 Peluso, et al. Expires September 7, 2010 [Page 3] Internet-Draft Flow Selection Techniques March 2010 1. Introduction This document describes flow selection techniques for traffic measurements. [RFC5475], describes packet selection techniques, which describe sampling and filtering techniques for selecting a subset of packets The element on which this selection mechanism is performed is a packet and the selection decision is based on packet propeorties in contrats to this, flow selection techniques consider flows as the basic elements on which a selection process is performed In the IPFIX architecture the basis element on which the selection process is performed are the IPFIX flow records. For several applications it makes sense to select only the flows of interest if resources are scarce. Examples are accounting or attack detection applications. One example are attack detection techniques. If an attack is ongoing in the network, resources are quite scarce. Maintainign and exporting all flow records to the collecting process would increse resource demands even further with the result that data is randomly discarded. A better solution is to export only a representative subset of flows. For some botnet attacks many similar flows occur from different source addresses. So it can be useful to include some meta-data ,to indicate that there are multiple flows active with similar characteristics. Another example are accounting applications. In many networks few large flows contribute to the majority of the overall traffic volume [DuLT01a], [DuLT01b]. This phenomenon is also referred to as "Quasi-Zipf-Law" [KuXW04] or as "elephant and mice phenomenon". For accounting puposes it may be useful to concentrate on the large so-called "heavy hitter" flows to cope with a limited flow cache size or limited transmission capacity in times when resources are scarce. 2. Scope This document describes flow selection techniques and its parameters. It addresses the configuration of flow selection techniques and defines which information should be reported by devices that perform flow selection. It only describes processes directly acting on traffic flows during the metering phase and/or the exporting phase. Therefore it is assumed that flow selection is performed after packets are classified into flows. This document does not address the flow selection effects that might result from the sampling or filtering of packets in the metering process before the classification process is performed. Such packet selection techniques are described in [RFC5475] and, therefore, outside the scope of this document. Peluso, et al. Expires September 7, 2010 [Page 4] Internet-Draft Flow Selection Techniques March 2010 3. Terminology This document uses the terminology introduced in [RFC5470] and [RFC5475] In this section, some additional terms are presented which extend the terminology introduced in [RFC5475]. * Flow Selection Process A Flow Selection Process takes a set of Flow Records as its input and selects a subset of that set as its output. * Flow Selection State A Flow Selection Process may maintain state information for use by the Flow Selection Process. At a given time, the Flow Selection State may depend on flows observed at and before that time, and other variables. Examples include: (i) number of accounted flow records; (ii) memory space available for flow recording; (iii) state of the pseudorandom number generators; (iv) hash values calculated during selection. * Flow Selector A Flow Selector defines the action of a Flow Selection Process on a single flow of its input. The Flow Selector can make use of the following information in determining whether a flow is selected: (i) the content of the flow record; (ii) any information state related to the flow recording; (iii) any selection state that may be maintained by the Flow Selection Process. 4. Position of the Flow Selection Process Figure 1 shows the IPFIX reference model as defined in [RFC5470], and extends it by introducing the functional components where flow selection can take place. Peluso, et al. Expires September 7, 2010 [Page 5] Internet-Draft Flow Selection Techniques March 2010 Packet(s) coming in to Observation Point(s) | | v v +----------------+---------------------------+ +-----+-------+ | Metering Process on an | | | | Observation Point | | | | | | | | packet header capturing | | | | | |...| Metering | | timestamping | | Process N | | | | | | | packet selection | | | | | | | | | classification | | | | | | | | | field match fitering on flow keys or | | | | flow state dependent packet sampling (*) | | | | | | | | | aggregation | | | | | | | | | flow recording (*) | | | | | | | | | | Timing out Flows | | | | | Handle resource overloads | | | +--------|-----------------------------------+ +-----|-------+ | | Flow Records (selected by Observation Domain) Flow Records | | +----------------------+----------------------+ | +----------------------|---------------+ | Exporting Process v | | +---------------+-----------+ | | | flow export (*) | | | +---------------+-----------+ | | | | +----------------------+---------------+ | v IPFIX export packet to Collector (*) indicates where flow selection can take place. Figure 1 In contrast to packet selection, flow selection is always applied after the packets are classified into flows. Flows can be selected at different stages of the measurement chain: Peluso, et al. Expires September 7, 2010 [Page 6] Internet-Draft Flow Selection Techniques March 2010 1. during metering [RFC5475] 2. during flow recording 3. during flow export 5. Flow selection techniques We can distinguish the following selection techniques: 1. based on flow record content (i.e. flow characteristics that are described in the flow report); 2. based on flow record arrival time; 3. based on external events like the exhaustion of local resources. 5.1. Flow selection based on flow record content Flow selection can be done based on fields in an IPFIX flow record. This can be done analogous to field match filtering for packet selection described in [RFC5475]. The difference here is that instead of packets here field of the flow record content are used for the selection decision. An example would be to select flow records with regard to the flow size in bytes or number of packets. Another application would be to select flow records based on flow start time or on flow keys (IP addresses, ports) of the stored flow record. 5.2. Flow selection based on flow record arrival time or sequence Flow records can be selected based on their arrival time at the exporting process. An example would be to select a number of flow records for certain periods of time. Another option is to select flow records based on the order at which they arrive at the exporting process. With this one can select systematically every kth record or select randomly a set of flow records. 5.3. Flow selection on external events The selection of flow records can be alsio triggered by external events. An example would be router state like number of entries in flow cache. 6. Reporting of Flow Selection Information In this section we identify and describe in more detail some possible Peluso, et al. Expires September 7, 2010 [Page 7] Internet-Draft Flow Selection Techniques March 2010 causes of flow selection, along with the information that can be beneficial to make available to applications about it. 6.1. Flow selection in the metering process The main reason for applying in the metering process a flow state dependent sampling is that the flow recording process may not have, at a certain point in time, enough positions to record all observable flows. Another reason may be that there may not be enough processing resources to create and manage a new flow record. To overcome with these limitations, a number of possible policies can be applied, the simplest one being not to consider for measurement the new packets that do not belong to already existing flow records (i.e. that would require the creation of a new one). More refined policies are however possible, mainly aimed at the so called elephant flow detection, i.e. to give priority in the flow recording process to flows carrying more traffic. For instance, [EsVa01] proposes criteria to define a packet eligible to create a new flow record (sample and hold, multistage filters). Independently of the specific algorithms, we are concerned here about defining what information it makes sense to keep about the flow state dependent packet sampling and make available to applications (by exporting it out of an IPFIX device). It is certainly possible to keep a cumulative counter of the total number of packets and bytes that were not considered for measurement because of flow state dependent sampling. Also, it is possible to keep a timestamp for the first and last of these non measured packets. This means, in practice, to aggregate all these packets in a macro flow, and keep track of its volume and duration. Imagining keeping more detailed information about packets not measured because of flow state dependent sampling would contradict the fact that the sampling is done because of lack of memory and/or processing resources. 6.2. Flow selection in the flow recording process This block is optional in the IPFIX framework architecture. However, we address here the case where it is present. We already described in the previous section that because of lack of memory positions in the flow recording process some incoming packets may be discarded if they lead to the opening of a new flow record. However, under certain circumstances, it may be advantageous to discard an existing flow record in the flow recording process to make room for the new record opened by an arriving packet. For example, an algorithm for taking the decision whether to discard the new arriving packet or an existing flow record is described in [Moli03]. In this section we are not concerned about the algorithm details but about what information to store about this record removal. For the same reasons expressed before, we argue that it does not make sense to store Peluso, et al. Expires September 7, 2010 [Page 8] Internet-Draft Flow Selection Techniques March 2010 separate information for each discarded flow record, as it would contradict the motivation itself for which the discarding is done (i.e. lack of memory resources). The information that is certainly possible to keep with a limited effort is a cumulative counter of the total number of not yet exported packets and bytes belonging to flow records that were eliminated from the flow recording process. Ideally, we would also like to keep a timestamp for the first (T_fd) and last (T_ld) not yet exported packets belonging to all these discarded flow records. This would mean, in practice, to aggregate all these packets in a macro flow, and keep track of its volume and duration. To do so precisely, we would need to keep in each flow record a timestamp for the first and last non-exported packets, and whenever a record is discarded look at these timestamps to see if they are smaller or larger (respectively) of T_fd and T_ld and if yes update them. Another information that can be easily kept is the number of these discarding events, along with a timestamp of the first and last of them. This information should not be used by applications to re- normalize their received per flow statistics (because a flow may be discarded and re-created multiple times) but rather to keep under control the good functioning of the implemented policy. Note that we consider a discarding event only when the discarded flow record contains some not exported traffic. Otherwise, the removal of a record whose traffic was fully exported (after a timeout or after the arrival of specific packets, e.g. TCP FIN or RST) is part of the normal functioning of an IPFIX flow metering system. Note also that we consider only the case when an elimination of a flow record from the flow recording process leads to the complete loss of all the information contained in the flow record. If on the contrary another policy is implemented, like immediate exporting of the flow record before elimination, or freezing of the flow record and moving it in an area of memory different from which is considered the flow recording process for later exporting, this is not considered an elimination and therefore is out of the scope of this document. In parallel to the information about the number of discarded flow records and associated packets and bytes, it is useful to keep cumulative information about the number of flow records containing not yet exported traffic that exist in the flow recording process, along with the cumulative number of not exported packets and bytes contained in them. This information is useful also for exporting process related reasons, as clarified in the following paragraph. 6.3. Flow selection in the exporting process The exporting process may implement policies for not exporting the whole set of flow records of the flow recording process. In case of absence of the flow recording process, when the metering process directly feeds the exporting process (i.e. directly put the exported Peluso, et al. Expires September 7, 2010 [Page 9] Internet-Draft Flow Selection Techniques March 2010 packets in the IPFIX format), the following reasoning does not apply. The motivations for not exporting some flow records (containing non exported traffic) can be two: there are explicit configured policies or the exporting process faces resource limitation. An example of explicit policy can be not to export the flows whose accounted traffic is below a certain threshold, or a more complex mechanism such as the one described in [DuLT01a] or [DuLT01b]. An example of resource limitation is that the exporting process has an assigned, limited time slot to operate or a limited predefined number of export packets that it can send. There can also be hybrid cases where there are resource limitations and policies are applied in order to optimize the exported information (e.g. given that we want to export only N flow records, select a subset so that the overall number of reported packets and bytes belonging to the subset is maximized). Coming to the issue of which information it makes sense to keep about this flow selection, there are two cases to consider. If a flow is not exported and because of this decision is deleted from the flow recording process, we are in the same case described before (where the deletion was triggered by the need to make room for another record). The information to keep is then naturally the same as described before (cumulative packets and bytes for all the flows not exported, timestamps of the first and last packets belonging to non exported flow records, counter of dropping events and timestamp of first and last dropping event). Only the reason for this removal is different. If on the contrary a record eligible for exporting is not exported but it remains in the flow recording process it has always a chance to be exported in the future. For an application, however, it would be beneficial to know what it is not currently being exported because of exporting process policies/resource limitations, in terms of flow records, packets and bytes. This, not to re-normalize its estimates (it would be dangerous and error prone because the exporting of these records may be simply delayed), but rather to keep under control what is happening: for example, understand if there are pathologic situations where a large number of flow records and/or associated traffic are never exported, or if the number of flow records in the flow recording process is growing, etc. When it comes to understanding if this information can be easily available, however, we recognize that there is the problem that in order to be aware that it has not exported a flow record, an exporting process should at least have browsed through it. In other words, we would have to assume that there is always a full scanning of the flow recording process associated to the exporting process selection decision. However, there may be more efficient implementations where this does not happen. Therefore, even if we provide support in the information model for this information, defining it as mandatory in the protocol definition would put a constraint on the exporting process implementation, which is undesirable. Peluso, et al. Expires September 7, 2010 [Page 10] Internet-Draft Flow Selection Techniques March 2010 7. Information model for flow selection information exporting We formally define the elements to contain the information described in the previous section. Some elements have an associated couple of timestamps, which we reference for brevity (when it is not ambiguous) as Tfirst and Tlast (instead of element_nameTfirst, element_nameTlast). Note that all the following information elements are aimed at describing macro flows (e.g. the total number of packets and bytes contained in all dropped or not created flow records). Some of these macro flows are additive only, in the sense that they only add contributions to them, but never subtract. E.g. the macro flow of the packets contained in flow records that are discarded from the flow reporting process receives a contribution when a flow record is discarded, and this contribution can never be subtracted. On the contrary, some of the macro flows can dynamically receive and loose contributions. E.g. the macro flows of packets not yet exported receives a contribution when a new packets arrives, and looses some contribution when there is an exporting event. Associating a timestamp for the oldest and most recent contributions to additive only flow is easy, while for the others is not (would require to maintain full state) and that is why we did not define timestamps for these information elements. The information elements here introduced are defined in accordance with the IPFIX information model [RFC5102] to which reference should be made for more detailed information. Furthermore, the data types used to formally rappresent the Flow Selection related information elements are those defined in section 3.1 of the IPFIX information model [RFC 2051]. For that reason, they are not redefined in this section. List of additional Flow Selection information elements: Peluso, et al. Expires September 7, 2010 [Page 11] Internet-Draft Flow Selection Techniques March 2010 +-------+------------------------------------+ | ID | Name | +-------+------------------------------------+ | TBD1 | FsMeter_UnmeasPacketCount | +-------+------------------------------------+ | TBD2 | FsMeter_UnmeasBytesCount | +-------+------------------------------------+ | TBD3 | FsFrec_PacketInDroppedRecsCount | +-------+------------------------------------+ | TBD4 | FsFrec_ByteInDroppedRecsCount | +-------+------------------------------------+ | TBD5 | FsFrec_FrecDroppedCount | +-------+------------------------------------+ | TBD6 | FsFrec_UnexportedFrecCount | +-------+------------------------------------+ | TBD7 | FsFrec_UnexportedPacketInFrecCount | +-------+------------------------------------+ | TBD8 | FsRec_UnexportedBytesInFrecCount | +-------+------------------------------------+ | TBD9 | FsExp_PacketInDroppedRecsCount | +-------+------------------------------------+ | TBD10 | FsExp_BytesInDroppedRecsCount | +-------+------------------------------------+ | TBD11 | FsExp_FrecDroppedCount | +-------+------------------------------------+ | TBD12 | FsExp_UnexportedCount | +-------+------------------------------------+ | TBD13 | FsExp_UnexportedPacketCount | +-------+------------------------------------+ | TBD14 | FsExp_UnexportedByteInExpCount | +-------+------------------------------------+ 7.1. Meter process related (TBD1-TBD2) Information Elements in this section are related to Flow Selection at the Matering Process. +------+---------------------------+ | ID | Name | +------+---------------------------+ | TBD1 | FsMeter_UnmeasPacketCount | +------+---------------------------+ | TBD2 | FsMeter_UnmeasBytesCount | +------+---------------------------+ Peluso, et al. Expires September 7, 2010 [Page 12] Internet-Draft Flow Selection Techniques March 2010 7.1.1. FsMeter_UnmeasPacketCount Contains the count of packets that were not measured because of flow state dependent sampling, in terms of: TsFirst: timestamp of the first packet not measured because of flow state dependent sampling (Type: dateTime) TsLast: timestamp of the last packet not measured because of flow state dependent sampling (Type: dataTime) 7.1.2. FsMeter_UnmeasBytesCount Description: This Information Elements contains the count of bytes that were not measured because of flow state dependent sampling Abstract Data Type: unsigned64 Data Type Semantics: quantity ElementId: TBD2 Status: Proposed Units: bytes 7.2. Flow recording process related (TBD3-TBD8) Information Elements in this section are related to Flow Selection at the Flow Recording Process if present. +------+------------------------------------+ | ID | Name | +------+------------------------------------+ | TBD3 | FsFrec_PacketInDroppedRecsCount | +------+------------------------------------+ | TBD4 | FsFrec_ByteInDroppedRecsCount | +------+------------------------------------+ | TBD5 | FsFrec_FrecDroppedCount | +------+------------------------------------+ | TBD6 | FsFrec_UnexportedFrecCount | +------+------------------------------------+ | TBD7 | FsFrec_UnexportedPacketInFrecCount | +------+------------------------------------+ | TBD8 | FsFrec_UnexportedBytesInFrecCount | +------+------------------------------------+ Peluso, et al. Expires September 7, 2010 [Page 13] Internet-Draft Flow Selection Techniques March 2010 7.2.1. FsFrec_PacketInDroppedRecsCount Contains the count of non exported packets that were contained in flow records eliminated from the flow recording process because of resource limitations/policies in the flow recording process. It is defined in terms of: TsFirst: timestamp of the first non-exported packet belonging to a eliminated flow record (Type: dateTime) TsLast: timestamp of the last non-exported packet belonging to a eliminated flow record (Type: dateTime) 7.2.2. FsFrec_ByteInDroppedRecsCount Description: This Information Elements contains the count of non exported bytes that were contained in flow records eliminated from the flow recording process because of resource limitations/policies in the flow recording process. Abstract Data Type: unsigned64 Data Type Semantics: quantity ElementId: TBD4 Status: Proposed Units: bytes 7.2.3. FsFrec_FrecDroppedCount Contains the count of flow records containing non exported packets eliminated from the flow recording process because of resources limitations/policies in the flow recording process. It is defined in terms of: TsFirst: timestamp of the first flow record elimination event from the flow recording process (Type: dateTime) TsLast: timestamp of the last flow record elimination event from the flow recording process (Type: dateTime) Peluso, et al. Expires September 7, 2010 [Page 14] Internet-Draft Flow Selection Techniques March 2010 7.2.4. FsFrec_UnexportedFrecCount Description: This Information Elements contains the count of the flow records currently existing in the flow recording process containing at least one non exported packet. Abstract Data Type: unsigned32 Data Type Semantics: quantity ElementId: TBD6 Status: Proposed Units: flow records 7.2.5. FsFrec_UnexportedPacketInFrecCount Description: This Information Elements contains the count of non exported packets contained in flow records of the flow recording process. Abstract Data Type: unsigned32 Data Type Semantics: quantity ElementId: TBD7 Status: Proposed Units: packets 7.2.6. FsFrec_UnexportedBytesInFrecCount Description: This Information Elements contains the count of non exported bytes contained in flow records of the flow recording process. Abstract Data Type: unsigned64 Data Type Semantics: quantity ElementId: TBD8 Peluso, et al. Expires September 7, 2010 [Page 15] Internet-Draft Flow Selection Techniques March 2010 Status: Proposed Units: bytes 7.3. Flow exporting process related (TBD9-TBD14) Information Elements in this section are related to Flow Selection at the Flow Exporting Process. +-------+--------------------------------+ | ID | Name | +-------+--------------------------------+ | TBD9 | FsExp_PacketInDroppedRecsCount | +-------+--------------------------------+ | TBD10 | FsExp_ByteInDroppedRecsCount | +-------+--------------------------------+ | TBD11 | FsExp_FrecDroppedCount | +-------+--------------------------------+ | TBD12 | FsExp_UnexportedCount | +-------+--------------------------------+ | TBD13 | FsExp_UnexportedPacketCount | +-------+--------------------------------+ | TBD14 | FsExp_UnexportedByteInExpCount | +-------+--------------------------------+ 7.3.1. FsExp_PacketInDroppedRecsCount Contains the count of non exported packets that were contained in flow records eliminated from the flow recording process because of resource limitations/policies in the exporting process. It is defined in terms of: TsFirst: timestamp of the first non exported packet belonging to a eliminated flow record (Type: dateTime) TsLast: timestamp of the last non exported packet belonging to a eliminated flow record (Type: dateTime) 7.3.2. FsExp_ByteInDroppedRecsCount Description: This Information Elements contains the count of non exported bytes that were contained in flow records eliminated from the flow recording process because of resource limitations/policies in the exporting process. Abstract Data Type: unsigned64 Peluso, et al. Expires September 7, 2010 [Page 16] Internet-Draft Flow Selection Techniques March 2010 Data Type Semantics: quantity ElementId: TBD10 Status: Proposed Units: bytes 7.3.3. FsExp_FrecDroppedCount Contains the count of flow records containing non exported packets eliminated from the flow recording process because of resource limitations/policies in the exporting process. It is defined in terms of: TsFirst: timestamp of the first flow record elimination event from the flow recording process (Type: dateTime) TsLast: timestamp of the last flow record elimination event from the flow recording process (Type: dateTime) 7.3.4. FsExp_UnexportedCount Description: This Information Elements contains the count of the flow records currently existing in the flow recording process containing non- exported traffic and not being exported because of exporting process resource lmitations/policies. Abstract Data Type: unsigned32 Data Type Semantics: quantity ElementId: TBD12 Status: Proposed Units: flow records 7.3.5. FsExp_UnexportedPacketCount Description: This Information Elements contains the count of non exported packets contained in flow records of the flow recording process not being exported because of exporting process resource limitations/policies. Peluso, et al. Expires September 7, 2010 [Page 17] Internet-Draft Flow Selection Techniques March 2010 Abstract Data Type: unsigned32 Data Type Semantics: quantity ElementId: TBD13 Status: Proposed Units: packets 7.3.6. FsExp_UnexportedByteInExpCount Description: This Information Elements contains the count of non exported bytes contained in flow records of the flow recording process not being exported because of exporting process resource limitations/ policies. Abstract Data Type: unsigned64 Data Type Semantics: quantity ElementId: TBD14 Status: Proposed Units: bytes 8. Requirements put on implementations To support the described information model an implementation must keep, in the flow records, counts for non-exported packets and bytes. Sometimes these are referred as delta counts. An implementation may also keep absolute counts for scopes not specified in this information model (it appears that both delta and absolute counters can be exported in the IPFIX information model, see [RFC5102]). In addition, to fully support this information model, it would be required to keep in a flow record a timestamp for the first and last non-exported packets. An implementation may need to keep timestamps for the first and last exported packets as well for scopes not specified in this information model, or to join the two timers for the last exported and first exported packets (which is of course an approximation) or to approximate them with the time of the exporting event. Peluso, et al. Expires September 7, 2010 [Page 18] Internet-Draft Flow Selection Techniques March 2010 9. Information Model for Configuration of Flow Selection Techniques This section aims at describing the representative parameters of the above presented flow selection techniques. To this regard, it provides the basis for an information model to adopt in order to configure the flow selection process at an IPFIX device. The information elements here introduced are defined in accordance with the IPFIX information model [RFC5102] to which reference should be made for more detailed information. Furthermore, the data types used to formally rappresent the Flow Selection related information elements are those defined in section 3.1 of the IPFIX information model [RFC 2051]. For that reason, they are not redefined in this section. List of additional Flow Selection information elements: +-------+-------------------------+-------+-----------------------+ | ID | Name | ID | Name | +-------+-------------------------+-------+-----------------------+ | TBD15 | selectorMethod | TBD18 | flowRecordPacketsSize | +-------+-------------------------+-------+-----------------------+ | TBD16 | flowMaxAdmitFlowRecords | TBD19 | flowInactivityTime | +-------+-------------------------+-------+-----------------------+ | TBD17 | flowRecordBytesSize | ... | ... | +-------+-------------------------+-------+-----------------------+ 9.1. selectorMethod Description: This Information Element identifies the flow selection method that are applied by the Flow Selection process, in accordance to what described in the section 5 of this document. Same of these methods may have parameters in order to fully support the selected technique. For that reason, further Information Elements are defined in the following subsections. The following flow selection methods identifiers are defined here: +----+----------------------------+---------------------------------+ | ID | Method | Parameters | +----+----------------------------+---------------------------------+ | 1 | Selection based on flow | flowMaxAdmitFlowRecords | | | size count | flowRecordBytesSize | | | | flowRecordPacketsSize | +----+----------------------------+---------------------------------+ Peluso, et al. Expires September 7, 2010 [Page 19] Internet-Draft Flow Selection Techniques March 2010 +----+----------------------------+---------------------------------+ | 2 | Selection based on flow | flowMaxAdmitFlowRecords | | | content property match | ........... | +----+----------------------------+---------------------------------+ | 3 | Selection based on flow | flowMaxAdmitFlowRecords | | | record arrival time or | flowInactivityTime | | | sequence | | +----+----------------------------+---------------------------------+ | 4 | Selection based on | flowMaxAdmitFlowRecords | | | external events | ........... | +----+----------------------------+---------------------------------+ Abstract Data Type: unsigned16 Data Type Semantics: identifier ElementId: TBD15 Status: Proposed 9.2. flowMaxAdmitFlowRecords Description: This Information Element specifies the maximum number of elegible flow records which might be created in to the flow cache. It is used by the Selector Process in order to identify the time when flow selection should be triggered. A value of 0 means that the Flow Selection State related to the memory space available for flow recording must be used to estimate the max flow cache size. For example, this Information Element may be used to describe the configuration of a flow size count Flow Selector. Abstract Data Type: unsigned32 Data Type Semantics: quantity ElementId: TBD16 Status: Proposed Units: flow records 9.3. flowRecordBytesSize Description: Peluso, et al. Expires September 7, 2010 [Page 20] Internet-Draft Flow Selection Techniques March 2010 This Information Element specifies the minimum number of bytes contained in a flow record to be considered not elegible for removal. It may be used in order to identify elephant flows. For example, this Information Element may be used to describe the configuration of a flow size count Flow Selector. Abstract Data Type: unsigned64 Data Type Semantics: quantity ElementId: TBD17 Status: Proposed Units: bytes 9.4. flowRecordPacketsSize Description: This Information Element specifies the minimum number of packets contained in a flow record to be considered not elegible for removal. It may be used in order to identify elephant flows. For example, this Information Element may be used to describe the configuration of a flow size count Flow Selector. Abstract Data Type: unsigned32 Data Type Semantics: quantity ElementId: TBD18 Status: Proposed Units: packets 9.5. flowInactivityTime Description: This Information Element specifies the time interval in microseconds during which the corresponding flow record may be considered still active. It is used by the metering process and/or the flow recording process in order to take the decision whether to discard an existing flow to make room for a new one. Peluso, et al. Expires September 7, 2010 [Page 21] Internet-Draft Flow Selection Techniques March 2010 For example, this Information Element may be used to describe the configuration of a flow arrival time Flow Selector. Abstract Data Type: dateTimeMicroseconds Data Type Semantics: quantity ElementId: TBD19 Status: Proposed Units: microseconds 10. Security Considerations This document descirbes methods for flow selection techniques that are applied in network measurements. If users know or can guess the selection policies they may craft flows in a way to avoid beeing selected. Furthermore network measurements are often used for the detecction of network attacks. Therefore it has to be taken into account that flow selection may remove flows that are of interest for the detection taks. [more here] 11. IANA Considerations This document introduces several new information elements as an extension to the IPFIX information model. IANA assignments should be created for the information elements described in this document. 12. References 12.1. Normative References [RFC2119] Bradner, S., "Key words for use in RFCs to Indicate Requirement Levels", BCP 14, RFC 2119, March 1997. 12.2. Informative References [DuLT01a] Duffield, N., Lund, C., and M. Thorup, "Charging from Sampled Network Usage", ACM Internet Measurement Workshop IMW 2001, San Francisco, USA, November 2001. [DuLT01b] Duffield, N., Lund, C., and M. Thorup, "Properties and Prediction of Flow Statistics from Sampled Packet Streams", ACM SIGCOMM Internet Measurement Workshop 2002, Peluso, et al. Expires September 7, 2010 [Page 22] Internet-Draft Flow Selection Techniques March 2010 November 2002. [DuLT01c] Duffield, N., Lund, C., and M. Thorup, "Learn More, sample less: control of volume and variance in network measurement", IEEE Transactions on Information Theory, May 2005. [DuLT01d] Duffield, N., Lund, C., and M. Thorup, "Flow Sampling under Hard Resource Constraints", ACM IFIP Conference on Measurement and Modeling of Computer Systems SIGMETRICS, June 2004. [EsVa01] Estan, C. and G,. Varghese, "New Directions in Traffic Measurement and Accounting: Focusing on the Elephants, Ignoring the Mice", ACM SIGCOMM Internet Measurement Workshop 2001, San Francisco (CA), November 2001. [FeGL98] Feldmann, A., Rexford, J., and R. Caceres, "Efficient Policies for Carrying Web Traffic over Flow-Switched Networks", IEEE/ACM Transaction on Networking, December 1998. [KuXW04] Kumar, K., Xu, J., Wang, J., Spatschek, O., and L. Li, "Space-code bloom filter for efficient per-flow traffic measurement", INFOCOM 2004 Twenty-third AnnualJoint Conference of the IEEE Computer and Communications Societies, March 2004. [Moli03] Molina, M., "A scalable and efficient methodology for flow monitoring in the Internet", International Teletraffic Congress (ITC-18), Berlin, September 2003. [RFC5102] Quittek, J., Bryant, S., Claise, B., Aitken, P., and J. Meyer, "Information Model for IP Flow Information Export", RFC 5102, January 2008. [RFC5470] Sadasivan, G., Brownlee, N., Claise, B., and J. Quittek, "Architecture for IP Flow Information Export", RFC 5470, March 2009. [RFC5475] Zseby, T., Molina, M., Duffield, N., Niccolini, S., and F. Raspall, "Sampling and Filtering Techniques for IP Packet Selection", RFC 5475, March 2009. Peluso, et al. Expires September 7, 2010 [Page 23] Internet-Draft Flow Selection Techniques March 2010 Authors' Addresses Lorenzo Peluso University of Napoli Via Claudio 21 Napoli 80125 Italy Phone: +39 081 7683821 Email: lorenzo.peluso@unina.it Tanja Zseby Fraunhofer Institute FOKUS Kaiserin-Augusta-Allee 31 Berlin 10589 Germany Phone: +49 30 3463 7153 Email: tanja.zseby@fokus.fraunhofer.de Salvatore D'Antonio CINI Consortium/University of Napoli "Parthenope" Monte S.Angelo, Via Cinthia Napoli 80126 Italy Phone: +39 081 679944 Email: saldanto@unina.it Maurizio Molina DANTE Hills Road 126-130 Cambridge CB2 1PQ United Kingdom Phone: +44 1223 371300 Email: maurizio.molina@dante.org.uk Peluso, et al. Expires September 7, 2010 [Page 24]