Internet Engineering Task Force Brownlee, Mills, Ruth INTERNET-DRAFT The University of Auckland July 98 Expires January 1999 Traffic Flow Measurement: Architecture Status of this Memo This document is an Internet-Draft. Internet-Drafts are working documents of the Internet Engineering Task Force (IETF), its Areas, and its Working Groups. Note that other groups may also distribute working documents as Internet-Drafts. This Internet Draft is a product of the Realtime Traffic Flow Measurement Working Group of the IETF. Internet Drafts are draft documents valid for a maximum of six months. Internet Drafts may be updated, replaced, or obsoleted by other documents at any time. It is not appropriate to use Internet Drafts as reference material or to cite them other than as a "working draft" or "work in progress." To view the entire list of current Internet-Drafts, please check the "1id-abstracts.txt" listing contained in the Internet-Drafts Shadow Directories on ftp.is.co.za (Africa), ftp.nordu.net (Northern Europe), ftp.nis.garr.it (Southern Europe), munnari.oz.au (Pacific Rim), ftp.ietf.org (US East Coast), or ftp.isi.edu (US West Coast). Abstract This document provides a general framework for describing network traffic flows, presents an architecture for traffic flow measurement and reporting, discusses how this relates to an overall network traffic flow architecture and indicates how it can be used within the Internet. Contents 1 Statement of Purpose and Scope 3 1.1 Brief Technical Specification (TS) . . . . . . . . . . . . . . 3 1.2 Applicability Statement (AS) . . . . . . . . . . . . . . . . . 3 1.3 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . 4 INTERNET-DRAFT Traffic Flow Measurement: Architecture July 98 2 Traffic Flow Measurement Architecture 6 2.1 Meters and Traffic Flows . . . . . . . . . . . . . . . . . . . 6 2.2 Interaction Between METER and METER READER . . . . . . . . . . 8 2.3 Interaction Between MANAGER and METER . . . . . . . . . . . . 8 2.4 Interaction Between MANAGER and METER READER . . . . . . . . . 9 2.5 Multiple METERs or METER READERs . . . . . . . . . . . . . . . 9 2.6 Interaction Between MANAGERs (MANAGER - MANAGER) . . . . . . . 10 2.7 METER READERs and APPLICATIONs . . . . . . . . . . . . . . . . 11 3 Traffic Flows and Reporting Granularity 11 3.1 Flows and their Attributes . . . . . . . . . . . . . . . . . . 11 3.2 Granularity of Flow Measurements . . . . . . . . . . . . . . . 14 3.3 Rolling Counters, Timestamps, Report-in-One-Bucket-Only . . . 15 4 Meters 17 4.1 Meter Structure . . . . . . . . . . . . . . . . . . . . . . . 18 4.2 Flow Table . . . . . . . . . . . . . . . . . . . . . . . . . . 19 4.3 Packet Handling, Packet Matching . . . . . . . . . . . . . . . 20 4.4 Rules and Rule Sets . . . . . . . . . . . . . . . . . . . . . 23 4.5 Maintaining the Flow Table . . . . . . . . . . . . . . . . . . 27 4.6 Handling Increasing Traffic Levels . . . . . . . . . . . . . . 28 5 Meter Readers 29 5.1 Identifying Flows in Flow Records . . . . . . . . . . . . . . 29 5.2 Usage Records, Flow Data Files . . . . . . . . . . . . . . . . 29 5.3 Meter to Meter Reader: Usage Record Transmission . . . . . . 30 6 Managers 31 6.1 Between Manager and Meter: Control Functions . . . . . . . . 31 6.2 Between Manager and Meter Reader: Control Functions . . . . . 32 6.3 Exception Conditions . . . . . . . . . . . . . . . . . . . . . 33 6.4 Standard Rule Sets . . . . . . . . . . . . . . . . . . . . . . 34 7 Security Considerations 35 7.1 Threat Analysis . . . . . . . . . . . . . . . . . . . . . . . 35 7.2 Countermeasures . . . . . . . . . . . . . . . . . . . . . . . 36 8 APPENDICES 38 8.1 Appendix A: Network Characterisation . . . . . . . . . . . . . 38 8.2 Appendix B: Recommended Traffic Flow Measurement Capabilities 39 8.3 Appendix C: List of Defined Flow Attributes . . . . . . . . . 40 8.4 Appendix D: List of Meter Control Variables . . . . . . . . . 41 8.5 Appendix E: Changes Introduced Since RFC 2063 . . . . . . . . 41 9 Acknowledgments 42 10 References 42 11 Author's Address 43 Brownlee, Mills, Ruth [Page 2] INTERNET-DRAFT Traffic Flow Measurement: Architecture July 98 1 Statement of Purpose and Scope 1.1 Brief Technical Specification (TS) RTFM provides for the measurement of network traffic FLOWS, i.e. - a method of specifying traffic flows within a network - a hierarchy of devices (meters, meter readers, managers) for measuring the specified flows - a mechanism for configuring meters and meter readers, and for collecting the flow data from remote meters The RTFM Meter is designed so as to do as much data reduction work as possible. This minimizes the amount of data to be read and the amount of processing needed to produce useful reports from it. RTFM flow data can be used for a wide range of purposes, such as usage accounting, long-term recording of network usage (classified by IP address attributes) and real-time analysis of traffic flows at remote metering points. 1.2 Applicability Statement (AS) To use RTFM for collecting network traffic information one must first consider where in the network traffic flows are to be measured. Once this is decided, an RTFM Meter must be installed at each chosen measurement point. At least one Meter Reader is needed to collect the measured data from the meters, and a single Manager is needed to control the meters and meter readers. RTFM Meters may be single- or multi-user hosts running a meter program (one such program is available as free software, a second is under development at IBM Research). Alternatively, meters could be run as firmware in switches or routers. A hybrid approach, where an RTFM meter takes raw traffic data from a router, provides another useful implementation path. Brownlee, Mills, Ruth [Page 3] INTERNET-DRAFT Traffic Flow Measurement: Architecture July 98 RTFM Managers are programs running on a Unix host, communicating with meters and meter readers via the network. In principle any protocol could be used for this, but current implementations use the RTFM Meter MIB [4] to store and access the flow data. This will require networks using RTFM to implement the IP protocol so as to send and receive SNMP requests carried in UDP datagrams. As indicated above, a meter must handle SNMP over UDP. If it runs on a dedicated host it must also respond to ARP requests. To aid in troubleshooting, it should respond to ICMP echo (ping) requests. These remarks also apply to RTFM Meter Readers and Managers. (The paragraphs above describe the current implementations of RTFM. Its architecture is not limited to having the meter be an SNMP agent - other communication and storage methods could be used. For example, configuration data could be downloaded via FTP, data could be stored in an SQL database and meter readers could get it via SQL requests.) Users of RTFM should note that installing meters, meter readers and managers merely provides one with the capability to collect flow data. Further installation work will be needed to develop configuration files ('RTFM rulesets') for each meter, data processing applications to analyse the flow data, and various scripts, cron jobs, etc. so as to create a useful production-quality measurement system which suits a user's particular needs. 1.3 Introduction This document describes an architecture for traffic flow measurement and reporting for data networks; it has the following characteristics: - The traffic flow model can be consistently applied to any protocol/application at any network layer (e.g. network, transport, application layers). - Traffic flow attributes are defined in such a way that they are valid for multiple networking protocol stacks, and that traffic flow measurement implementations are useful in MULTI-PROTOCOL environments. - Users may specify their traffic flow measurement requirements by writing 'rule sets,' allowing them to collect the flow data they need while ignoring other traffic. - The data reduction effort to produce requested traffic flow information is placed as near as possible to the network measurement point. This minimises the volume of data to be Brownlee, Mills, Ruth [Page 4] INTERNET-DRAFT Traffic Flow Measurement: Architecture July 98 obtained (and transmitted across the network for storage), and reduces the amount of processing required in traffic flow analysis applications. The architecture specifies common metrics for measuring traffic flows. By using the same metrics, traffic flow data can be exchanged and compared across multiple platforms. Such data is useful for: - Understanding the behaviour of existing networks, - Planning for network development and expansion, - Quantification of network performance, - Verifying the quality of network service, and - Attribution of network usage to users. The traffic flow measurement architecture is deliberately structured so that specific protocol implementations may extend coverage to multi-protocol environments and to other protocol layers, such as usage measurement for application-level services. Use of the same model for both network- and application-level measurement may simplify the development of generic analysis applications which process and/or correlate any or all levels of traffic and usage information. Within this docuemt the term 'usage data' is used as a generic term for the data obtained using the traffic flow measurement architecture. This document is not a protocol specification. It specifies and structures the information that a traffic flow measurement system needs to collect, describes requirements that such a system must meet, and outlines tradeoffs which may be made by an implementor. For performance reasons, it may be desirable to use traffic information gathered through traffic flow measurement in lieu of network statistics obtained in other ways. Although the quantification of network performance is not the primary purpose of this architecture, the measured traffic flow data may be used as an indication of network performance. A cost recovery structure decides "who pays for what." The major issue here is how to construct a tariff (who gets billed, how much, for which things, based on what information, etc). Tariff issues include fairness, predictability (how well can subscribers forecast their network charges), practicality (of gathering the data and administering the tariff), incentives (e.g. encouraging off-peak use), and cost recovery goals (100% recovery, subsidisation, profit making). Issues such as these are not covered here. Brownlee, Mills, Ruth [Page 5] INTERNET-DRAFT Traffic Flow Measurement: Architecture July 98 Background information explaining why this approach was selected is provided by the 'Internet Accounting: Background' RFC [1]. 2 Traffic Flow Measurement Architecture A traffic flow measurement system is used by Network Operations personnel to aid in managing and developing a network. It provides a tool for measuring and understanding the network's traffic flows. This information is useful for many purposes, as mentioned in section 1 (above). The following sections outline a model for traffic flow measurement, which draws from working drafts of the OSI accounting model [2]. Future extensions are anticipated as the model is refined to address additional protocol layers. 2.1 Meters and Traffic Flows At the heart of the traffic measurement model are network entities called traffic METERS. Meters observe packets as they pass by a single point on their way through the network and classify them into certain groups. For each such group a meter will accumulate certain attributes, for example the numbers of packets and bytes observed for the group. These METERED TRAFFIC GROUPS may correspond to a user, a host system, a network, a group of networks, a particular transport address (e.g. an IP port number), any combination of the above, etc, depending on the meter's configuration. We assume that routers or traffic monitors throughout a network are instrumented with meters to measure traffic. Issues surrounding the choice of meter placement are discussed in the 'Traffic Flow Measurement: Background' RFC [1]. An important aspect of meters is that they provide a way of succinctly aggregating traffic information. For the purpose of traffic flow measurement we define the concept of a TRAFFIC FLOW, which is like an artificial logical equivalent to a call or connection. A flow is a portion of traffic, delimited by a start and stop time, that belongs to one of the metered traffic groups mentioned above. Attribute values (source/destination addresses, packet counts, byte counts, etc.) associated with a flow are aggregate quantities reflecting events which take place in the DURATION between the start and stop times. The start time of a flow is fixed for a given flow; the stop time may increase with the age of the flow. For connectionless network protocols such as IP there is by definition no way to tell whether a packet with a particular source/destination Brownlee, Mills, Ruth [Page 6] INTERNET-DRAFT Traffic Flow Measurement: Architecture July 98 combination is part of a stream of packets or not - each packet is completely independent. A traffic meter has, as part of its configuration, a set of 'rules' which specify the flows of interest, in terms of the values of their attributes. It derives attribute values from each observed packet, and uses these to decide which flow they belong to. Classifying packets into 'flows' in this way provides an economical and practical way to measure network traffic and subdivide it into well-defined groups. Usage information which is not derivable from traffic flows may also be of interest. For example, an application may wish to record accesses to various different information resources or a host may wish to record the username (subscriber id) for a particular network session. Provision is made in the traffic flow architecture to do this. In the future the measurement model may be extended to gather such information from applications and hosts so as to provide values for higher-layer flow attributes. As well as FLOWS and METERS, the traffic flow measurement model includes MANAGERS, METER READERS and ANALYSIS APPLICAIONS, which are explained in following sections. The relationships between them are shown by the diagram below. Numbers on the diagram refer to sections in this document. MANAGER / \ 2.3 / \ 2.4 / \ / \ ANALYSIS METER <-----> METER READER <-----> APPLICATION 2.2 2.7 - MANAGER: MANAGER: A traffic measurement manager is an application which configures 'meter' entities and controls 'meter reader' entities. It sends configuration commands to the meters, and supervises the proper operation of each meter and meter reader. It may well be convenient to combine the functions of meter reader and manager within a single network entity. - METER: Meters are placed at measurement points determined by Network Operations personnel. Each meter selectively records network activity as directed by its configuration settings. It can also aggregate, transform and further process the recorded activity before the data is stored. The processed and stored results are called the 'usage data.' - METER READER: A meter reader reliably transports usage data from meters so that it is available to analysis applications. Brownlee, Mills, Ruth [Page 7] INTERNET-DRAFT Traffic Flow Measurement: Architecture July 98 - ANALYSIS APPLICATION: An analysis application processes the usage data so as to provide information and reports which are useful for network engineering and management purposes. Examples include: - TRAFFIC FLOW MATRICES, showing the total flow rates for many of the possible paths within an internet. - FLOW RATE FREQUENCY DISTRIBUTIONS, summarizing flow rates over a period of time. - USAGE DATA showing the total traffic volumes sent and received by particular hosts. The operation of the traffic measurement system as a whole is best understood by considering the interactions between its components. These are described in the following sections. 2.2 Interaction Between METER and METER READER The information which travels along this path is the usage data itself. A meter holds usage data in an array of flow data records known as the FLOW TABLE. A meter reader may collect the data in any suitable manner. For example it might upload a copy of the whole flow table using a file transfer protocol, or read the records in the current flow set one at a time using a suitable data transfer protocol. Note that the meter reader need not read complete flow data records, a subset of their attribute values may well be sufficient. A meter reader may collect usage data from one or more meters. Data may be collected from the meters at any time. There is no requirement for collections to be synchronized in any way. 2.3 Interaction Between MANAGER and METER A manager is responsible for configuring and controlling one or more meters. Each meter's configuration includes information such as: - Flow specifications, e.g. which traffic flows are to be measured, how they are to be aggregated, and any data the meter is required to compute for each flow being measured. - Meter control parameters, e.g. the 'inactivity' time for flows (if no packets belonging to a flow are seen for this time the flow is considered to have ended, i.e. to have become idle). Brownlee, Mills, Ruth [Page 8] INTERNET-DRAFT Traffic Flow Measurement: Architecture July 98 - Sampling rate. Normally every packet will be observed. It may sometimes be necessary to use sampling techniques to observe only some of the packets. (Sampling algorithms are not prescribed by the architecture; it should be noted that before using sampling one should verify the statistical validity of the algorithm used). Current experience with the measurement architecture shows that a carefully-designed and implemented meter compresses the data such that in normal LANs and WANs of today sampling is really not needed. A meter may run several rule sets concurrently on behalf of one or more managers, and any manager may download a set of flow specifications (i.e. a 'rule set') to a meter. Control parameters which apply to an individual rule set should be set by the manager after it downloads that rule set. One manager should be designated as the 'master' for a meter. Parameters such as sampling rate, which affect the overall operation of the meter, should only be set by the master manager. 2.4 Interaction Between MANAGER and METER READER A manager is responsible for configuring and controlling one or more meter readers. A meter reader may only be controlled by a single manager. A meter reader needs to know at least the following for every meter it is collecting usage data from: - The meter's unique identity, i.e. its network name or address. - How often usage data is to be collected from the meter. - Which flow records are to be collected (e.g. all flows, flows for a particular rule set, flows which have been active since a given time, etc.). - Which attribute values are to be collected for the required flow records (e.g. all attributes, or a small subset of them) Since redundant reporting may be used in order to increase the reliability of usage data, exchanges among multiple entities must be considered as well. These are discussed below. Brownlee, Mills, Ruth [Page 9] INTERNET-DRAFT Traffic Flow Measurement: Architecture July 98 2.5 Multiple METERs or METER READERs -- METER READER A -- / | \ / | \ =====METER 1 METER 2=====METER 3 METER 4===== \ | / \ | / -- METER READER B -- Several uniquely identified meters may report to one or more meter readers. The diagram above gives an example of how multiple meters and meter readers could be used. In the diagram above meter 1 is read by meter reader A, and meter 4 is read by meter reader B. Meters 1 and 4 have no redundancy; if either meter fails, usage data for their network segments will be lost. Meters 2 and 3, however, measure traffic on the same network segment. One of them may fail leaving the other collecting the segment's usage data. Meters 2 and 3 are read by meter reader A and by meter reader B. If one meter reader fails, the other will continue collecting usage data from both meters. The architecture does not require multiple meter readers to be synchronized. In the situation above meter readers A and B could both collect usage data at the same intervals, but not neccesarily at the same times. Note that because collections are asynchronous it is unlikely that usage records from two different meter readers will agree exactly. If precisely synchronized collections are required this can be achieved by having one manager request each meter to begin running a new set of rules at the same time, then allowing all meter readers to collect the usage data from the flows recorded by the old rule sets. If there is only one meter reader and it fails, the meters continue to run. When the meter reader is restarted it can collect all of the accumulated flow data. Should this happen, time resolution will be lost (because of the missed collections) but overall traffic flow information will not. The only exception to this would occur if the traffic volume was sufficient to 'roll over' counters for some flows during the failure; this is addressed in the section on 'Rolling Counters.' Brownlee, Mills, Ruth [Page 10] INTERNET-DRAFT Traffic Flow Measurement: Architecture July 98 2.6 Interaction Between MANAGERs (MANAGER - MANAGER) Synchronization between multiple management systems is the province of network management protocols. This traffic flow measurement architecture specifies only the network management controls necessary to perform the traffic flow measurement function and does not address the more global issues of simultaneous or interleaved (possibly conflicting) commands from multiple network management stations or the process of transferring control from one network management station to another. 2.7 METER READERs and APPLICATIONs Once a collection of usage data has been assembled by a meter reader it can be processed by an analysis application. Details of analysis applications - such as the reports they produce and the data they require - are outside the scope of this architecture. It should be noted, however, that analysis applications will often require considerable amounts of input data. An important part of running a traffic flow measurement system is the storage and regular reduction of flow data so as to produce daily, weekly or monthly summary files for further analysis. Again, details of such data handling are outside the scope of this architecture. 3 Traffic Flows and Reporting Granularity A flow was defined in section 2.1 above in abstract terms as follows: "A TRAFFIC FLOW is an artifical logical equivalent to a call or connection, belonging to a (user-specieied) METERED TRAFFIC GROUP." In practical terms, a flow is a stream of packets observed by the meter as they pass across a network between two end points (or from a single end point), which have been summarized by a traffic meter for analysis purposes. Brownlee, Mills, Ruth [Page 11] INTERNET-DRAFT Traffic Flow Measurement: Architecture July 98 3.1 Flows and their Attributes Every traffic meter maintains a table of 'flow records' for flows seen by the meter. A flow record holds the values of the ATTRIBUTES of interest for its flow. These attributes might include: - ADDRESSES for the flow's source and destination. These comprise the protocol type, the source and destination addresses at various network layers (extracted from the packet header), and the number of the interface on which the packet was observed. - First and last TIMES when packets were seen for this flow, i.e. the 'creation' and 'last activity' times for the flow. - COUNTS for 'forward' (source to destination) and 'backward' (destination to source) components (e.g. packets and bytes) of the flow's traffic. The specifying of 'source' and 'destination' for flows is discussed in the section on packet matching below. - OTHER attributes, e.g. the index of the flow's record in the flow table and the rule set id for the rules which the meter was running while the flow was observed. The values of these attributes provide a way of distinguishing flows observed by a meter at different times. The attributes listed in this document (Appendix C) provide a basic (i.e. useful minimum) set; they are assigned attribute numbers in the range 0 to 63. The RTFM working group is working on an extended set of attributes, which will have numbers in the range 64 to 127. Implementors wishing to experiment with further new attributes should use attribute numbers 128 and above. A flow's METERED TRAFFIC GROUP is specified by the values of its ADDRESS attributes. For example, if a flow's address attributes specified only that "source address = IP address 10.1.0.1," then all IP packets from and to that address would be counted in that flow. If a flow's address list were specified as "source address = IP address 10.1.0.1, destination address = IP address 26.1.0.1" then only IP packets between 10.1.0.1 and 26.1.0.1 would be counted in that flow. The addresses specifying a flow's address attributes may include one or more of the following types: - The INTERFACE NUMBER for the flow, i.e. the interface on which the meter measured the traffic. Together with a unique address for the meter this uniquely identifies a particular physical-level port. Brownlee, Mills, Ruth [Page 12] INTERNET-DRAFT Traffic Flow Measurement: Architecture July 98 - The ADJACENT ADDRESS, i.e. the [n-1] layer address of the immediate source or destination on the path of the packet. For example, if flow measurement is being performed at the IP layer on an Ethernet LAN [3], an adjacent address will normally be a six-octet Media Access Control (MAC) address. For a host connected to the same LAN segment as the meter the adjacent address will be the MAC address of that host. For hosts on other LAN segments it will be the MAC address of the adjacent (upstream or downstream) router carrying the traffic flow. - The PEER ADDRESS, which identifies the source or destination of the PEER-LEVEL packet. The form of a peer address will depend on the network-layer protocol in use, and the network layer [n] at which traffic measurement is being performed. - The TRANSPORT ADDRESS, which identifies the source or destination port for the packet, i.e. its [n+1] layer address. For example, if flow measurement is being performed at the IP layer a transport address is a two-octet UDP or TCP port number. The four definitions above specify addresses for each of the four lowest layers of the OSI reference model, i.e. Physical layer, Link layer, Network layer and Transport layer. A FLOW RECORD stores both the VALUE for each of its addresses (as described above) and a MASK specifying which bits of the address value are being used and which are ignored. Note that if address bits are being ignored the meter will set them to zero, however their actual values are undefined. One of the key features of the traffic measurement architecture is that attributes have essentially the same meaning for different protocols, so that analysis applications can use the same reporting formats for all protocols. This is straightforward for peer addresses; although the form of addresses differs for the various protocols, the meaning of a 'peer address' remains the same. It becomes harder to maintain this correspondence at higher layers - for example, at the Network layer IP, Novell IPX and AppleTalk all use port numbers as a 'transport address,' but CLNP and DECnet have no notion of ports. Further work is needed here, particularly in selecting attributes which will be suitable for the higher layers of the OSI reference model. Reporting by adjacent intermediate sources and destinations or simply by meter interface (most useful when the meter is embedded in a router) supports hierarchical Internet reporting schemes as described in the 'Internet Accounting: Background' RFC [1]. That is, it allows backbone and regional networks to measure usage to just the next lower level of granularity (i.e. to the regional and stub/enterprise levels, respectively), with the final breakdown according to end user (e.g. to source IP address) performed by the stub/enterprise networks. Brownlee, Mills, Ruth [Page 13] INTERNET-DRAFT Traffic Flow Measurement: Architecture July 98 In cases where network addresses are dynamically allocated (e.g. mobile subscribers), further subscriber identification will be necessary if flows are to ascribed to individual users. Provision is made to further specify the metered traffic group through the use of an optional SUBSCRIBER ID as part of the flow id. A subscriber ID may be associated with a particular flow either through the current rule set or by proprietary means within a meter, for example via protocol exchanges with one or more (multi-user) hosts. At this time a subscriber ID is an arbitrary text string; later versions of the architecture may specify its contents in more detail. 3.2 Granularity of Flow Measurements GRANULARITY is the 'control knob' by which an application and/or the meter can trade off the overhead associated with performing usage reporting against its level of detail. A coarser granularity means a greater level of aggregation; finer granularity means a greater level of detail. Thus, the number of flows measured (and stored) at a meter can be regulated by changing the granularity of their attributes. Flows are like an adjustable pipe - many fine-granularity streams can carry the data with each stream measured individually, or data can be bundled in one coarse-granularity pipe. Time granularity may be controlled by varying the reporting interval, i.e. the time between meter readings. Flow granularity is controlled by adjusting the level of detail for the following: - The metered traffic group (address attributes, discussed above). - The categorisation of packets (other attributes, discussed below). - The lifetime/duration of flows (the reporting interval needs to be short enough to measure them with sufficient precision). The set of rules controlling the determination of each packet's metered traffic group is known as the meter's CURRENT RULE SET. As will be shown, the meter's current rule set forms an integral part of the reported information, i.e. the recorded usage information cannot be properly interpreted without a definition of the rules used to collect that information. Settings for these granularity factors may vary from meter to meter. They are determined by the meter's current rule set, so they will change if network Operations personnel reconfigure the meter to use a new rule set. It is expected that the collection rules will change rather Brownlee, Mills, Ruth [Page 14] INTERNET-DRAFT Traffic Flow Measurement: Architecture July 98 infrequently; nonetheless, the rule set in effect at any time must be identifiable via a RULE SET ID. Granularity of metered traffic groups is further specified by additional ATTRIBUTES. These attributes include: - Attributes which record information derived from other attribute values. Six of these are defined (SourceClass, DestClass, FlowClass, SourceKind, DestKind, FlowKind), and their meaning is determined by the meter's rule set. For example, one could have a subroutine in the rule set which determined whether a source or destination peer address was a member of an arbitrary list of networks, and set SourceClass/DestClass to one if the source/dest peer address was in the list or to zero otherwise. - Administratively specified attributes such as Quality Of Service and Priority, etc. These are not defined at this time. - Higher-layer (especially application-level) attributes. These are not defined at this time. Settings for these granularity factors may vary from meter to meter. They are determined by the meter's current rule set, so they will change if Network Operations personnel reconfigure the meter to use a new rule set. The LIFETIME of a flow is the time interval which began when the meter observed the first packet belonging to the flow and ended when it saw the last packet. Flow lifetimes are very variable, but many - if not most - are rather short. A meter cannot measure lifetimes directly; instead a meter reader collects usage data for flows which have been active since the last collection, and an analysis application may compare the data from each collection so as to determine when each flow actually stopped. The meter does, however, need to reclaim memory (i.e. records in the flow table) being held by idle flows. The meter configuration includes a variable called InactivityTimeout, which specifies the minimum time a meter must wait before recovering the flow's record. In addition, before recovering a flow record the meter must be sure that the flow's data has been collected by all meter readers which registered to collect it. These 'lifetime' issues are considered further in the section on meter readers (below). A complete list of the attributes currently defined is given in Appendix C later in this document. Brownlee, Mills, Ruth [Page 15] INTERNET-DRAFT Traffic Flow Measurement: Architecture July 98 3.3 Rolling Counters, Timestamps, Report-in-One-Bucket-Only Once a usage record is sent, the decision needs to be made whether to clear any existing flow records or to maintain them and add to their counts when recording subsequent traffic on the same flow. The second method, called rolling counters, is recommended and has several advantages. Its primary advantage is that it provides greater reliability - the system can now often survive the loss of some usage records, such as might occur if a meter reader failed and later restarted. The next usage record will very often contain yet another reading of many of the same flow buckets which were in the lost usage record. The 'continuity' of data provided by rolling counters can also supply information used for "sanity" checks on the data itself, to guard against errors in calculations. The use of rolling counters does introduce a new problem: how to distinguish a follow-on flow record from a new flow record. Consider the following example. CONTINUING FLOW OLD FLOW, then NEW FLOW start time = 1 start time = 1 Usage record N: flow count = 2000 flow count = 2000 (done) start time = 1 start time = 5 Usage record N+1: flow count = 3000 new flow count = 1000 Total count: 3000 3000 In the continuing flow case, the same flow was reported when its count was 2000, and again at 3000: the total count to date is 3000. In the OLD/NEW case, the old flow had a count of 2000. Its record was then stopped (perhaps because of temporary idleness), but then more traffic with the same characteristics arrived so a new flow record was started and it quickly reached a count of 1000. The total flow count from both the old and new records is 3000. The flow START TIMESTAMP attribute is sufficient to resolve this. In the example above, the CONTINUING FLOW flow record in the second usage record has an old FLOW START timestamp, while the NEW FLOW contains a recent FLOW START timestamp. Each packet is counted in at most one flow for each running ruleset, so as to avoid multiple counting of a single packet. The record of a single flow is informally called a "bucket." If multiple, sometimes overlapping, records of usage information are required (aggregate, individual, etc), the network manager should collect the counts in sufficiently detailed granularity so that aggregate and combination Brownlee, Mills, Ruth [Page 16] INTERNET-DRAFT Traffic Flow Measurement: Architecture July 98 counts can be reconstructed in post-processing of the raw usage data. Alternatively, multiple rulesets could be used to collect data at different granularities. For example, consider a meter from which it is required to record both 'total packets coming in interface #1' and 'total packets arriving from any interface sourced by IP address = a.b.c.d,' using a single rule set. Although a bucket can be declared for each case, it is not clear how to handle a packet which satisfies both criteria. It must only be counted once. By default it will be counted in the first bucket for which it qualifies, and not in the other bucket. Further, it is not possible to reconstruct this information by post-processing. The solution in this case is to define not two, but THREE buckets, each one collecting a unique combination of the two criteria: Bucket 1: Packets which came in interface 1, AND were sourced by IP address a.b.c.d Bucket 2: Packets which came in interface 1, AND were NOT sourced by IP address a.b.c.d Bucket 3: Packets which did NOT come in interface 1, AND were sourced by IP address a.b.c.d (Bucket 4: Packets which did NOT come in interface 1, AND NOT sourced by IP address a.b.c.d) The desired information can now be reconstructed by post-processing. "Total packets coming in interface 1" can be found by adding buckets 1 & 2, and "Total packets sourced by IP address a.b.c.d" can be found by adding buckets 1 & 3. Note that in this case bucket 4 is not explicitly required since its information is not of interest, but it is supplied here in parentheses for completeness. Alternatively, the above could be achieved by running two rule sets (A and B), as follows: Bucket 1: Packets which came in interface 1; counted by rule set A. Bucket 2: Packets which were sourcede by IP address a.b.c.d; counted by rule set B. Brownlee, Mills, Ruth [Page 17] INTERNET-DRAFT Traffic Flow Measurement: Architecture July 98 4 Meters A traffic flow meter is a device for collecting data about traffic flows at a given point within a network; we will call this the METERING POINT. The header of every packet passing the network metering point is offered to the traffic meter program. A meter could be implemented in various ways, including: - A dedicated small host, connected to a LAN (so that it can see all packets as they pass by) and running a traffic meter program. The metering point is the LAN segment to which the meter is attached. - A multiprocessing system with one or more network interfaces, with drivers enabling a traffic meter program to see packets. In this case the system provides multiple metering points - traffic flows on any subset of its network interfaces can be measured. - A packet-forwarding device such as a router or switch. This is similar to (b) except that every received packet should also be forwarded, usually on a different interface. 4.1 Meter Structure An outline of the meter's structure is given in the following diagram: Briefly, the meter works as follows: - Incoming packet headers arrive at the top left of the diagram and are passed to the PACKET PROCESSOR. - The packet processor passes them to the Packet Matching Engine (PME) where they are classified. - The PME is a Virtual Machine running a pattern matching program contained in the CURRENT RULE SET. It is invoked by the Packet Processor, and returns instructions on what to do with the packet. - Some packets are classified as 'to be ignored.' They are discarded by the Packet Processor. - Other packets are matched by the PME, which returns a FLOW KEY describing the flow to which the packet belongs. Brownlee, Mills, Ruth [Page 18] INTERNET-DRAFT Traffic Flow Measurement: Architecture July 98 - The flow key is used to locate the flow's entry in the FLOW TABLE; a new entry is created when a flow is first seen. The entry's data fields (e.g. packet and byte counters) are updated. - A meter reader may collect data from the flow table at any time. It may use the 'collect' index to locate the flows to be collected within the flow table. packet +------------------+ header | Current Rule Set | | +--------+---------+ | | +--------*---------+ +----------*-------------+ | Packet Processor |<----->| Packet Matching Engine | +--+------------+--+ +------------------------+ | | Ignore * | Count via flow key | +--*--------------+ | 'Search' index | +--------+--------+ | +--------*--------+ | | | Flow Table | | | +--------+--------+ | +--------*--------+ | 'Collect' index | +--------+--------+ | * Meter Reader The discussion above assumes that a meter will only be running a single rule set. A meter may, however, run several rule sets concurrently. To do this the meter maintains a table of current rulesets. The packet processor matches each packet against every current ruleset, producing a single flow table with flows from all the rule sets. The overall effect of doing this is somewhat similar to running several independent meters, one for each rule set. 4.2 Flow Table Every traffic meter maintains a table of TRAFFIC FLOW RECORDS for flows seen by the meter. A flow record contains attribute values for its flow, including: Brownlee, Mills, Ruth [Page 19] INTERNET-DRAFT Traffic Flow Measurement: Architecture July 98 - Addresses for the flow's source and destination. These include addresses and masks for various network layers (extracted from the packet), and the identity of the interface on which the packet was observed. - First and last times when packets were seen for this flow. - Counts for 'forward' (source to destination) and 'backward' (destination to source) components of the flow's traffic. - Other attributes, e.g. state of the flow record (discussed below). The state of a flow record may be: - INACTIVE: The flow record is not being used by the meter. - CURRENT: The record is in use and describes a flow which belongs to the 'current flow set,' i.e. the set of flows recently seen by the meter. - IDLE: The record is in use and the flow which it describes is part of the current flow set. In addition, no packets belonging to this flow have been seen for a period specified by the meter's InactivityTime variable. 4.3 Packet Handling, Packet Matching Each packet header received by the traffic meter program is processed as follows: - Extract attribute values from the packet header and use them to create a MATCH KEY for the packet. - Match the packet's key against the current rule set, as explained in detail below. The rule set specifies whether the packet is to be counted or ignored. If it is to be counted the matching process produces a FLOW KEY for the flow to which the packet belongs. This flow key is used to find the Brownlee, Mills, Ruth [Page 20] INTERNET-DRAFT Traffic Flow Measurement: Architecture July 98 flow's record in the flow table; if a record does not yet exist for this flow, a new flow record may be created. The data for the matching flow record can then be updated. For example, the rule set could specify that packets to or from any host in IP network 130.216 are to be counted. It could also specify that flow records are to be created for every pair of 24-bit (Class C) subnets within network 130.216. Each packet's match key is passed to the meter's PATTERN MATCHING ENGINE (PME) for matching. The PME is a Virtual Machine which uses a set of instructions called RULES, i.e. a RULE SET is a program for the PME. A packet's match key contains source (S) and destination (D) interface identities, address values and masks. If measured flows were unidirectional, i.e. only counted packets travelling in one direction, the matching process would be simple. The PME would be called once to match the packet. Any flow key produced by a successful match would be used to find the flow's record in the flow table, and that flow's counters would be updated. Flows are, however, bidirectional, reflecting the forward and reverse packets of a protocol interchange or 'session.' Maintaining two sets of counters in the meter's flow record makes the resulting flow data much simpler to handle, since analysis programs do not have to gather together the 'forward' and 'reverse' components of sessions. Implementing bi-directional flows is, of course, more difficult for the meter, since it must decide whether a packet is a 'forward' packet or a 'reverse' one. To make this decision the meter will often need to invoke the PME twice, once for each possible packet direction. The diagram below describes the algorithm used by the traffic meter to process each packet. Flow through the diagram is from left to right and top to bottom, i.e. from the top left corner to the bottom right corner. S indicates the flow's source address (i.e. its set of source address attribute values) from the packet header, and D indicates its destination address. There are several cases to consider. These are: - The packet is recognised as one which is TO BE IGNORED. - The packet would MATCH IN EITHER DIRECTION. One situation in which this could happen would be a rule set which matches flows within network X (Source = X, Dest = X) but specifies that flows are to be created for each subnet within network X, say subnets y and z. If, for example a packet is seen for y->z, the meter must check that flow z->y is not already current before creating y->z. - The packet MATCHES IN ONE DIRECTION ONLY. If its flow is already current, its forward or reverse counters are incremented. Otherwise it is added to the flow table and then counted. Brownlee, Mills, Ruth [Page 21] INTERNET-DRAFT Traffic Flow Measurement: Architecture July 98 Ignore --- match(S->D) -------------------------------------------------+ | Suc | NoMatch | | | Ignore | | match(D->S) -----------------------------------------+ | | Suc | NoMatch | | | | | | | +-------------------------------------------+ | | | | | Suc | | current(D->S) ---------- count(D->S,r) --------------+ | | Fail | | | | | create(D->S) ----------- count(D->S,r) --------------+ | | | Suc | current(S->D) ------------------ count(S->D,f) --------------+ | Fail | | Suc | current(D->S) ------------------ count(D->S,r) --------------+ | Fail | | | create(S->D) ------------------- count(S->D,f) --------------+ | * The algorithm uses four functions, as follows: match(A->B) implements the PME. It uses the meter's current rule set to match the attribute values in the packet's match key. A->B means that the assumed source address is A and destination address B, i.e. that the packet was travelling from A to B. match() returns one of three results: 'Ignore' means that the packet was matched but this flow is not to be counted. 'NoMatch' means that the packet did not match. It might, however match with its direction reversed, i.e. from B to A. 'Suc' means that the packet did match, i.e. it belongs to a flow which is to be counted. current(A->B) succeeds if the flow A-to-B is current - i.e. has a record in the flow table whose state is Current - and fails otherwise. Brownlee, Mills, Ruth [Page 22] INTERNET-DRAFT Traffic Flow Measurement: Architecture July 98 create(A->B) adds the flow A-to-B to the flow table, setting the value for attributes - such as addresses - which remain constant, and zeroing the flow's counters. count(A->B,f) increments the 'forward' counters for flow A-to-B. count(A->B,r) increments the 'reverse' counters for flow A-to-B. 'Forward' here means the counters for packets travelling from A to B. Note that count(A->B,f) is identical to count(B->A,r). When writing rule sets one must remember that the meter will normally try to match each packet in the reverse direction if the forward match does not succeed. It is particularly important that the rule set does not contain inconsistencies which will upset this process. Consider, for example, a rule set which counts packets from source network A to destination network B, but which ignores packets from source network B. This is an obvious example of an inconsistent rule set, since packets from network B should be counted as reverse packets for the A-to-B flow. This problem could be avoided by devising a language for specifying rule files and writing a compiler for it, thus making it much easier to produce correct rule sets. Another approach would be to write a 'rule set consistency checker' program, which could detect problems in hand-written rule sets. Normally, the best way to avoid these problems is to write rule sets which only classify flows in the forward direction, and rely on the meter to handle reverse-travelling packets. Occasionally there can be situations when a rule set needs to know the direction in which a packet is being matched. Consider, for example, a rule set which wants to save some attribute values (source and destination addresses perhaps) for any 'unusual' packets. The rule set will contain a sequence of tests for all the 'usual' source addresses; follwed by a rule which will execute a 'NoMatch' action. If the match fails in the S->D direction, the NoMatch action will cause it to be retried. If it fails in the D->S direction, the packet can be counted as an 'unusual' packet. To count such an 'unusual' packet we need to know the matching direction: the MatchingStoD attribute provides this. To use it, one follows the source address tests with a rule which tests whether the matching direction is S->D (MatchingStoD value is 1). If so, a 'NoMatch' action is executed. Otherwise, the packet has failed to match in both directions; we can Push whatever attribute values are of interest and count the 'unusual' packet. Brownlee, Mills, Ruth [Page 23] INTERNET-DRAFT Traffic Flow Measurement: Architecture July 98 4.4 Rules and Rule Sets A rule set is an array of rules. Rule sets are held within a meter as entries in an array of rule sets. Rule set 1 (the first entry in the rule set table) is built-in to the meter and cannot be changed. It is run when the meter is started up, and provides a very coarse reporting granularity; it is mainly useful for verifying that the meter is running, before a 'useful' rule set is downloaded to it. A meter also maintains an array of 'tasks,' which specify what rule sets the meter is running. Each task has a 'current' rule set (the one which it normally uses), and a 'standby' rule set (which will be used when the overall traffic level is unusually high). If a task is instructed to use rule set 0, it will cease measuring; all packets will be ignored until another (non-zero) rule set is made current. Each rule in a rule set is structured as follows: +-------- test ---------+ +---- action -----+ attribute & mask = value: opcode, parameter; Opcodes contain two flags: 'goto' and 'test.' The PME maintains a Boolean indicator called the 'test indicator,' which is initially set (true). Execution begins with rule 1, the first in the rule set. It proceeds as follows: If the test indicator is true: Perform the test, i.e. AND the attribute value with the mask and compare it with the value. If these are equal the test has succeeded; perform the rule's action (below). If the test fails execute the next rule in the rule set. If there are no more rules in the rule set, return from the match() function indicating NoMatch. If the test indicator is false, or the test (above) succeeded: Set the test indicator to this opcode's test flag value. Determine the next rule to execute. If the opcode has its goto flag set, its parameter value specifies the number of the next rule. Opcodes which don't have their goto flags set either determine the next rule in special ways (Return), or they terminate execution (Ignore, NoMatch, Count, CountPkt). Perform the action. Brownlee, Mills, Ruth [Page 24] INTERNET-DRAFT Traffic Flow Measurement: Architecture July 98 The PME maintains two 'history' data structures. The first, the 'return' stack, simply records the index (i.e. 1-origin rule number) of each Gosub rule as it is executed; Return rules pop their Gosub rule index. The second, the 'pattern' queue, is used to save information for later use in building a flow key. A flow key is built by zeroing all its attribute values, then copying attribute and mask information from the pattern queue in the order it was enqueued. The opcodes are: opcode goto test 1 Ignore 0 - 2 NoMatch 0 - 3 Count 0 - 4 CountPkt 0 - 5 Return 0 0 6 Gosub 1 1 7 GosubAct 1 0 8 Assign 1 1 9 AssignAct 1 0 10 Goto 1 1 11 GotoAct 1 0 12 PushRuleTo 1 1 13 PushRuleToAct 1 0 14 PushPktTo 1 1 15 PushPktToAct 1 0 16 PopTo 1 1 17 PopToAct 1 0 The actions they perform are: Ignore: Stop matching, return from the match() function indicating that the packet is to be ignored. NoMatch: Stop matching, return from the match() function indicating failure. Count: Stop matching. Save this rule's attribute name, mask and value in the PME's pattern queue, then construct a flow key for the flow to which this packet belongs. Return from the match() function indicating success. The meter will use the flow key to search for the flow record for this packet's flow. CountPkt: As for Count, except that the masked value from the packet is saved in the PME's pattern queue instead of the rule's value. Brownlee, Mills, Ruth [Page 25] INTERNET-DRAFT Traffic Flow Measurement: Architecture July 98 Gosub: Call a rule-matching subroutine. Push the current rule number on the PME's return stack, set the test indicator then goto the specified rule. GosubAct: Same as Gosub, except that the test indicator is cleared before going to the specified rule. Return: Return from a rule-matching subroutine. Pop the number of the calling gosub rule from the PME's 'return' stack and add this rule's parameter value to it to determine the 'target' rule. Clear the test indicator then goto the target rule. A subroutine call appears in a rule set as a Gosub rule followed by a small group of following rules. Since a Return action clears the test flag, the action of one of these 'following' rules will be executed; this allows the subroutine to return a result (in addition to any information it may save in the PME's pattern queue). Assign: Set the attribute specified in this rule to the value specified in this rule. Set the test indicator then goto the specified rule. AssignAct: Same as Assign, except that the test indicator is cleared before going to the specified rule. Goto: Set the test indicator then goto the specified rule. GotoAct: Clear the test indicator then goto the specified rule. PushRuleTo: Save this rule's attribute name, mask and value in the PME's pattern queue. Set the test indicator then goto the specified rule. PushRuleToAct: Same as PushRuleTo, except that the test indicator is cleared before going to the specified rule. PushRuleTo actions may be used to save the value and mask used in a test, or (if the test is not performed) to save an arbitrary value and mask. PushPktTo: Save this rule's attribute name, mask, and the masked value from the packet header, in the PME's pattern queue. Set the test indicator then goto the specified rule. Brownlee, Mills, Ruth [Page 26] INTERNET-DRAFT Traffic Flow Measurement: Architecture July 98 PushPktToAct: Same as PushPktTo, except that the test indicator is cleared before going to the specified rule. PushPktTo actions may be used to save a value from the packet using a specified mask. The test in PushPktTo rules will almost never be executed. PopTo: Delete the most recent item from the pattern queue, so as to remove the information saved by an earlier 'push' action. Set the test indicator then goto the specified rule. PopToAct: Same as PopTo, except that the test indicator is cleared before going to the specified rule. As well as the attributes applying directly to packets (such as SourcePeerAddress, DestTransAddress, etc.) the PME implements several further attribtes. These are: Null: Tests performed on the Null attribute always succeed. MatchingStoD: Indicates whether the PME is matching the packet with its addresses in 'wire order' or with its addresses reversed. MatchingStoD's value is 1 if the addresses are in wire order (StoD), and != 1 otherwise. v1 .. v5: v1, v2, v3, v4 and v5 are 'meter variables.' They provide a way to pass parameters into rule-matching subroutines. Each may hold the number of a normal attribute; its value is set by an Assign action. When a meter variable appears as the attribute of a rule, its value specifies the actual attribute to be tested. For example, if v1 had been assigned SourcePeerAddress as its value, a rule with v1 as its attribute would actually test SourcePeerAddress. SourceClass, DestClass, FlowClass, SourceKind, DestKind, FlowKind: These six attributes may be set by executing PushRuleto actions. They allow the PME to save (in flow records) information which has been built up during matching. Their values may be tested in rules; this allows one to set them early in a rule set, and test them later. Brownlee, Mills, Ruth [Page 27] INTERNET-DRAFT Traffic Flow Measurement: Architecture July 98 4.5 Maintaining the Flow Table The flow table may be thought of as a 1-origin array of flow records. (A particular implementation may, of course, use whatever data structure is most suitable). When the meter starts up there are no known flows; all the flow records are in the 'inactive' state. Each time a packet is matched for a flow which is not in a current flow set a flow record is created for it; the state of such a record is 'current.' When selecting a record for the new flow the meter searches the flow table for an 'inactive' record. If no inactive records are available it will search for an 'idle' one instead. Note that there is no particular significance in the ordering of records within the table. Flow data may be collected by a 'meter reader' at any time. There is no requirement for collections to be synchronized. The reader may collect the data in any suitable manner, for example it could upload a copy of the whole flow table using a file transfer protocol, or it could read the records in the current flow set row by row using a suitable data transfer protocol. The meter keeps information about collections, in particular it maintains ReaderLastTime variables which remember the time the last collection was made by each reader. A second variable, InactivityTime, specifies the minimum time the meter will wait before considering that a flow is idle. The meter must recover records used for idle flows, if only to prevent it running out of flow records. Recovered flow records are returned to the 'inactive' state. A variety of recovery strategies are possible, including the following: One possible recovery strategy is to recover idle flow records as soon as possible after their data has been collected by all readers which have registered to do so. To implement this the meter could run a background process which scans the flow table looking for 'current' flows whose 'last packet' time is earlier than the meter's LastCollectTime. Another recovery strategy is to leave idle flows alone as long as possible, which would be suitable if one was only interested in measuring total traffic volumes. It could be implemented by having the meter search for collected idle flows only when it ran low on 'inactive' flow records. One further factor a meter should consider before recovering a flow is the number of meter readers which have collected the flow's data. If there are multiple meter readers operating, each reader should collect a flow's data before its memory is recovered. Brownlee, Mills, Ruth [Page 28] INTERNET-DRAFT Traffic Flow Measurement: Architecture July 98 4.6 Handling Increasing Traffic Levels Under normal conditions the meter reader specifies which set of usage records it wants to collect, and the meter provides them. If memory usage rises above the high-water mark the meter should switch to a STANDBY RULE SET so as to decrease the rate at which new flows are created. When the manager, usually as part of a regular poll, becomes aware that the meter is using its standby rule set, it could decrease the interval between collections. The meter should also increase its efforts to recover flow memory so as to reduce the number of idle flows in memory. When the situation returns to normal, the manager may request the meter to switch back to its normal rule set. 5 Meter Readers Usage data is accumulated by a meter (e.g. in a router) as memory permits. It is collected at regular reporting intervals by meter readers, as specified by a manager. The collected data is recorded in a disk file called a FLOW DATA FILE, as a sequence of USAGE RECORDS. The following sections describe the contents of usage records and flow data files. Note, however, that at this stage the details of such records and files is not specified in the architecture. Specifying a common format for them would be a worthwhile future development. 5.1 Identifying Flows in Flow Records Once a packet has been classified and is ready to be counted, an appropriate flow data record must already exist in the flow table; otherwise one must be created. The flow record has a flexible format where unnecessary identification attributes may be omitted. The determination of which attributes of the flow record to use, and of what values to put in them, is specified by the current rule set. Note that the combination of start time, rule set id and subscript (row number in the flow table) provide a unique flow identifier, regardless of the values of its other attributes. The current rule set may specify additional information, e.g. a computed attribute value such as FlowKind, which is to be placed in the attribute section of the usage record. That is, if a particular flow is matched by the rule set, then the corresponding flow record should be marked not only with the qualifying identification attributes, but also with the additional information. Using this feature, several flows may Brownlee, Mills, Ruth [Page 29] INTERNET-DRAFT Traffic Flow Measurement: Architecture July 98 each carry the same FlowKind value, so that the resulting usage records can be used in post-processing or between meter reader and meter as a criterion for collection. 5.2 Usage Records, Flow Data Files The collected usage data will be stored in flow data files on the meter reader, one file for each meter. As well as containing the measured usage data, flow data files must contain information uniquely identifiying the meter from which it was collected. A USAGE RECORD contains the descriptions of and values for one or more flows. Quantities are counted in terms of number of packets and number of bytes per flow. Each usage record contains the metered traffic group identifier of the meter (a set of network addresses), a time stamp and a list of reported flows (FLOW DATA RECORDS). A meter reader will build up a file of usage records by regularly collecting flow data from a meter, using this data to build usage records and concatenating them to the tail of a file. Such a file is called a FLOW DATA FILE. A usage record contains the following information in some form: +-------------------------------------------------------------------+ | RECORD IDENTIFIERS: | | Meter Id (& digital signature if required) | | Timestamp | | Collection Rules ID | +-------------------------------------------------------------------+ | FLOW IDENTIFIERS: | COUNTERS | | Address List | Packet Count | | Subscriber ID (Optional) | Byte Count | | Attributes (Optional) | Flow Start/Stop Time | +-------------------------------------------------------------------+ 5.3 Meter to Meter Reader: Usage Record Transmission The usage record contents are the raison d'etre of the system. The accuracy, reliability, and security of transmission are the primary concerns of the meter/meter reader exchange. Since errors may occur on networks, and Internet packets may be dropped, some mechanism for ensuring that the usage information is transmitted intact is needed. Flow data is moved from meter to meter reader via a series of protocol exchanges between them. This may be carried out in various ways, moving individual attribute values, complete flows, or the entire flow table Brownlee, Mills, Ruth [Page 30] INTERNET-DRAFT Traffic Flow Measurement: Architecture July 98 (i.e. all the active and idle flows). One possible method of achieving this transfer is to use SNMP; the 'Traffic Flow Measurement: Meter MIB' document [4] gives details. Note that this is simply one example; the transfer of flow data from meter to meter reader is not specified in this document. The reliability of the data transfer method under light, normal, and extreme network loads should be understood before selecting among collection methods. In normal operation the meter will be running a rule file which provides the required degree of flow reporting granularity, and the meter reader(s) will collect the flow data often enough to allow the meter's garbage collection mechanism to maintain a stable level of memory usage. In the worst case traffic may increase to the point where the meter is in danger of running completely out of flow memory. The meter implementor must decide how to handle this, for example by switching to a default (extremely coarse granularity) rule set, by sending a trap message to the manager, or by attempting to dump flow data to the meter reader. Users of the Traffic Flow Measurement system should analyse their requirements carefully and assess for themselves whether it is more important to attempt to collect flow data at normal granularity (increasing the collection frequency as needed to keep up with traffic volumes), or to accept flow data with a coarser granularity. Similarly, it may be acceptable to lose flow data for a short time in return for being sure that the meter keeps running properly, i.e. is not overwhelmed by rising traffic levels. 6 Managers A manager configures meters and controls meter readers. It does this via the interactions described below. 6.1 Between Manager and Meter: Control Functions - DOWNLOAD RULE SET: A meter may hold an array of rule sets. One of these, the 'default' rule set, is built in to the meter and cannot be changed; the others must be downloaded by the manager. A manager may use any suitable protocol exchange to achieve this, for example an FTP file transfer or a series of SNMP SETs, one for each row of the rule set. Brownlee, Mills, Ruth [Page 31] INTERNET-DRAFT Traffic Flow Measurement: Architecture July 98 - SPECIFY METER TASK: Once the rule sets have been downloaded, the manager must instruct the meter which rule sets will be the 'current' and 'standby' ones for each task the meter is to perform. - SET HIGH WATER MARK: A percentage of the flow table capacity, used by the meter to determine when to switch to its standby rule set (so as to increase the granularity of the flows and conserve the meter's flow memory). Once this has happened, the manager may also change the polling frequency or the meter's control parameters (so as to increase the rate at which the meter can recover memory from idle flows). If the high traffic levels persist, the meter's normal rule set may have to be rewritten to permanently reduce the reporting granularity. - SET FLOW TERMINATION PARAMETERS: The meter should have the good sense in situations where lack of resources may cause data loss to purge flow records from its tables. Such records may include: - Flows that have already been reported to all registered meter readers, and show no activity since the last report, - Oldest flows, or - Flows with the smallest number of observed packets. - SET INACTIVITY TIMEOUT: This is a time in seconds since the last packet was seen for a flow. Flow records may be reclaimed if they have been idle for at least this amount of time, and have been collected in accordance with the current collection criteria. 6.2 Between Manager and Meter Reader: Control Functions Because there are a number of parameters that must be set for traffic flow measurement to function properly, and viable settings may change as a result of network traffic characteristics, it is desirable to have dynamic network management as opposed to static meter configurations. Many of these operations have to do with space tradeoffs - if memory at the meter is exhausted, either the collection interval must be decreased or a coarser granularity of aggregation must be used to reduce the number of active flows. Increasing the collection interval effectively stores data in the meter; usage data in transit is limited by the effective bandwidth of the virtual link between the meter and the meter reader, and since these Brownlee, Mills, Ruth [Page 32] INTERNET-DRAFT Traffic Flow Measurement: Architecture July 98 limited network resources are usually also used to carry user data (the purpose of the network), the level of traffic flow measurement traffic should be kept to an affordable fraction of the bandwidth. ("Affordable" is a policy decision made by the Network Operations personnel). At any rate, it must be understood that the operations below do not represent the setting of independent variables; on the contrary, each of the values set has a direct and measurable effect on the behaviour of the other variables. Network management operations follow: - MANAGER and METER READER IDENTIFICATION: The manager should ensure that meters are read by the correct set of meter readers, and take steps to prevent unauthorised access to usage information. The meter readers so identified should be prepared to poll if necessary and accept data from the appropriate meters. Alternate meter readers may be identified in case both the primary manager and the primary meter reader are unavailable. Similarly, alternate managers may be identified. - REPORTING INTERVAL CONTROL: The usual reporting interval should be selected to cope with normal traffic patterns. However, it may be possible for a meter to exhaust its memory during traffic spikes even with a correctly set reporting interval. Some mechanism should be available for the meter to tell the manager that it is in danger of exhausting its memory (by declaring a 'high water' condition), and for the manager to arbitrate (by decreasing the polling interval, letting nature take its course, or by telling the meter to ask for help sooner next time). - GRANULARITY CONTROL: Granularity control is a catch-all for all the parameters that can be tuned and traded to optimise the system's ability to reliably measure and store information on all the traffic (or as close to all the traffic as an administration requires). Granularity - Controls the amount of address information identifying each flow, and - Determines the number of buckets into which user traffic will be lumped together. Since granularity is controlled by the meter's current rule set, the manager can only change it by requesting the meter to switch to a different rule set. The new rule set could be downloaded when required, or it could have been downloaded as part of the meter's initial configuration. Brownlee, Mills, Ruth [Page 33] INTERNET-DRAFT Traffic Flow Measurement: Architecture July 98 - FLOW LIFETIME CONTROL: Flow termination parameters include timeout parameters for obsoleting inactive flows and removing them from tables, and maximum flow lifetimes. This is intertwined with reporting interval and granularity, and must be set in accordance with the other parameters. 6.3 Exception Conditions Exception conditions must be handled, particularly occasions when the meter runs out of space for flow data. Since - to prevent an active task from counting any packet twice - packets can only be counted in a single flow, discarding records will result in the loss of information. The mechanisms to deal with this are as follows: - METER OUTAGES: In case of impending meter outages (controlled restarts, etc.) the meter could send a trap to the manager. The manager could then request one or more meter readers to pick up the data from the meter. Following an uncontrolled meter outage such as a power failure, the meter could send a trap to the manager indicating that it has restarted. The manager could then download the meter's correct rule set and advise the meter reader(s) that the meter is running again. Alternatively, the meter reader may discover from its regular poll that a meter has failed and restarted. It could then advise the manager of this, instead of relying on a trap from the meter. - METER READER OUTAGES: If the collection system is down or isolated, the meter should try to inform the manager of its failure to communicate with the collection system. Usage data is maintained in the flows' rolling counters, and can be recovered when the meter reader is restarted. - MANAGER OUTAGES: If the manager fails for any reason, the meter should continue measuring and the meter reader(s) should keep gathering usage records. - BUFFER PROBLEMS: The network manager may realise that there is a 'low memory' condition in the meter. This can usually be attributed to the interaction between the following controls: - The reporting interval is too infrequent, or - The reporting granularity is too fine. Brownlee, Mills, Ruth [Page 34] INTERNET-DRAFT Traffic Flow Measurement: Architecture July 98 Either of these may be exacerbated by low throughput or bandwidth of circuits carrying the usage data. The manager may change any of these parameters in response to the meter (or meter reader's) plea for help. 6.4 Standard Rule Sets Although the rule table is a flexible tool, it can also become very complex. It may be helpful to develop some rule sets for common applications: - PROTOCOL TYPE: The meter records packets by protocol type. This will be the default rule table for Traffic Flow Meters. - ADJACENT SYSTEMS: The meter records packets by the MAC address of the Adjacent Systems (neighbouring originator or next-hop). (Variants on this table are "report source" or "report sink" only.) This strategy might be used by a regional or backbone network which wants to know how much aggregate traffic flows to or from its subscriber networks. - END SYSTEMS: The meter records packets by the IP address pair contained in the packet. (Variants on this table are "report source" or "report sink" only.) This strategy might be used by an End System network to get detailed host traffic matrix usage data. - TRANSPORT TYPE: The meter records packets by transport address; for IP packets this provides usage information for the various IP services. - HYBRID SYSTEMS: Combinations of the above, e.g. for one interface report End Systems, for another interface report Adjacent Systems. This strategy might be used by an enterprise network to learn detail about local usage and use an aggregate count for the shared regional network. 7 Security Considerations 7.1 Threat Analysis A traffic flow measurement system may be subject to the following kinds of attacks: Brownlee, Mills, Ruth [Page 35] INTERNET-DRAFT Traffic Flow Measurement: Architecture July 98 - UNAUTHORIZED USE OF SYSTEM RESOURCES: An attacker may wish to gain advantage or cause mischief (e.g. denial of service) by subverting any of the system elements - meters, meter readers or managers. - UNAUTHORIZED DISCLOSURE OF DATA: Any data that is sensitive to disclosure can be read through active or passive attacks unless it is suitably protected. Usage data may or may not be of this type. Control messages, traps, etc. are not likely to be considered sensitive to disclosure. - UNAUTHORIZED ALTERATION, REPLACEMENT OR DESTRUCTION OF DATA: Similarly, any data whose integrity is sensitive can be altered, replaced/injected or deleted through active or passive attacks unless it is suitably protected. Attackers may modify message streams to falsify usage data or interfere with the proper operation of the traffic flow measurement system. Therefore, all messages, both those containing usage data and those containing control data, should be considered vulnerable to such attacks. 7.2 Countermeasures The following countermeasures are recommended to address the possible threats enumerated above: - UNAUTHORIZED USE OF SYSTEM RESOURCES is countered through the use of authentication and access control services. - UNAUTHORIZED DISCLOSURE OF DATA is countered through the use of a confidentiality (encryption) service. - UNAUTHORIZED ALTERATION, REPLACEMENT OR DESTRUCTION OF DATA is countered through the use of an integrity service. A Traffic Measurement system must address all of these concerns. Since a high degree of protection is required, the use of strong cryptographic methodologies is recommended. The security requirements for communication between pairs of traffic measurmement system elements are summarized in the table below. It is assumed that meters do not communicate with other meters, and that meter readers do not communicate directly with other meter readers (if synchronization is required, it is handled by the manager, see Section 2.5). Each entry in the table indicates which kinds of security services are required. Basically, the requirements are as follows: Brownlee, Mills, Ruth [Page 36] INTERNET-DRAFT Traffic Flow Measurement: Architecture July 98 Security Service Requirements for RTFM elements +------------------------------------------------------------------+ | from\to | meter | meter reader | application | manager | |---------+--------------+--------------+-------------+------------| | meter | N/A | authent | N/A | authent | | | | acc ctrl | | acc ctrl | | | | integrity | | | | | | confid ** | | | |---------+--------------+--------------+-------------+------------| | meter | authent | N/A | authent | authent | | reader | acc ctrl | | acc ctrl | acc ctrl | | | | | integrity | | | | | | confid ** | | |---------+--------------+--------------+-------------+------------| | appl | N/A | authent | | | | | | acc ctrl | ## | ## | |---------+--------------+--------------+-------------+------------| | manager | authent | authent | ## | authent | | | acc ctrl | acc ctrl | | acc ctrl | | | integrity | integrity | | integrity | +------------------------------------------------------------------+ N/A = Not Applicable ** = optional ## = outside RTFM scope - When any two elements intercommunicate they should mutually authenticate themselves to one another. This is indicated by 'authent' in the table. Once authentication is complete, an element should check that the requested type of access is allowed; this is indicated on the table by 'acc ctrl.' - Whenever there is a transfer of information its integrity should be protected. - Whenever there is a transfer of usage data it should be possible to ensure its confidentiality if it is deemed sensitive to disclosure. This is indicated by 'confid' in the table. Security protocols are not specified in this document. The system elements' management and collection protocols are responsible for providing sufficient data integrity, confidentiality, authentication and access control services. Brownlee, Mills, Ruth [Page 37] INTERNET-DRAFT Traffic Flow Measurement: Architecture July 98 8 APPENDICES 8.1 Appendix A: Network Characterisation Internet users have extraordinarily diverse requirements. Networks differ in size, speed, throughput, and processing power, among other factors. There is a range of traffic flow measurement capabilities and requirements. For traffic flow measurement purposes, the Internet may be viewed as a continuum which changes in character as traffic passes through the following representative levels: International | Backbones/National --------------- / \ Regional/MidLevel ---------- ---------- / \ \ / / \ Stub/Enterprise --- --- --- ---- ---- ||| ||| ||| |||| |||| End-Systems/Hosts xxx xxx xxx xxxx xxxx Note that mesh architectures can also be built out of these components, and that these are merely descriptive terms. The nature of a single network may encompass any or all of the descriptions below, although some networks can be clearly identified as a single type. BACKBONE networks are typically bulk carriers that connect other networks. Individual hosts (with the exception of network management devices and backbone service hosts) typically are not directly connected to backbones. REGIONAL networks are closely related to backbones, and differ only in size, the number of networks connected via each port, and geographical coverage. Regionals may have directly connected hosts, acting as hybrid backbone/stub networks. A regional network is a SUBSCRIBER to the backbone. STUB/ENTERPRISE networks connect hosts and local area networks. STUB/ENTERPRISE networks are SUBSCRIBERS to regional and backbone networks. END SYSTEMS, colloquially HOSTS, are SUBSCRIBERS to any of the above networks. Providing a uniform identification of the SUBSCRIBER in finer granularity than that of end-system, (e.g. user/account), is beyond the scope of the current architecture, although an optional attribute in the Brownlee, Mills, Ruth [Page 38] INTERNET-DRAFT Traffic Flow Measurement: Architecture July 98 traffic flow measurement record may carry system-specific 'user identification' labels so that meters can implement proprietary or non-standard schemes for the attribution of network traffic to responsible parties. 8.2 Appendix B: Recommended Traffic Flow Measurement Capabilities Initial recommended traffic flow measurement conventions are outlined here according to the following Internet building blocks. It is important to understand what complexity reporting introduces at each network level. Whereas the hierarchy is described top-down in the previous section, reporting requirements are more easily addressed bottom-up. End-Systems Stub Networks Enterprise Networks Regional Networks Backbone Networks END-SYSTEMS are currently responsible for allocating network usage to end-users, if this capability is desired. From the Internet Protocol perspective, end-systems are the finest granularity that can be identified without protocol modifications. Even if a meter violated protocol boundaries and tracked higher-level protocols, not all packets could be correctly allocated by user, and the definition of user itself varies widely from operating system to operating system (e.g. how to trace network usage back to users from shared processes). STUB and ENTERPRISE networks will usually collect traffic data either by end-system network address or network address pair if detailed reporting is required in the local area network. If no local reporting is required, they may record usage information in the exit router to track external traffic only. (These are the only networks which routinely use attributes to perform reporting at granularities finer than end-system or intermediate-system network address.) REGIONAL networks are intermediate networks. In some cases, subscribers will be enterprise networks, in which case the intermediate system network address is sufficient to identify the regional's immediate subscriber. In other cases, individual hosts or a disjoint group of hosts may constitute a subscriber. Then end-system network address pairs need to be tracked for those subscribers. When the source may be an aggregate entity (such as a network, or adjacent router representing traffic from a world of hosts beyond) and the destination is a singular entity (or vice versa), the meter is said to be operating as a HYBRID system. Brownlee, Mills, Ruth [Page 39] INTERNET-DRAFT Traffic Flow Measurement: Architecture July 98 At the regional level, if the overhead is tolerable it may be advantageous to report usage both by intermediate system network address (e.g. adjacent router address) and by end-system network address or end-system network address pair. BACKBONE networks are the highest level networks operating at higher link speeds and traffic levels. The high volume of traffic will in most cases preclude detailed traffic flow measurement. Backbone networks will usually account for traffic by adjacent routers' network addresses. 8.3 Appendix C: List of Defined Flow Attributes This Appendix provides a checklist of the attributes defined to date; others will be added later as the Traffic Measurement Architecture is further developed. 0 Null 1 Flow Subscript Integer Flow table info 4 Source Interface Integer Source Address 5 Source Adjacent Type Integer 6 Source Adjacent Address String 7 Source Adjacent Mask String 8 Source Peer Type Integer 9 Source Peer Address String 10 Source Peer Mask String 11 Source Trans Type Integer 12 Source Trans Address String 13 Source Trans Mask String 14 Destination Interface Integer Destination Address 15 Destination Adjacent Type Integer 16 Destination Adjacent Address String 17 Destination AdjacentMask String 18 Destination PeerType Integer 19 Destination PeerAddress String 20 Destination PeerMask String 21 Destination TransType Integer 22 Destination TransAddress String 23 Destination TransMask String 26 Rule Set Number Integer Meter attribute 27 Forward Bytes Integer Source-to-Dest counters 28 Forward Packets Integer Brownlee, Mills, Ruth [Page 40] INTERNET-DRAFT Traffic Flow Measurement: Architecture July 98 29 Reverse Bytes Integer Dest-to-Source counters 30 Reverse Packets Integer 31 First Time Timestamp Activity times 32 Last Active Time Timestamp 33 Source Subscriber ID String Session attributes 34 Destination Subscriber ID String 35 Session ID String 36 Source Class Integer 'Computed' attributes 37 Destination Class Integer 38 Flow Class Integer 39 Source Kind Integer 40 Destination Kind Integer 41 Flow Kind Integer 50 MatchingStoD Integer PME variable 65 .. 'Extended' attributes (to be defined by the RTFM working group) 127 8.4 Appendix D: List of Meter Control Variables Meter variables: Flood Mark Percentage Inactivity Timeout (seconds) Integer 'per task' variables: Current Rule Set Number Integer Standby Rule Set Number Integer High Water Mark Percentage 'per reader' variables: Reader Last Time Timestamp 8.5 Appendix E: Changes Introduced Since RFC 2063 The first version of the Traffic Flow Measurement Architecture was published as RFC 2063 in January 1997. The most significant changes made since then are summarised below. - A Traffic Meter can now run multiple rule sets concurrently. This makes a meter much more useful, and required only minimal changes to the architecture. Brownlee, Mills, Ruth [Page 41] INTERNET-DRAFT Traffic Flow Measurement: Architecture July 98 - 'NoMatch' replaces 'Fail' as an action. This name was agreed to at the Working Group 1996 meeting in Montreal; it better indicates that although a particular match has failed, it may be tried again with the packet's addresses reversed. - The 'MatchingStoD' attribute has been added. This is a Packet Matching Engine (PME) attribute indicating that addresses are being matched in StoD (i.e. 'wire') order. It can be used to perform different actions when the match is retried, thereby simplifying some kinds of rule sets. It was discussed and agreed to at the San Jose meeting in 1996. - Computed attributes (Class and Kind) may now be tested within a rule set. This lifts an unneccessary earlier restriction. - The list of attribute numbers has been extended to define ranges for 'basic' attributes (in this document) and 'extended' attributes (currently being developed by the RTFM Working Group). - The 'Security Considerations' section has been completely rewritten. It provides an evaluation of traffic measurement security risks and their countermeasures. 9 Acknowledgments An initial draft of this document was produced under the auspices of the IETF's Internet Accounting Working Group with assistance from SNMP, RMON and SAAG working groups. This version documents the implementation work done by the Internet Accounting Working Group, and is intended to provide a starting point for the Realtime Traffic Flow Measurement Working Group. Particular thanks are due to Stephen Stibler (IBM Research) for his patient and careful comments during the preparation of this draft. 10 References [1] Mills, C., Hirsch, G. and Ruth, G., "Internet Accounting Background", RFC 1272, Bolt Beranek and Newman Inc., Meridian Technology Corporation, November 1991. [2] International Standards Organisation (ISO), "Management Framework," Part 4 of Information Processing Systems Open Systems Interconnection Basic Reference Model, ISO 7498-4, 1994. Brownlee, Mills, Ruth [Page 42] INTERNET-DRAFT Traffic Flow Measurement: Architecture July 98 [3] IEEE 802.3/ISO 8802-3 Information Processing Systems - Local Area Networks - Part 3: Carrier sense multiple access with collision detection (CSMA/CD) access method and physical layer specifications, 2nd edition, September 21, 1990. [4] Brownlee, N., "Traffic Flow Measurement: Meter MIB," Internet Draft, 'Working draft' to become an experimental RFC. 11 Author's Addresses Nevil Brownlee Information Technology Systems & Services The University of Auckland Phone: +64 9 373 7599 x8941 E-mail: n.brownlee@auckland.ac.nz Cyndi Mills GTE Laboratories, Inc Phone: +1 617 466 4278 E-mail: cmills@gte.com Greg Ruth GTE Laboratories, Inc Phone: +1 617 466 2448 E-mail: gruth@gte.com Expires January 1999 Brownlee, Mills, Ruth [Page 43]