IPFIX working group Internet Draft L. Deri Document: draft-deri-impl-00.txt NETikos S.p.A. Expires: January 2005 July 2004 IPFIX Implementation Notes Status of this Memo This document is an Internet-Draft and is in full conformance with all provisions of Section 10 of RFC2026. Internet-Drafts are working documents of the Internet Engineering Task Force (IETF), its areas, and its working groups. Note that other groups may also distribute working documents as Internet- Drafts. The list of current Internet-Drafts can be accessed at http://www.ietf.org/ietf/1id-abstracts.txt The list of Internet-Draft Shadow Directories can be accessed at http://www.ietf.org/shadow.html. Abstract Providing an early implementation report is important in order to help validating a standard before its finalization. This document describes the lessons learnt while implementing both an IPFIX probe and collector. Its goals are manifold and include feedback to the IPFIX working group, in addition to suggestions for tackling open protocol issues that can be addressed in future IPFIX drafts. Conventions used in this document The key words "MUST", "MUST NOT", "REQUIRED", "SHALL", "SHALL NOT", "SHOULD", "SHOULD NOT", "RECOMMENDED", "MAY", and "OPTIONAL" in this document are to be interpreted as described in RFC-2119. Table of Contents 1. Introduction...................................................2 2. IPFIX Implementation...........................................2 2.1 General Implementation Notes...............................3 2.2 Vendor-specific Extensions.................................4 2.3 SCTP Protocol Implementation...............................4 3. Implementation Evaluation......................................6 Deri Expires - January 2005 [Page 1] IPFIX Implementation Notes July 2004 4. Open IPFIX Issues..............................................7 5. Final Remarks..................................................9 6. Code Availability..............................................9 References........................................................9 Acknowledgments..................................................10 Author's Addresses...............................................10 1. Introduction The IETF IPFIX working group [ipfix-wg] is working towards the definition of a standard for the export of IP flows. At present a draft of the IPFIX protocol specification has been released. The interest of the author for IPFIX is because in the past years he has developed two open source applications: - A passive traffic monitoring application named ntop [ntop] that can also act as a NetFlow collector. - A libpcap-based [pcap] NetFlow v5/v9 probe named nProbe [nprobe]. In order to both evaluate IPFIX and provide feedback to the working group, the author has extended both applications with IPFIX support so that it is possible to have control over both the probe and the collector. The author believes that this is a typical scenario at least for early IPFIX applications, as they will likely be derived from existing NetFlow applications, considering that IPFIX is very similar to NetFlow v9. For this reason the paper does not cover NetFlow or general flow-based implementation issues. Instead it focuses on the effort required to enhance an existing NetFlow application so that it can also support IPFIX. Since IPFIX uses SCTP [sctp] as default transport, in order to ease the development cycle until SCTP will be supported in all the main operating systems, Linux has been used as reference platform. However this report should be considered platform-neutral, as both probe and collector do not use Linux-only facilities. The author is not aware of any IPFIX implementations so interoperability issues have not been addressed here. 2. IPFIX Implementation Implementing IPFIX is rather straightforward if the starting point is a NetFlow v9 application. This is because NetFlow v9 has been used as base for the IPFIX protocol [ipfix-eval]. In a nutshell [ipfix-nf- eval] IPFIX is basically NetFlow v9 over SCTP with little differences such as templates definition when non-IETF defined flow fields are used. Deri Expires - January 2005 [Page 2] IPFIX Implementation Notes July 2004 As many NetFlow implementations do not support v9 yet, it is worth to spend a few words about the complexity required to enhance an existing NetFlow application (e.g. based on v5) to v9 [nfv9-spec]. The main difference between v5 and v9 is the template concept, in addition to other minor issues such as little changes in the flow header (e.g. the sequence number). Implementing templates is a major issue and it definitively costs more, in terms of code complexity, to move from NetFlow v5 to v9 than from v9 to IPFIX. This is for several reasons: - The probe cannot guess in advance the template formats so all the possible fields combinations are possible. Due to the dynamic flow size, the probe must carefully allocate buffers and make sure that exported flows can fit in a single UDP packet. - Optimizing template transmission is a complex activity in particular when the probe needs to send out flows to various collectors. For this reasons it is recommended that the probe does not send more that a few different flow/template formats otherwise the code complexity can increase significantly. - From the performance point of view, the probe must be able to collect all the possible flow information even if very few fields can be emitted. For instance VLAN tags were not supported in NetFlow v5 so the VLAN tag was ignored. Instead VLANs are supported in V9, so the probe must take them into account even if the VLAN tag will not be exported in the flow. Optimizing these kind of activities is very time consuming and often the results are not that great in term of performance and memory gain. - Collectors must be prepared to handle all the possible flows without breaking their logic. For instance a collector should not assume that the source/destination IP address are contained in the received flow as with previous NetFlow versions. The following sections describe the effort required to implement IPFIX starting from NetFlow v9 according to the role, either probe or collector. 2.1 General Implementation Notes This section contains general implementation notes that are applied to both probe and collector. For both sides implementing IPFIX support basically means to: - Implement vendor-specific extensions. - Support the SCTP protocol. Deri Expires - January 2005 [Page 3] IPFIX Implementation Notes July 2004 IPFIX support has been implemented according to the following documents: [ipfix-info] and [ipfix-spec]. Technical documentation about IPFIX is rather straightforward to undertstand in particular for those who are familiar with NetFlow as IPFIX is often a 1:1 mapping with the corresponding NetFlow v9 specification. 2.2 Vendor-specific Extensions Contrary to NetFlow v9, IPFIX vendor-specific extensions are specified by means of the vendor specified field type defined using the IANA enterprise number (PEN) [pen]. Although the use of PEN numbers is general (e.g. it has been used into the SNMP MIB II for the sysObjectID [mib2]), it has the drawback of requiring a different template format from IETF-defined fields where a numeric identifier is used. This difference requires that both probe and collector support both formats, with and without the PEN. It should be investigated whether it is feasible to merge both template formats (IETF and vendor-specific extensions) into one, so that the implementation complexity is reduced. For instance the IETF format can be replaced with the vendor-specific one, using an enterprise number for IETF. Although this is a minor issue, it definitively simplifies the development work and troubleshooting. 2.3 SCTP Protocol Implementation The SCTP protocol is a reliable, congestion-aware transport protocol based on IP. It sports several features such as acknowledged error- free retransmission of messages, detection of data corruption and duplication. It is basically a general-purpose transport for message- oriented applications such as flow-based applications based on IPFIX. The flow peers handle SCTP very similar to TCP as they need to - Bind to a port. - Connect with a peer. - Start sending data. - Close the connection. The main difference with TCP is that in case of network events such as disconnection of the remote peer, the SCTP layer continues to work even if it reports the problem to the sender. This means that an application does not have to handle reconnection with the remote peer as this is done transparently by the transport when the remote peer is available although some emitted flows might be lost. Still the application must register with SCTP in order to receive events about the underlying network connection. This is because in case of reconnection, the application has to retransmit the template records so it is necessary to be informed about the state of the network transport. From the programming point of view, SCTP is very similar to TCP/UDP so little code needs to be changed while migrating existing Deri Expires - January 2005 [Page 4] IPFIX Implementation Notes July 2004 applications to SCTP. The main issues with SCTP support are related to the shift from a pure message oriented protocol such as UDP to a connection-oriented protocol. In fact UDP-based applications do not have to keep track of per-connection information nor handle transport messages. This has definitively a major impact on application complexity and performance that depends on the role of the application as described below. Probe Side Probes need to keep track of peer connections that means: - Templates need to be retransmitted per-connection every time a connection is reestablished; using a connectionless protocol such as UDP, this was a periodic operation as there is no simple way to detect whether a collector disconnected. Hence using SCTP, template sending can be optimized, as it is not really necessary to periodically send templates, because it is now possible to detected whether the collector has disconnected. On the other hand as a probe can send flows to several collectors (e.g. in round-robin or as a reflector) it must keep track of per- collector templates transmission. This means that if collector X reconnects, the probe must send the template only to this collector and not to all collectors. - The IPFIX specification allows a probe to open a connection for sending templates and another for sending flows, making the whole setup even trickier. - Transmitting templates is a time consuming activity for both the application and the transport, as they need to be sent quite often. For instance Cisco IOS sends NetFlow v9 templates every 10 flow packets (ip flow-export template refresh-rate IOS parameter [nfv9]). Sending templates using SCTP is a little more costly than with UDP, in particular if IPSec is used, so every transmission that is not necessary should be carefully avoided. Nevertheless, sending templates only when there is a reconnection it significantly reduces the network traffic with respect to a connectionless transport. Collector Side There are no major changes to be supported from the transport point of view with respect to NetFlow v9, beside little coding necessary to support SCTP. As stated before, SCTP is currently supported only on some selected OSs such as Linux and Solaris. At the moment it is not clear whether both popular operating systems such as Windows will ever support it natively (Microsoft has no plan to support SCTP in the current and next Windows version, codename Longhorn), and network manufacturers will implement it on their routers. In addition, being SCTP a totally Deri Expires - January 2005 [Page 5] IPFIX Implementation Notes July 2004 new protocol, it is unclear what kind of problems it will suffer when deployed in production environments or whether it will be vulnerable to network attacks. This makes the IPFIX future not too bright, as SCTP is the mandatory transport protocol. It is clear that the IPFIX WG is betting a lot on this protocol, however making it mandatory is probably a very strong position that will prevent IPFIX diffusion at least in its early days. 3. Implementation Evaluation From the coding point of view, implementing IPFIX without vendor- specific extensions is not much different from NetFlow v9 support. In fact the only major difference is just the transport that has to be handled slightly differently from UDP but that has the advantage of reducing template periodic transmission. In order to evaluate differences between transports, some packets (the sample contained 50'000 packets sent from the same address towards random destinations) have been captured and reproduced using nProbe. In order to evaluate the two transports regardless of the template transmission, templates are sent periodically on both transports, although templates transmission affects the test very little. Note that as explained before, periodic template transmission is not really necessary when SCTP is used, unless there is a transport disconnection/reconnection. The test outcome is shown in the following table. +-------------+-----------+-----------------+ | | UDP | SCTP | +-------------+-----------+-----------------+ | Packets | 1'667 | 2'514 [+150%] | +-------------+-----------+-----------------+ | Bytes | 2'509'540 | 2'595'952 [+3%] | +-------------+-----------+-----------------+ Table 1 - UDP vs. SCTP Network Usage Basically with SCTP the number of packets is significantly increased with respect to UDP. This is because with SCTP the flow collector acknowledges almost immediately all the received packets. However as these acknowledge packets are rather small (62 bytes) they have very little impact on the total amount of data transfer but just on the number of packets sent. Furthermore it is worth to remark that if there is no traffic (i.e. flows are not sent) for some time (the default is 30 seconds in the current Linux 2.6 implementation), the SCTP stack on the probe side sends a heartbeat message that is acknowledged by the collector side. Heartbeat messages are very small (86 bytes) however they influence the amount of traffic exchanged. Deri Expires - January 2005 [Page 6] IPFIX Implementation Notes July 2004 From the performance point of view, IPFIX is basically as fast as NetFlow v9. This is because there are not many differences between the two protocols. However it has to be remarked that the use of templates has some little performance drawbacks with respect to 'static' NetFlow as v5. This is because: - Applications (probes and collectors) know in advance the data format so they can be optimized and tuned in advance. - Several routines that handle templates (definition, transmission, check) are avoided, so there are less per-flow CPU cycles needed. 4. Open IPFIX Issues As stated already, IPFIX is very similar to NetFlow v9. On one hand this is an advantage as it allows existing NetFlow v9 applications to be made IPFIX aware with little effort, however on the other hand this is a serious limitation. In fact the NetFlow design has evolved very little since its introduction in 90s, and most of the changes are minor and often due to the need to support new features while maintaining a wide compatibility with past versions. This explains for instance why NetFlow v7 is very similar to v1, and because the first Cisco IOS implementation of v9 is basically a "port" of v5 over templates. On the other hand, IPFIX is a brand new effort to define a standard flow-based protocol that should capitalize on experience and lessons learnt after ten years of NetFlow. This obviously did not fully happen for many reasons, as this protocol is far too similar to v9 and not really introduces new features. The following list summarizes some of the missing features that according to the author should be introduced before the protocol is finalized: - As with NetFlow, the IPFIX flow format is fixed. Consider the following scenario: a user requires in-depth ICMP flow analysis (e.g. ICMP code and type). There are basically two solutions: either a probe/collector agree on two different templates (one for ICMP and one for non-ICMP flows) or they agree on a common template for all protocols with the outcome that some flow fields are empty when non-ICMP flows are emitted. The ideal solution to this kind of scenario is dynamic flows, where emitted flows can have all the template fields or a subset of them according to the flow kind. This means that non-ICMP flows will not have ICMP code and type, that instead will be present on ICMP flows that will lack source and destination ports, meaningless for ICMP. Unfortunately due to the way templates and flows are defined, it is not possible to simply handle dynamic flows, unless an "inline" template is embedded in flows as with nFlow [nflow]. nFlow has no concept of template and it embeds in flows the field identifier. This practice slightly increases the flow size, however thanks to both on-the-fly Deri Expires - January 2005 [Page 7] IPFIX Implementation Notes July 2004 flow compression and lack of sending templates, the volume of data transmitted is significantly reduced. This feature will save processing time and bandwidth while enabling probes and collectors to have a rich template format. - IPFIX templates specify just the format of flows and cannot modify the flow header. This means that it is not possible to define flow header attributes. For instance it is not possible to define attributes like MD5 flow message checksum for non- repudiation of flows, or attributes that can be transported in piggy-back instead of putting them into the options template. Adding this capability would enable developers to have the same level of freedom in both header and flow content. - The SCTP protocol is definitively an enhancement over UDP, although it has some severe drawbacks in terms of number of packets transmitted. The main issue is that every packet is acknowledged almost immediately, that can for instance significantly amplify the effect of a DoS attack. Note that NetFlow amplifies the effects of a DoS when small packets are used, because NetFlow flows are larger than packets used for the attack. It should be explored whether SCTP ACKs can be reduced for instance using mechanisms like those used by TCP, in order to reduce the (almost) 1:1 traffic flow (one ACK for each flow received) from the collector to the probe. Finally it should be explored whether it still makes sense to have a flow identifier in the flow header when SCTP is taking care of issues such as flow retransmission or duplicate discard. - IPFIX does not specify a complete, two-way protocol but just a one-way (from probe to collector) format for exported flows. The drawback is that the collector has no way to query the probe configuration or to modify it. This statement is true unless an offline protocol (e.g. via the CLI interface) is used. The probe to inform the collector uses the options template/flow, although this is not an effective solution as: - It does not allow the collector to query the probe but it is the probe that decides when to inform the collector and what to report. - It is used to periodically send information that could be discarded by the collector. It would make more sense to allow the collector to query the probe when needed. It is not rich enough to let the probe inform the collector about issues such as lack of memory for handling incoming flows. A possible solution could either be the approach used in other flow protocols (e.g. sFlow [sflow]) that defines an SNMP MIB implemented by the probe, or enrich the IPFIX specification with a real two-way protocol. Deri Expires - January 2005 [Page 8] IPFIX Implementation Notes July 2004 5. Final Remarks This document reported about implementing IPFIX in both a probe and a collector. It covers issues encountered during the implementation and describes open issues that can be tackled before IPFIX is released that include: - Limited innovation, if any, with respect to existing technologies such as NetFlow. - Use of different data formats for IETF and non-IETF information. - Use of SCTP requires additional coding and is not supported in all platforms. - Lack of configuration and monitoring that can be solved through an SNMP MIB. This paper has also proved that the IPFIX protocol is feasible to implement in existing NetFlow applications and shown two open source applications that can be used by those interested in evaluating IPFIX and developing IPFIX-aware applications. 6. Code Availability The applications described in this document are distributed under the GPL2 license and available from the ntop home page (http://www.ntop.org/). References [ipfix-wg] IETF, "IP Flow Information Export Working Group", http://www.ietf.org/html.charters/ipfix-charter.html. [ipfix-eval] S. Leinen, "Evaluation of Candidate Protocols for IP Flow Information Export (IPFIX)", draft-leinen-ipfix-eval-contrib-02.txt, January 2003. [ipfix-info] P. Calato and others, "Information Model for IP Flow Information Export", draft-ietf-ipfix-info-03.txt, February 2004. [ipfix-nf-eval] B. Claise, "Evaluation Of NetFlow Version 9 Against IPFIX Requirements", draft-claise-ipfix-eval-netflow-04.txt, February 2003. Deri Expires - January 2005 [Page 9] IPFIX Implementation Notes July 2004 [ipfix-spec] B. Claise and others, "IPFIX Protocol Specifications", draft-ietf-ipfix-protocol-03.txt, January 2003. [mib2] K. McCloghrie, and M. Rose, "Management Information Base for Network Management of TCP/IP-based internets: MIB-II", RFC 1213, March 1991. [nfv9] Cisco Systems Inc., "NetFlow v9 Data Format", http://www.cisco.com/univercd/cc/td/doc/product/software/ios120/ 120newft/120limit/120s/120s24/nfv9expf.htm, 2003. [nfv9-spec] B. Claise, "Cisco Systems NetFlow Services Export Version 9", draft-claise-netflow-9-08.txt, April 2004. [ntop] L. Deri and others, "Monitoring Networks using ntop", Proceedings of IM 2001, Seattle, May 2001. [nprobe] L. Deri, "nProbe: an Open Source NetFlow Probe for Gigabit Networks", Proceedings of Terena TNC 2003, Zagreb, May 2003. [nflow] L. Deri, "nFlow", http://www.ntop.org/nFlow/, March 2003. [pen] IANA, "Private Enterprise Numbers Registry", http://www.iana.org/assignments/enterprise-numbers [sctp] R. Stewart and others, "Stream Control Transmission Protocol (SCTP)", RFC 2960, October 2000. [sflow] P. Phaal and others, "sFlow: A Method for Monitoring Traffic in Switched and Routed Networks", RFC 3176, September 2001. Acknowledgments This work was supported in part by the IST project SCAMPI (IST-2001- 32404) funded by the European Union. The author would like to thank Fulvio Risso and Simon Leinen for the valuable discussions and suggestions. Author's Addresses Luca Deri NETikos S.p.A. Via del Brennero km 4 Deri Expires - January 2005 [Page 10] IPFIX Implementation Notes July 2004 56123 Pisa Italy Phone: +39 050 80091 Email: luca.deri@netikos.com Deri Expires - January 2005 [Page 11]