Advantages of using an IPFIX File Format for SIPCLF
draft-niccolini-sipclf-ipfix-02

Abstract

The SIPCLF WG is currently trying to identify file format(s) suitable for its scope as defined in the current charter. The group has so far received proposals for a text format and an indexed-text format that are going to be soon combined. Several people believe that IPFIX could be a valid alternative to these two proposals for several reasons detailed in this document.

This document discusses the main advantages related to the usage of IPFIX file format for SIPCLF. Additionally it gives preliminary indications on the basic set of information elements that would need to be defined by the SIPCLF WG.

Status of this Memo

This Internet-Draft is submitted in full conformance with the provisions of BCP 78 and BCP 79.

Internet-Drafts are working documents of the Internet Engineering Task Force (IETF). Note that other groups may also distribute working documents as Internet-Drafts. The list of current Internet-Drafts is at http://datatracker.ietf.org/drafts/current/.

Internet-Drafts are draft documents valid for a maximum of six months and may be updated, replaced, or obsoleted by other documents at any time. It is inappropriate to use Internet-Drafts as reference material or to cite them other than as “work in progress.”

This Internet-Draft will expire on October 21, 2010.

Copyright Notice

Copyright (c) 2010 IETF Trust and the persons identified as the document authors. All rights reserved.

This document is subject to BCP 78 and the IETF Trust's Legal Provisions Relating to IETF Documents (http://trustee.ietf.org/license-info) in effect on the date of publication of this document. Please review these documents carefully, as they describe your rights and restrictions with respect to this document. Code Components extracted from this document must include Simplified BSD License text as described in Section 4.e of the Trust Legal Provisions and are provided without warranty as described in the Simplified BSD License.

Table of Contents

1.  Introduction
2.  Why IPFIX makes sense for SIPCLF
    2.1.  Thinking about the future
3.  IPFIX File Format
4.  Information Model and Information Elements for SIPCLF
5.  IPFIX file format for SIPCLF: an example visualized
    5.1.  A tool producing an IPFIX file format for SIPCLF
6.  Summary
7.  Security Considerations
8.  IANA Considerations
9.  Acknowledgments
10.  Future improvements of the draft
11.  Informative References
§  Authors' Addresses

1.  Introduction

The SIPCLF WG is chartered to produce a format suitable for logging at any SIP element. The charter explicitly addresses the need to search, merge and summarize log records from one or more possibly diverse elements as well as the need to correlate messages from multiple elements. An additional consideration to be taken into account when defining the file format is the extensibility of SIP. The SIPCLF WG is chartered to identify the fields to appear in a log record and provide one or more formats for encoding those fields. Specifying the mechanics of exchanging, transporting, and storing SIP Common Log Format records is explicitly out of scope.

This document tries to detail why a file format based on IPFIX would make sense for SIPCLF, trying to highlight the main advantages correlated to it. The intention of the authors is to consider both the needs of finding an easy solution today, as well as one that is also ready to accommodate future requirements asily.

2.  Why IPFIX makes sense for SIPCLF

These are the main advantages in favor of IPFIX:

• The IPFIX Information Model. The primary advantage of IPFIX is that it already contains an Information Model, initially based on [RFC5102] (Quittek, J., Bryant, S., Claise, B., Aitken, P., and J. Meyer, “Information Model for IP Flow Information Export,” January 2008.) and updated with IANA http://www.iana.org/assignments/ipfix/ipfix.xhtml. The PSAMP protocol [RFC5476] (Claise, B., Johnson, A., and J. Quittek, “Packet Sampling (PSAMP) Protocol Specifications,” March 2009.) also uses the IPFIX protocol and therefore the same IANA IPFIX registry. Note that, referring to “On the Difference between Information Models and Data Models” [RFC3444] (Pras, A. and J. Schoenwaelder, “On the Difference between Information Models and Data Models,” January 2003.), one could argue whether the IANA IPFIX registry is an information model or a data model.
• IPFIX has a self-describing syntax model that allows the definition of a common set of "standard" fields to be recorded in the SIPCLF record. Specifically, a standard-defined IPFIX record template can be agreed upon and published, which contains the common SIP fields the group wishes to record.
• Both SIP extensibility and vendor-specific requirements (e.g., vendor X want to record proprietary info A/B/C, vendor Y want to record D/E/F info) indicate that more will be needed on top of the "standard" subset (in addition to the fact that the "standard set" will undoubtedly grow over time, too). This will drive the need for a way to type-indicate that vendor-proprietary and future spec info in the file, which the IPFIX format already supports.
• There are a number of applicable tools that already parse IPFIX today from routers/data-boxes. The ability to reuse these tools for SIPCLF scopes would avoid re-inventing the wheel for SIP.
• The requirements that lead to the definition of a file format today will soon be augmented by additional requirements that lead to the definition of a protocol mechanism to export the log record to collectors, then filtering controls/config., etc., which IPFIX has already defined in the past.
• IPFIX supports both binary and ascii record field values. If, at some point in time, SIPCLF needs to support encoding the entire SIP Message, a binary-capable encoding is necessary since SIP Messages can contain binary bodies (e.g., ISUP, QSIG).
• IPFIX records support length encoding, thereby enabling a parser to skip past record fields or entire records without parsing their contents.

These points suggest that, even if today file format has potentially not much overlap with the IPFIX one now, there are advantages in choosing IPFIX file format from the beginning of SIPCLF design. This solution will allow quicker adoption in the beginning (thanks to the tools parsing IPFIX that are already available and can be reused right away) and avoid sub-optimality (e.g., proxy translating formats between each other) in the future.

2.1.  Thinking about the future

Thinking about the future (even if the charter doesn’t address these points), there are high chances that:

• The SIPCLF file format will have to include information elements that are already defined in the IANA IPFIX registry, combining SIP related information elements with flow related information elements. For example, simply an IP address (refer to Section 5 (IPFIX file format for SIPCLF: an example visualized)).
• The SIPCLF file format will have to include proprietary information elements, as multiple vendors will have to distinguish from each other. IPFIX offers the ability to have proprietary information elements, called Enterprise-Specific Information Elements in [RFC5101] (Claise, B., “Specification of the IP Flow Information Export (IPFIX) Protocol for the Exchange of IP Traffic Flow Information,” January 2008.). In the future,
• In the future, the information contained the SIPCLF file format will have to be transferred (pushed or pulled) in order to do some correlation. While a file can transferred with any protocol, such as FTP, TFTP, RCP, etc., using the IPFIX protocols offers some advantages: for example, a push mechanism and a single collection protocol for the performance management product, avoiding additional costly correlation tools for the integration of IPFIX information elements with SIPCLF file format. When the charter mentions “Furthermore, these log records can also be used to train anomaly detection systems and feed events into a security event management system.”, most tools in the market depend one way or the other on NetFlow version 9 [RFC3954] (Claise, B., “Cisco Systems NetFlow Services Export Version 9,” October 2004.), which is the basis protocol for IPFIX.

3.  IPFIX File Format

A draft on using IPFIX for SIPCLF does not need to completely detail a file format - the "file format" is already defined in [RFC5655] (Trammel, B., Boschi, E., Mark, L., Zseby, T., and A. Wagner, “Specification of the IP Flow Information Export (IPFIX) File Format,” October 2009.) and the SIPCLF one would be based on that. The advantage of this file format is that it was already designed to facilitate interoperability and reusability among a wide variety of storage, processing, and analysis tools.

4.  Information Model and Information Elements for SIPCLF

As defined in [RFC3444] (Pras, A. and J. Schoenwaelder, “On the Difference between Information Models and Data Models,” January 2003.) the main purpose of an Information Model is to model managed objects at a conceptual level, independent of any specific implementations or protocols used to transport the data.

This draft does not yet attempt to formally define information elements for SIPCLF. Below you can find initial reasoning about the "standard set" needed for logging SIP events.

Main information for SIPCLF log record is largely already defined in other documents, e.g., in [RFC3261] (Rosenberg, J., Schulzrinne, H., Camarillo, G., Johnston, A., Peterson, J., Sparks, R., Handley, M., and E. Schooler, “SIP: Session Initiation Protocol,” June 2002.).

There are fundamentally two options depending on how detailed the information elements needs to be. The first possible option is to keep the information elements coarse grained, e.g., for each SIP message that is logged, a Status-Line or Request-Line, an ordered list of pairs of where each pair has a header-name and a header-value, and finally zero or one message-bodies. Additional information could be about where the message was coming from and going to from the transport layer as well as a timestamp. The second option is to have fine grained definition of information elements, e.g., defining SIP to and from tags as elements themselves (the list could be extended to include Call-Id, CSeq, Max-Forwards, Via, Contact, etc.). This draft does not favor one option or another as they both have pros and cons; the authors just want to point out that the definition of such information elements is going to be quite straightforward once the SIPCLF WG has agreed on the level of detail and complexity that SIPCLF log record requires (an example of these fields is available here [I‑D.gurbani‑sipclf‑problem‑statement] (Gurbani, V., Burger, E., Anjali, T., Abdelnur, H., and O. Festor, “The Common Log Format (CLF) for the Session Initiation Protocol (SIP),” January 2010.).

The authors wish to point out that there have been already attempts to define IPFIX information elements for when using IPFIX for SIP previously. This has been discussed already in the IPFIX WG based on an expired draft (draft-huici-ipfix-sipfix-00), the paper that was linked with that draft is referenced here [IM.sipfix] (Anderson, S., Niccolini, S., and D. Hogrefe, “SIPFIX: A Scheme For Distributed SIP Monitoring,” .). This solution clearly goes beyond what is the current chartered scope of SIPCLF WG but it is an useful reference with concrete examples on how the information elements that could be defined (excluding the information elements for the media plane that are out of scope for the SIPCLF WG).

5.  IPFIX file format for SIPCLF: an example visualized

This section is meant to give a short example of how an IPFIX file format would look like in the case of SIPCLF in order to increase understanding of what IPFIX would mean for SIPCLF. The SIPCLF fields used for defining the information elements (IEs) are meant for example purposes only and are taken from available references of the SIPCLF WG [I‑D.gurbani‑sipclf‑problem‑statement] (Gurbani, V., Burger, E., Anjali, T., Abdelnur, H., and O. Festor, “The Common Log Format (CLF) for the Session Initiation Protocol (SIP),” January 2010.).

The information elements used by this example are shown in the table below; new Information Elements necessary for SIPCLF that are not yet registered with IANA are provisionally located within the number space assigned to Private Enterprise Number (PEN) (here registered within the private enterprise number 35566, which is controlled by one of the authors of the draft).

Note: the current example does not include a L4 port number.

The information model in a form to be parsed by a tool processing the IPFIX/SIPCLF would look like the following (in squared brackets you can see the number of bytes for the field):

sipAuthUsername(35566/401)<string>[65535]
sipMethod(35566/402)<unsigned8>[1]
sipRequestURI(35566/403)<string>[65535]
sipFromURI(35566/404)<string>[65535]
sipFromTag(35566/405)<string>[65535]
sipToURI(35566/406)<string>[65535]
sipToTag(35566/407)<string>[65535]
sipCallId(35566/408)<string>[65535]
sipResponseStatus(35566/409)<unsigned16>[2]
sipServerTransaction(35566/410)<string>[65535]
sipClientTransaction(35566/411)<string>[65535]

A request record is then described by the following template:

and a response record by the following template:

A SIPCLF IPFIX file then consists of an IPFIX Message containing the template definitions above, followed by multiple IPFIX Messages containing the records in Data Sets corresponding to those template definitions, as in the figure below:

          +=================================================+
          | IPFIX Message                       seq. 0      |
          | +---------------------------------------------+ |
          | | Message Header (16 bytes)                   | |
          | +---------------------------------------------+ |
          | | Template Set (ID 2)                  5 recs | |
          | |   SIPCLF IPv4 request tmpl          ID 256  | |
          | |   SIPCLF IPv4 auth request tmpl     ID 257  | |
          | |   SIPCLF IPv6 request tmpl          ID 258  | |
          | |   SIPCLF IPv6 auth request tmpl     ID 259  | |
          | |   SIPCLF response tmpl              ID 260  | |
          | +---------------------------------------------+ |
          +=================================================+
          | IPFIX Message                       seq. 0      |
          | +---------------------------------------------+ |
          | | Message Header (16 bytes)                   | |
          | +---------------------------------------------+ |
          | | Data Set (ID 256)                    1 rec  | |
          | |  contains an unauthenticated IPv4 request   | |
          | +---------------------------------------------+ |
          | | Data Set (ID 260)                    1 rec  | |
          | |  contains a response                        | |
          | +---------------------------------------------+ |
          | | Data Set (ID 259)                    2 recs | |
          | |  contains authenticated IPv6 requests       | |
          | +---------------------------------------------+ |
          | | Data Set (ID 260)                    1 rec  | |
          | |  contains a response                        | |
          | +---------------------------------------------+ |
          +=================================================+
          | IPFIX Message                       seq. 5      |
          |                    . . .                        |

Within each data set is one or more records, packed and encoded according to the template. Fixed length fields appear in place without additional framing or length prefixing. Variable length fields use length prefixing. Values 0-254 bytes long can be represented using a single-byte length prefix, 0x00 - 0xFE. Values 0-65516 bytes long can be represented using a three-byte length prefix, 0xFF followed by the length in network byte order. All SIPCLF variable-length fields are strings, and all IPFIX strings are encoded using UTF8.

[Note that the placement of data set and message boundaries are implementation dependent; some implementations may group and emit records by template, some may have only one record per data set or data set per message. If SIPCLF requires strict ordering of records by time, this can be specified, but is not mandatory IPFIX or IPFIX File behavior.]

5.1.  A tool producing an IPFIX file format for SIPCLF

This section describes visually how the input for the tool (that was written by one of the authors of this draft, and will shortly see open-source release) that produces an IPFIX file format for SIPCLF would look like and what would its binary output be. For this purpose we took examples from [RFC3665] (Johnston, A., Donovan, S., Sparks, R., Cunningham, C., and K. Summers, “Session Initiation Protocol (SIP) Basic Call Flow Examples,” December 2003.).

Section 5 (IPFIX file format for SIPCLF: an example visualized) showed the compact definition of the information model that the tool producing the IPFIX file format for SIPCLF uses (see Figure 1 (Information Model).

The additional input to the tool is shown in Figure 3 (Input file); it includes two templates (one for a request and one for a response) and two examples of records (one for a request, one for a response). Both record examples were extracted by messages included in Section 3.1 of [RFC3665] (Johnston, A., Donovan, S., Sparks, R., Cunningham, C., and K. Summers, “Session Initiation Protocol (SIP) Basic Call Flow Examples,” December 2003.).

       template(1234)
       observationTimeSeconds
       sourceIPv4Address
       sipMethod
       sipAuthUsername
       sipRequestURI
       sipFromURI
       sipFromTag
       sipToURI
       sipCallId
       sipServerTransaction
       sipClientTransaction

       template(4321)
       observationTimeSeconds
       sourceIPv4Address
       sipResponseStatus
       sipServerTransaction
       sipClientTransaction
       sipToURI
       sipToTag

       _ipfix_tid: 1234
       observationTimeSeconds: 2010-04-01 01:10:12 +0000
       sourceIPv4Address: 192.168.0.1
       sipMethod: 1
       sipAuthUsername: bob
       sipRequestURI: sip:bob@biloxi.example.com
       sipFromURI: Alice <sip:alice@atlanta.example.com>
       sipFromTag: 9fxced76sl
       sipToURI: Bob <sip:bob@biloxi.example.com>
       sipCallId: 3848276298220188511@atlanta.example.com
       sipServerTransaction: 123
       sipClientTransaction: ABC

       _ipfix_tid: 4321
       observationTimeSeconds: 2010-04-01 01:10:14 +0000
       sourceIPv4Address: 192.168.0.2
       sipResponseStatus: 180
       sipServerTransaction: 123
       sipClientTransaction: ABC
       sipToURI: Bob <sip:bob@biloxi.example.com>
       sipToTag: 8321234356

The binary output (IPFIX file) of the tool is reported in Figure 4 (Output file (base64 encoded)) (base-64 encoded).

AAoBjEvEHxUAAAAAAArJ/gACAIwE0gALAUIABAAIAASBkgABAACK7oGR//8A
AIrugZP//wAAiu6BlP//AACK7oGV//8AAIrugZb//wAAiu6BmP//AACK7oGa
//8AAIrugZv//wAAiu4Q4QAHAUIABAAIAASBmQACAACK7oGa//8AAIrugZv/
/wAAiu6Blv//AACK7oGX//8AAIruBNIArkuz8nTAqAABAQNib2Iac2lwOmJv
YkBiaWxveGkuZXhhbXBsZS5jb20lQWxpY2UgPHNpcDphbGljZUBhdGxhbnRh
LmV4YW1wbGUuY29tPgo5ZnhjZWQ3NnNsIEJvYiA8c2lwOmJvYkBiaWxveGku
ZXhhbXBsZS5jb20+JzM4NDgyNzYyOTgyMjAxODg1MTFAYXRsYW50YS5leGFt
cGxlLmNvbQMxMjMDQUJDEOEAQkuz8nbAqAACALQDMTIzA0FCQyBCb2IgPHNp
cDpib2JAYmlsb3hpLmV4YW1wbGUuY29tPgo4MzIxMjM0MzU2

Figure 5 (Dump file) shows a debug dump, which is meant to illustrate the information that would be available at an IPFIX/SIPCLF collector.

----- template 707070/1234 -----
observationTimeSeconds(322)<dateTimeSeconds>[4]
sourceIPv4Address(8)<ipv4Address>[4]
sipMethod(35566/402)<unsigned8>[1]
sipAuthUsername(35566/401)<string>[65535]
sipRequestURI(35566/403)<string>[65535]
sipFromURI(35566/404)<string>[65535]
sipFromTag(35566/405)<string>[65535]
sipToURI(35566/406)<string>[65535]
sipCallId(35566/408)<string>[65535]
sipServerTransaction(35566/410)<string>[65535]
sipClientTransaction(35566/411)<string>[65535]
----- template 707070/4321 -----
observationTimeSeconds(322)<dateTimeSeconds>[4]
sourceIPv4Address(8)<ipv4Address>[4]
sipResponseStatus(35566/409)<unsigned16>[2]
sipServerTransaction(35566/410)<string>[65535]
sipClientTransaction(35566/411)<string>[65535]
sipToURI(35566/406)<string>[65535]
sipToTag(35566/407)<string>[65535]
----- record 707070/1234 -----
observationTimeSeconds => 2010-04-01 03:10:12 +0200
sourceIPv4Address => 192.168.0.1
sipMethod => 1
sipAuthUsername => bob
sipRequestURI => sip:bob@biloxi.example.com
sipFromURI => Alice <sip:alice@atlanta.example.com>
sipFromTag => 9fxced76sl
sipToURI => Bob <sip:bob@biloxi.example.com>
sipCallId => 3848276298220188511@atlanta.example.com
sipServerTransaction => 123
sipClientTransaction => ABC
----- record 707070/4321 -----
observationTimeSeconds => 2010-04-01 03:10:14 +0200
sourceIPv4Address => 192.168.0.2
sipResponseStatus => 180
sipServerTransaction => 123
sipClientTransaction => ABC
sipToURI => Bob <sip:bob@biloxi.example.com>
sipToTag => 8321234356

6.  Summary

Quoted from Dave Harrington on the mailing list: “IPFIX already provides a protocol and a data modeling language. In addition, [RFC5655] (Trammel, B., Boschi, E., Mark, L., Zseby, T., and A. Wagner, “Specification of the IP Flow Information Export (IPFIX) File Format,” October 2009.) specifies a file format for storing data that has been received in the ipfix file format. The IPFIX File format is designed to facilitate interoperability and reusability among a wide variety of flow storage, processing, and analysis tools.”, let us ask the question differently: why would we have yet another data modeling language?

7.  Security Considerations

To be worked out.

8.  IANA Considerations

None.

9.  Acknowledgments

Cullen Jennings and many others provided insightful discussions, specific comments and much needed corrections. Nico d'Heureuse helped with the RFC 3665 examples.

10.  Future improvements of the draft

Improvements are still needed in this draft in order to better highlight the synergies between SIP and IPFIX. Additionally authors are planning to further work on:

• an example showing how to store a whole bunch of records to highlight the low overhead of the chosen approach;
• an example showing translation to the apache-like format;
• an analysis of space efficiency
• adding references to the actual tool used which will be released soon

11. Informative References

Authors' Addresses