Intrusion Detection Working Group G. Mansfield/D. Curry draft-mansfield-curry-idmef-xmlsmi-00.txt Cyber Solutions/ISS Expires: July 19, 2000 January 20, 2000 Intrusion Detection Message Exchange Format Comparison of SMI and XML Implementations Status of this Memo This document is an Internet-Draft and is in full conformance with all provisions of Section 10 of RFC 2026 [1]. Internet-Drafts are working documents of the Internet Engineering Task Force (IETF), its areas, and its working groups. Note that other groups may also distribute working documents as Internet- Drafts. Internet-Drafts are draft documents valid for a maximum of six months and may be updated, replaced, or obsoleted by other documents at any time. It is inappropriate to use Internet-Drafts as reference material or to cite them other than as "work in progress." The list of current Internet-Drafts can be accessed at http://www.ietf.org/ietf/1id-abstracts.txt The list of Internet-Draft Shadow Directories can be accessed at http://www.ietf.org/shadow.html. Distribution of this memo is unlimited. This Internet-Draft expires July 19, 2000. Abstract The purpose of the Intrusion Detection Message Exchange Format (IDMEF) is to define data formats and exchange procedures for sharing information of interest to intrusion detection and response systems, and to the management systems which may need to interact with them. The goals and requirements of the IDMEF are described in [3]. Two implementations of the IDMEF data format have been proposed: one using the Structure of Management Information (SMI) to describe an SNMP MIB, and the other using a Document Type Definition (DTD) to describe XML documents. Both representations appear to have their good and bad traits, and deciding between them is difficult. To arrive at an informed decision, the working group tasked the authors to identify and analyze the pros and cons of both approaches, and present the results in the form of an Internet-Draft. Internet-Draft IDMEF SMI/XML Comparison January 20, 2000 TABLE OF CONTENTS 1. Conventions used in this document ............................... 4 2. Methods for communicating intrusion detection alert data ........ 4 2.1 Tell-only .................................................. 4 2.2 Tell-and-ask ............................................... 5 3. Overview of proposed implementations ............................ 7 3.1 SMI ........................................................ 7 3.2 XML ........................................................ 8 4. Comparison criteria ............................................. 10 4.1 Representation issues ...................................... 10 4.1.1 Naming ............................................... 10 4.1.1.1 SMI ............................................ 11 4.1.1.2 XML ............................................ 12 4.1.2 Data model ........................................... 12 4.1.2.1 SMI ............................................ 12 4.1.2.2 XML ............................................ 12 4.1.3 Data format .......................................... 14 4.1.3.1 SMI ............................................ 14 4.1.3.2 XML ............................................ 14 4.2 Operational issues ......................................... 14 4.2.1 Bits on the wire ..................................... 14 4.2.1.1 SMI ............................................ 15 4.2.1.2 XML ............................................ 16 4.2.2 Load on the CPU ...................................... 17 4.2.2.1 SMI ............................................ 17 4.2.2.2 XML ............................................ 17 4.3 Implementation issues ...................................... 18 4.3.1 Size of code ......................................... 18 4.3.1.1 SMI ............................................ 18 4.3.1.2 XML ............................................ 19 4.3.2 Availability of code ................................. 19 4.3.2.1 SMI ............................................ 19 4.3.2.2 XML ............................................ 19 4.4 End point issues ........................................... 20 4.4.1 Data display aspects ................................. 20 4.4.1.1 SMI ............................................ 20 4.4.1.2 XML ............................................ 20 4.4.2 Data transfer aspects ................................ 21 4.4.2.1 SMI ............................................ 21 4.4.2.2 XML ............................................ 21 4.5 Deployment issues .......................................... 21 4.5.1 SMI .................................................. 21 4.5.2 XML .................................................. 22 4.6 Transport issues ........................................... 22 4.6.1 TCP/UDP .............................................. 22 4.6.1.1 SMI ............................................ 22 Mansfield/Curry Expires: July 19, 2000 [Page 2] Internet-Draft IDMEF SMI/XML Comparison January 20, 2000 4.6.1.2 XML ............................................ 22 4.6.2 Intrusion Alert Protocol (IAP) ....................... 22 4.6.2.1 SMI ............................................ 23 4.6.2.2 XML ............................................ 23 5. Security Considerations ......................................... 23 6. References ...................................................... 23 7. Authors' Addresses .............................................. 24 Mansfield/Curry Expires: July 19, 2000 [Page 3] Internet-Draft IDMEF SMI/XML Comparison January 20, 2000 1. Conventions used in this document The key words "MUST," "MUST NOT," "REQUIRED," "SHALL," "SHALL NOT," "SHOULD," "SHOULD NOT," "RECOMMENDED," "MAY," and "OPTIONAL" in this document are to be interpreted as described in RFC 2119 [2]. 2. Methods for communicating intrusion detection alert data The requirements document defines the intrusion detection model that we are assuming: One or more sensors monitor some number of data sources for signs of intrusions, and report their observations to one or more analyzers. When an analyzer determines somehow that these observations represent a suspicious event, it sends an alert to one or more managers. A manager may respond to an alert by notifying an operator, investigating further by exchanging data with its peers, querying the source of the alert, or communicating the event to a higher level manager. The format of the alert sent by the analyzer to the manager, and the method of communicating it, are what the IDMEF proposes to standardize. In discussions within the working group, two different modes of operation have been suggested for communicating alert data between analyzers and managers. It should be noted that the IDMEF concerns itself only with the format of the alert, and not the design of the system that delivers them, or the protocols used to do so. However, it is important to have an idea of the communications mode(s) that will be used by intrusion detection systems when choosing an intrusion detection alert format, because some formats may support certain modes better than others. 2.1 Tell-only This mode provides for a unidirectional communications flow, from an analyzer to one or more managers, as shown in Figure 1. (Managers may also pass the alert to other managers, in a hierarchical arrangement.) Mansfield/Curry Expires: July 19, 2000 [Page 4] Internet-Draft IDMEF SMI/XML Comparison January 20, 2000 +--------+ +----------+ +---------+ | | data | | alert | | | Sensor |------->| Analyzer |-------->| Manager | | | | | | | +--------+ +----------+ +---------+ IDS #1 | .....................................................|............... IDS #2 | | alert V +---------+ | | | Manager | | | +---------+ Figure 1. Tell-only mode. When an analyzer detects an event (or some sequence of events) that must be communicated to a manager, it uses an alert message to do this. The analyzer places all the available information about the event(s) into the alert, and sends it to the manager. Once the alert has been sent to the manager, the analyzer generally "forgets" about both the alert and the events that led up to it. The principal advantage to this mode is that analyzers can be kept both simple and small, allowing them to be deployed with less impact on performance, and using smaller and less expensive hardware. The principal disadvantage is that a manager is "stuck" with whatever the analyzer sends it -- this may be not enough information, or it may be too much. The latter case can be particularly problematic; the possibility exists for a poorly configured analyzer to inundate a manager with messages the manager has no use for but cannot ignore. The IDMEF requirements document assumes a tell-only mode for the communication of IDMEF messages (Section 3.1, paragraph 4). Most intrusion detection products on the market today implement the tell-only model. 2.2 Tell-and-ask This mode provides for bidirectional communication -- from an analyzer to one or more managers and also from the managers to the analyzers, for alert data exchange, as shown in Figure 2. Mansfield/Curry Expires: July 19, 2000 [Page 5] Internet-Draft IDMEF SMI/XML Comparison January 20, 2000 +--------+ +----------+ +---------+ | | data | | alert | | | Sensor |------->| Analyzer |--------->| Manager | | | | | | | | | | | query | | | | | |<---------| | | | | | response | | | | | |--------->| | | | | | | | +--------+ +----------+ +---------+ IDS #1 | ......................................................|.............. IDS #2 | | alert V +---------+ | | | Manager | | | +---------+ Figure 2. Tell-and-ask mode. As with the tell-only mode, when an analyzer detects an event (or some sequence of events) that must be communicated to a manager, it uses an alert message to do this. However, instead of sending all the available event information along with the alert, the analyzer may choose to send only the "vital" data about the alert (type, time, priority, etc.). The decision about which information is considered "vital" and which is not will depend on the local circumstances and configuration. If the manager receiving the alert is "interested" in the events that caused it to be generated, it can (either automatically or under the control of an operator) contact the analyzer and request additional information. The manager might ask for: - more information on the alert pointers provided as URLs in the alerts from the agents (using appropriate protocols such as HTTP, FTP, etc.); - more host-related information on the circumstances under which the intrusion/attack was detected -- this may involve fetching further information from the various MIBs of the entity that originated the notification; or - more network-related information on the circumstances under which the intrusion/attack was detected -- this may involve fetching further information from the various MIBs of the relevant network entities in the network. The principal advantage of this mode of operation is its flexibility. Mansfield/Curry Expires: July 19, 2000 [Page 6] Internet-Draft IDMEF SMI/XML Comparison January 20, 2000 Alerts can be made smaller. This addresses the inundation problem described above. Moreover, since the query facility is available the analyzers may choose to retain some or all relevant information, e.g., analysis-logs, packet-dumps, etc. for some period of time. It may also provide any information it has and in which the manager is interested. The principal disadvantage of this mode is that analyzers will be relatively more complex compared to those operating in the tell-only mode. They must store historical data for some "reasonable" period of time, either in memory or on disk, and be able to retrieve that data when asked, all the while still performing their primary function, detection of intrusions. The IDMEF requirements document indicates that the IDMEF data format may support the tell-and-ask communications mode (Section 3.1, paragraph 4). A Remote Monitoring (RMON) device, a fairly popular network managent agent, can be configured to function as an IDS (with limited functionality) and is an example of an IDS that operates in the tell-and-ask mode. 3. Overview of proposed implementations Two implementations of the IDMEF data format have been proposed: one using the Structure of Management Information (SMI) to describe an SNMP MIB, and the other using a Document Type Definition (DTD) to describe XML documents. The implementations are presented briefly in this section; for more detail on either implementation consult the relevant Internet-Drafts [4, 5]. 3.1 SMI Network management involves monitoring and manipulating information about the elements to be managed. These bits and pieces of information are abstracted as Managed Objects (MO). A detailed introduction to the current SNMP Management Framework can be found in [7]. Managed objects are accessed via a virtual information store, termed the Management Information Base or MIB. Objects in the MIB are defined using the mechanisms defined in the SMI. Each resource to be managed is represented as an object in the MIB. The MIB is a database structure in the form of a tree in which each node has a sub-identifier. The identifier of the node is a concatenation of all the sub-identifiers starting at the root down to the node itself. The Structure of Management Information (SMI) defines the general framework within which a MIB may be defined. It identifies the Mansfield/Curry Expires: July 19, 2000 [Page 7] Internet-Draft IDMEF SMI/XML Comparison January 20, 2000 datatypes that can be used in the MIB and specifies how resources within the MIB can be represented and named. A restricted subset of ASN.1 is used for this purpose. The types defined in the SMI are - Simple types: INTEGER, OCTET STRING, OBJECT IDENTIFIER - Structure types: SEQUENCE and SEQUENCE OF Thus, in effect, only scalars and two dimensional arrays of scalars can be represented (though only scalars can be manipulated). Every object has a unique name, its Object Identifier (OID) - the identifier of the corresponding node in the MIB tree. Further, there is an indexing mechanism whereby the information related to a set of similar objects (for example, interfaces or routes) may be represented and accessed as rows in a table. Thus the operational status of an interface will have the name .1.3.6.1.2.2.8 And if the index for the interface that we are interested in is 2 then we would "get" the value of the object named .1.3.6.1.2.2.8.2 The MO definition of the operational status object will also tell us (and the Network Management Station) that the object is of type INTEGER. The proposed SMI-MIB implementation of IDMEF works well in the tell-and-ask mode. The effective naming scheme and potentially fine granularity of the named objects makes it an ideal candidate for sending a small alert message and then respond to queries from the manager. Nevertheless, the SMI-MIB also works in the tell-only mode. But it should be noted that large messages are generally frowned upon by managers/operators as they consume network and system resources, and, when being transported over UDP using an SNMP-like application protocol, large messages will have to be IP-fragmented or broken into multiple messages. Over TCP, this problem can be avoided. 3.2 XML The Extensible Markup Language (XML) is a text markup syntax defined by the World Wide Web Consortium (W3C). It is gaining widespread attention as a language for representing and exchanging documents and data on the Internet. XML is currently being used in a variety of projects, including the Text Encoding Initiative, Microsoft Channel Definition Format, Wireless Application Protocol (WAP) Wireless Markup Language, Chemical Markup Language, Weather Observation Markup Format, Open Financial Exchange, OpenMLS (real estate), Mathematical Markup Language, Electronic Data Interchange (several projects), News Mansfield/Curry Expires: July 19, 2000 [Page 8] Internet-Draft IDMEF SMI/XML Comparison January 20, 2000 Industry Text Format, and a variety of others. For a comprehensive list, of XML applications, see http://www.oasis-open.org/cover/xml.html#xml-osd XML is a metalanguage -- a language for describing other languages -- that enables an application (such as IDMEF) to define its own markup. XML allows the definition of a customized markup language specific to an application. This differs from HTML, for example, in which a fixed set of markup tags with preset meanings must be "adapted" to uses for which they were not intended. Both XML and HTML use tags (identifiers delimited by '<' and '>') and attributes (of the form "name='value'"). But where "

" always means "paragraph" in HTML, it may mean "paragraph," "person," "price," or have no meaning at all in an XML application. Each XML application defines the tags and attributes it needs in an XML Document Type Definition (DTD). Tags are defined to identify the semantic elements of the marked-up data (e.g., paragraphs, tables, figures, section headings, footnotes, chapters, titles, etc.) The DTD also specifies how the semantic elements of the data relate to each other (e.g., a chapter may only have one title, sections may only occur inside chapters, second-level headings must follow a first-level heading, and so on). The XML IDMEF DTD defines the tags and attributes needed to identify the various elements of an intrusion detection alert, as set forth in the data model defined by Debar, Huang, and Donahoo [6]. A complete "document" (alert) might look like: 33 loadmodule forking shell 13243 joe loadmodule /usr/openwin/bin Mansfield/Curry Expires: July 19, 2000 [Page 9] Internet-Draft IDMEF SMI/XML Comparison January 20, 2000 machine.domain.com

123.45.67.89
The proposed XML implementation of the IDMEF allows the analyzers and managers to use either the tell-only or tell-and-ask communications modes. However, tell-and-ask functionality is limited to specific queries and responses, and does not implement a general query interface. 4. Comparison criteria The authors have identified a variety of criteria against which the two implementations should be evaluated. These are presented below together with descriptions of how each criterion is met by the individual implementations. 4.1 Representation issues Representation issues are those factors of the implementations that involve how intrusion detection alerts are represented: naming of alerts, fields in alerts, and alert-related information. 4.1.1 Naming Naming is important to enable querying for object values. Examples include - what is the complete list of hosts that were the target of a scan? - how many TCP-SYNs were sent to host foo.bar.com from network aaa.bbb.ccc.ddd during the attack? There are three characteristics of naming that are important to us: flexibility, granularity, and ease of use: - How flexible is the naming scheme used by the implementation? For example, can objects in lists and tables be named even when the length/size of the list or table is not known? - How granular is the naming scheme used by each implementation? Can individual objects in lists and tables be named? Can entire lists and tables be named? - How easy is it to use the naming scheme in the above cases? Mansfield/Curry Expires: July 19, 2000 [Page 10] Internet-Draft IDMEF SMI/XML Comparison January 20, 2000 4.1.1.1 SMI In the rest of this document, we will cite the example of SNMP as the application protocol that is used to access/exchange SMI-MIB objects. The rationale behind the choice of this example is that SNMP has been around for some time, is reasonably well understood among a larger community, is widely deployed and there is a lot of operational experience. Moreover the application protocol to handle these SMI-MIB objects, if it is not SNMP itself, is likely to be SNMP-like. Nevertheless, it should be noted that as of now SNMP is NOT the chosen application protocol and the authors have no intention of implying that it is. Management involves getting (reading) and setting (writing) the values of (elementary) objects thus SMI supports and requires all objects of interest to be named. If an object is not explicitly or implicitly named (defined) it cannot be accessed. (Note: defining an object does not mean that it will be accessible.) The naming offered by SMI is flexible to the extent that all objects of interest can (have to be) named. For data structures only two- dimensional tables are allowed. (Nesting is not allowed.) Tables are indexed and indices may be composed of one or more managed objects. The number of rows in a table do not need to be known before hand. The indexing mechanism takes care of that. A management application generally finds out the indexes of the rows (which represent e.g. interfaces/routes) before hand using the powerful "get-next" protocol operation (the responder returns the lexicographically next element in the MIB to the originator) so that when it wants to monitor a particular resource it knows the name that needs to be used. There is no limit to the granularity. MIBs can be defined to any level of granularity. For example an MO can well be defined for the nth bit of the mth packet of a certain protocol that passed through the interface of index i. Individual objects can be accessed with ease in a table. Entire lists and tables cannot be accessed, essentially the access is for each elementary (leaf) MO. However there are efficient mechanisms (SNMPget bulk) for fetching the MOs in a table or a subtree. The naming scheme requires that both managers and agents know the names. MIBs are published for that purpose. There are mechanisms to know all the object instances that may be resident in the MIB of a device, i.e. for discovering names/objects in a MIB (SNMPget next). This allows the manager to know all the manageable resources that are present at the other end. Mansfield/Curry Expires: July 19, 2000 [Page 11] Internet-Draft IDMEF SMI/XML Comparison January 20, 2000 4.1.1.2 XML The XML IDMEF DTD specifies that each semantic element of an alert will be individually tagged or contained in an attribute. As shown in the example in the previous section, tags within an alert are organized in a more or less hierarchical structure. There are no limits to the depth or breadth of the hierarchy (other than those imposed by the IDMEF data model itself). The Document Object Model (DOM), defined by the W3C, is a platform- and language-neutral interface that allows programs and scripts to dynamically access and update the content, structure, and style of XML documents. Through the DOM, an application that processes XML IDMEF messages can treat an XML document (IDMEF alert) as a "database," and extract individual elements from it based on tag or attribute name. The DOM allows on-the-fly manipulation of individual XML IDMEF messages by an application. The DOM does not, however, replace relational or object-oriented database management systems. An application that has a need to process historical alert data (e.g., for correlation or analysis purposes) would presumably use an RDBMS or OODBMS for this purpose. The XML IDMEF DTD's requirement that each semantic element of an alert be tagged or contained in an attribute, and the forthcoming XML Schema standard will make the conversion between the XML IDMEF message format and the database record format simple. 4.1.2 Data model Debar, Huang, and Donahoo [6] have proposed that intrusion detection events in the IDMEF be represented by a class hierarchy. The working group has adopted their model as the one that should be followed by any data format implementation. In this section, we examine how well the implementations match the data model, and any significant differences between the implementation and the model. 4.1.2.1 SMI The Alert-MIB fits the data model neatly. Moreover, the messages themselves are indexed by a unique message-id. The manager uses this message-id to obtain further information about the event that caused the message e.g., the destinations that were targeted by the attack and/or the packet-trace that contained the signature of the attack. [Note - there are deviations from the details given in data model document particularly related to data structures and data representations. These details are under discussion in the WG.] Mansfield/Curry Expires: July 19, 2000 [Page 12] Internet-Draft IDMEF SMI/XML Comparison January 20, 2000 4.1.2.2 XML The XML DTD is based on the IDMEF data model, but does not follow it exactly. The data model was completely rewritten after the IDWG meeting in November, 1999. It is therefore a first pass, and its authors have not had the opportunity to address the first round of comments. Accordingly, the XML DTD diverges from the data model in places where its author disagrees with the data model's approach. The DTD also extends the data model in several areas. The significant differences between the IDMEF data model and the XML DTD are: - The DTD does not support the "priority" and "impact" attributes of an alert; the author believes they overlap with the "severity" attribute and are therefore unnecessary. The DTD also does not support the "success" attribute, as the author sees no way for an analyzer to generate a meaningful value for it. - The DTD does not support the "confidence" attribute of sources and targets; the author believes it is too complex to spread this out into multiple confidence ratings. It also does not support the "spoofed" attribute, since there is no way to accurately determine this (except in trivial cases). - In addition to the alert generation time (required by the data model), the DTD provides the event detection time (needed for correlation) and the sender current time (needed to perform clock synchronization). - The DTD adds a "Portlist" type to the "Service" description in the data model, to support providing comma-separated lists of ports and hyphenated ranges. - The DTD dispenses with the "Analyzer" part of the data model that tries to associate analyzers and data sources, and replaces it with a more general "Sender." Senders are identified by "Host" and/or "Process" data, and they may be daisy-chained in the alert to show the entire forwarding path taken by the alert. - The DTD merges the "AdditionalData" portion of the "ExtendedAlert" in to the "Alert" itself, and dispenses with "ExtendedAlert." The author believes this is more generally useful, allowing extra data to be provided for any kind of alert, instead of having two similar but separate alert hierarchies. - The DTD provides a "Heartbeat" type of message for analyzers (and other components) to use to indicate their state of well-being to a manager. - The DTD provides "Query" and "Response" messages for managers and analyzers to use to exchange data about sources, targets, hosts, Mansfield/Curry Expires: July 19, 2000 [Page 13] Internet-Draft IDMEF SMI/XML Comparison January 20, 2000 users, processes, services, and networks. The "Query" allows a manager to ask an analyzer to fill in the data about something it gave only a reference id for, and the "Response" allows the analyzer to provide an answer to the query. Aside from the above, the XML DTD is very close to the data model. It can be easily modified to bring it into closer "compliance" if that is the will of the group; none of these changes has any major impact on the overall design of the format. 4.1.3 Data format What data representation format or formats are used by the implemen- tation? For example, is everything "ASCII text" or "binary"? In addition to impacting ease-of-use, data format may affect the overall performance of an implementation. 4.1.3.1 SMI The syntax of the managed objects are defined in the MIB - the data types used are INTEGER, OCTETSTRING, NULL and OBJECTIDENTIFIER. The encoding of the data for communication purposes is according to the Basic Encoding Rules (BER) associated with ASN.1. 4.1.3.2 XML The XML standard specifies that all XML documents (and therefore all XML IDMEF messages) be encoded in either the UTF-8 or UTF-16 encodings of ISO 10646 (Unicode). All data in an XML IDMEF message will be encoded as text; there will be no "binary" content. Numbers will be encoded as their formatted output equivalents (e.g., the number 123.45 would be represented by the six characters, '1,' '2,' '3,' '.,' '4,' and '5'). XML is capable of representing non-printable "binary" data, although the representation is not very efficient. Any arbitrary value can be encoded as decimal one-byte quantites (e.g., "<") or hexadecimal two-byte quantities (e.g., "<"). 4.2 Operational issues Operational issues are those factors of system and network operation that will be impacted by use of the implementation: network bandwidth consumed, memory and processor resources used, etc. 4.2.1 Bits on the wire Mansfield/Curry Expires: July 19, 2000 [Page 14] Internet-Draft IDMEF SMI/XML Comparison January 20, 2000 Intrusion detection systems are capable of sending many, many alerts in a very short period of time. Each of these alerts consumes some bandwidth on the network; if the number of alerts is too high, the network could become swamped, impacting not only the delivery of alerts, but all other applications using the network as well. 4.2.1.1 SMI First lets look at the number of bytes that goes on the wire when we do a get for say the number of bytes that came in (ifInOctets) through an interface (Index = 2). The "get" PDU containing the variable name iso.org.dod.internet.mgmt.mib-2.interfaces.ifTable. ifEntry.ifOperatstus.1 and a null binding will contain the UDP protocol header [8 bytes] Application protocol header [31 bytes (6 byte community string)] Payload: Variable Bindings [22 bytes] --------------- TOTAL 61 bytes (+ IP Header). Its response would be the get-response PDU containing the variable name and an integer (64bit counter) binding would contain UDP protocol header [8 bytes] Application protocol header [31 bytes (6 byte community string)] Payload: Variable Bindings [26 bytes] --------------- TOTAL 65 bytes (+ IP Header). In case we want to do this with TCP then we will have to add the 12 bytes of excess length the TCP header has. The point to note is that the average length of an MO name or an MOID is 10 bytes. When it is BER encoded and sent down the wire it is roughly 15 bytes. For a complete alert containing the following information: Alert.version = 1 Alert.priority = 1 Alert.confidence = 100 Alert.severity = 100 Alert.name = bugtraqid.33 Alert.signature = loadmodule forking shell Alert.method = 1 Alert.time.date = 1999/10/21 Alert.time.time = 08:12:32 Alert.analyzer.ident = 12345678 Alert.target[0].host.name = machine.domain.com Alert.target[0].host.address.type = 11 Alert.target[0].host.address.value = 123.234.345.456 Mansfield/Curry Expires: July 19, 2000 [Page 15] Internet-Draft IDMEF SMI/XML Comparison January 20, 2000 Alert.source[0].process.name = /usr/openwin/bin/loadmodule Alert.source[0].user.name = joe Alert.source[0].user.uid = 13243 The payload will be roughly: SNMP Protocol Header (v1) = 31 bytes + [ 15(MOIDs) * 15 (bytes/MOID) ] = 225 bytes + [ 10 (Integers) * 6 (bytes/Integer) ] = 60 bytes + [ 5 strings Length ={12,24,18,27} ] = 89 bytes 2 bytes overhead per string ----------------------------------------------------------- Total = 405 bytes The SNMP application level protocol does not offer any means of compressibility. However if the transport offers compressibility this will certainly reduce the number of bits on the wire. But there is probably no straightforward way of saying of whether the SNMP payload is "more compressible" than the HTTP payload 4.2.1.2 XML The XML IDMEF encoding of a complete alert containing the following information: Alert.version = 1 Alert.priority = 1 Alert.confidence = 100 Alert.severity = 100 Alert.name = bugtraqid.33 Alert.signature = loadmodule forking shell Alert.method = 1 Alert.time.date = 1999/10/21 Alert.time.time = 08:12:32 Alert.analyzer.ident = 12345678 Alert.target[0].host.name = machine.domain.com Alert.target[0].host.address.type = 11 Alert.target[0].host.address.value = 123.234.345.456 Alert.source[0].process.name = /usr/openwin/bin/loadmodule Alert.source[0].user.name = joe Alert.source[0].user.uid = 13243 is shown in Section 2.2. When formatted as it would be sent over an XML IDMEF message channel (no newlines or indentation), the encoding of this alert required 657 bytes. XML, because of the open-tag/close-tag syntax, has a relatively high overhead percentage. For the example above, XML tagging makes up about 66% of the total message. However, XML is also readily compressed. Using Lempel-Ziv coding, Mansfield/Curry Expires: July 19, 2000 [Page 16] Internet-Draft IDMEF SMI/XML Comparison January 20, 2000 the example above compresses to 407 bytes, a savings of 39%. By sending multiple alerts in the same message, compression results can be improved still further; savings of 80-90% are easily achieved. 4.2.2 Load on the CPU Encoding data on the analyzer for transmission to the manager will put additional load on the analyzer's processor. Likewise, decoding and parsing the data on the manager will put additional load on the manager's processor. Depending on the processing resources needed, this additional load may impact the ability of the analyzer/manager to perform its other tasks. 4.2.2.1 SMI There is an associated load with encoding data in the SNMP PDUs at the analyzer and then in decoding SNMP PDUs at the manager. The encoding is done in BER. However, the load is probably unavoidable if the data transferred is to be used for computational purposes. For example, the integer 389360048 is represented as 41 04 17 35 29 B0 Where the first byte (41) represents the datatype, the second byte (04) represents the length in bytes of the following contents, the remaining 4 bytes (17 35 29 B0) represent the contents the hexadecimal representation of 389360048. Incidentally if a plain ASCII text representation was used, the string would just be represented by 63 68 69 63 66 60 60 64 68 4.2.2.2 XML The load placed on an analyzer to generate XML IDMEF messages is minimal. The message itself is simply character string data, usually generated with a formatted output function such as printf() from C/C++. Only basic control structures are needed to create the format of the message (e.g., "if value known then print else don't"). The load placed on a manager to process XML IDMEF messages is somewhat higher, since the manager must parse (and optionally validate) the XML document that represents an IDMEF message. The following times to parse an "average" XML IDMEF message were measured on a 360 MHz UltraSPARC II with 128MB of memory: Mansfield/Curry Expires: July 19, 2000 [Page 17] Internet-Draft IDMEF SMI/XML Comparison January 20, 2000 Package Language Validation Time -------------------------------------------------------------------- IBM XML4J 3.0 SAXCount Java On 47-52 ms IBM XML4J 3.0 SAXCount Java Off 75-80 ms IBM XML4C 2.3.1 SAXCount C++ On 30-33 ms IBM XML4C 2.3.1 SAXCount C++ Off 27-29 ms Timings for other processors, other parser implementations, and other IDMEF messages will of course vary. 4.3 Implementation issues Implementation issues are those factors of the implementations that will impact vendors and authors of intrusion detection systems - i.e., how easy will it be to add support for IDMEF to existing and new intrusion detection systems? 4.3.1 Size of code Adding support for IDMEF will require adding code to both the analyzer and the manager. 4.3.1.1 SMI The minimum functionality that analyzers should have are - send alerts - receive (subsequent) queries - reply to the queries SNMP-agents can be very slim. They are meant to be slim and lightweight. A simple agent program with no MIB support would be 1.4MB. With MIBII support (i.e. with instrumentation for the MIBII objects - which means the manager can it questions about the number of TCP connections that are open and expect a reply) the agent size will be 2.5MB. An SNMP agent that just receives alerts is 1.0MB. The code required to send queries, receive responses, process/parse the responses and display them on the console in the (comparatively ugly but called "pretty print") form like interfaces.ifTable.ifEntry. ifInOctets.1 = 389360048 is roughly 1.0MB. Mansfield/Curry Expires: July 19, 2000 [Page 18] Internet-Draft IDMEF SMI/XML Comparison January 20, 2000 4.3.1.2 XML Code size on the analyzer to generate XML IDMEF messages is so small as to be insignificant. 1-2 KB of string storage space (for the tag and attribute names), and a few hundred bytes of control structures. Code size on the manager to parse the XML IDMEF message is also not large. The binary object or script file sizes for several freely available XML parsers is shown below: Package Language Validating Program Size -------------------------------------------------------------------- Expat 1.1 C No 128 KB IBM XML4J 3.0 Java Yes 865 KB (includes multiple parsers and document object model support) IBM XML4C 2.3.1 C++ Yes 20 KB TclXML 1.2 Tcl No 58 KB xmlproc 0.61 Python Yes 156 KB XP 0.5 Java No 166 KB The manager will also need space for the Java, Tcl, or Python runtime environments, if those are used, plus memory for the parser's data structures. 4.3.2 Availability of code Availability of code addresses how easy it is for a vendor or author of intrusion detection systems to obtain code to implement the IDMEF. 4.3.2.1 SMI The code is widely available and freely over a large range of platforms. The Perl version offers a complete and compact package which is of relatively small size (52K in tar gzipped form) but of course it will require the full Perl interpreter for its execution. Nevertheless the Perl package has made the package available over a very large range of platforms. 4.3.2.2 XML Mansfield/Curry Expires: July 19, 2000 [Page 19] Internet-Draft IDMEF SMI/XML Comparison January 20, 2000 Software tools for processing XML documents are widely available, in both commercial and open source forms. A variety of tools and APIs for parsing and/or validating XML are available in Java, C, C++, Tcl, Perl, Python, and GNU Emacs Lisp. For a comprehensive list of both commercial and open source XML tools, see http://www.oasis-open.org/cover/publicSW.html#xmlTools 4.4 End point issues End point issues are those factors of the implementations that impact how alert data will be handled once it reaches the manager. 4.4.1 Data display aspects Data display aspects are those features of the data format that impact how data can be displayed to users, on graphical displays as well as in printed documents. 4.4.1.1 SMI The MIB representation does not aid or hinder display at the end point. There are a couple of MIB browsers (Perl, TCL. JAVA, ... based), but none (that I know of) directly interface with a web browser. The SMI->HTML translation, though not an impossibility, has not been attempted (not to my knowledge). 4.4.1.2 XML One of the principal advantages of semantic tagging, such as that used by XML, is that the same document can be used for a variety of applications without having to translate it to another format. The Extensible Stylesheet Language (XSL), defined by the W3C, is a language for expressing stylesheets. Given a class of structured documents or data files in XML, designers use an XSL stylesheet to express their intentions about how that structured content should be presented; that is, how the source content should be styled, laid out and paginated onto some presentation medium such as a window in a Web browser or a set of physical pages in a book, report, pamphlet, or memo. To display an XML IDMEF message on a graphical display, all that is needed is a viewing program (such as a web browser) that supports XML, and a style sheet that tells the program how to display the content of the various tags. The Microsoft Internet Explorer and Mansfield/Curry Expires: July 19, 2000 [Page 20] Internet-Draft IDMEF SMI/XML Comparison January 20, 2000 Mozilla (not Netscape) web browsers have support for XML and XSL. There are also several free and commercial XML browsing tools available. To display an XML IDMEF message on the printed page, all that is needed is a formatting program that supports XML, and a style sheet (different from the browser style sheet) that tells the program how to print the content of the various tags. 4.4.2 Data transfer aspects Data transfer aspects are those features of the data format that impact how efficiently the implementation can transfer data. 4.4.2.1 SMI SMI-MIBs are not good for representing bulk data (and SNMP is not good for transferring bulk data). 4.4.2.2 XML The amount of data to be transferred is not a problem for XML, since the DTD just defines tags that identify that markup -- everything is treated as a simple stream of bytes. In situations where most of the data elements are small, however, XML may impose a lot of overhead, resulting in messages that require more bytes to represent the data tags than to represent the data itself. This overhead can be partially offset by XML's easy compressibility, but only if compression is available. XML is primarily intended to represent "printable" data (UTF-8 or UTF-16). Although it is capable of representing arbitrary "binary" data, its method for doing so is both cumbersome and inefficient, and should be avoided if at all possible. 4.5 Deployment issues Deployment issues address how easy it will be to actually "get IDMEF out there" once it has been standardized (and adopted by vendors). These issues affect existing deployed systems (which may have to be upgraded or replaced), and existing products (which may have to be modified). 4.5.1 SMI Most network devices have an SNMP agent in them, and thus the code to generate and handle SMI compliant (SNMP) messages is already there. Mansfield/Curry Expires: July 19, 2000 [Page 21] Internet-Draft IDMEF SMI/XML Comparison January 20, 2000 4.5.2 XML To the authors' knowledge, XML is not currently supported by any existing intrusion detection products, commercial or otherwise. However, some existing products already make use of the Java runtime environment to implement their management console functionality; this may make the integration of a Java-based XML parser somewhat easier than starting from scratch. 4.6 Transport issues Transport issues are those factors of the implementations that impact how IDMEF messages can be transmitted via the network. 4.6.1 TCP/UDP Can TCP-based protocols, UDP-based protocols, or both be used? 4.6.1.1 SMI SNMP goes over UDP. This is looked upon as a major plus as when the network is in trouble the UDP packets have a better chance of getting through with least additional damage. Probably ID falls in the same category as far as network troubles are concerned. The negative aspects of UDP is that it is unreliable. However by using Confirmed alerts (Informs) in which the receiver acknowledges the receipt can be (is) used to get around this problem. SNMP may also be sent over TCP Experimental versions are out. 4.6.1.2 XML XML IDMEF messages, since they are simply a stream of bytes, can be sent over TCP without problems. Indeed, a virtual circuit is the preferred transport method. XML IDMEF messages can, in general, be sent over UDP too. However, it is possible that messages containing long lists of hosts or services could exceed the UDP maximum packet size, making it a less desirable transport mechanism. 4.6.2 Intrusion Alert Protocol (IAP) Mansfield/Curry Expires: July 19, 2000 [Page 22] Internet-Draft IDMEF SMI/XML Comparison January 20, 2000 The Intrusion Alert Protocol (IAP) has been selected by the working group as the protocol to be used for sending and receiving IDMEF alerts. IAP is an HTTP-like protocol over TCP that uses the Transport Layer Security protocol (a superset of the Secure Sockets Layer, SSL) for security and authentication. 4.6.2.1 SMI SNMP PDUs will have no problem being transported over IAP. 4.6.2.2 XML As XML is simply a stream of bytes, it can be transported over the IAP without problems. 5. Security Considerations This Internet-Draft compares two data formats that have been proposed for the exchange of security-related data between security product implementations. There are no security considerations directly applicable to the format of this data. There may, however, be security considerations associated with the transport protocol chosen to move this data between communicating entities. 6. References [1] Bradner, S., "The Internet Standards Process -- Revision 3," BCP 9, RFC 2026, October, 1996. [2] Bradner, S., "Key words for use in RFCs to Indicate Requirement Levels," BCP 14, RFC 2119, March, 1997. [3] Wood, M., "Intrusion Detection Message Exchange Requirements," draft-ietf-idwg-requirements-02.txt, October 21, 1999, work in progress. [4] Mansfield, G. and D. Gupta, "Intrusion Detection Message MIB," draft-glenn-id-notification-mib-01.txt, October 23, 1999, work in progress. [5] Curry, D, "Intrusion Detection Message Exchange Format Extensible Markup Language (XML) Implementation," draft-curry-idef-xml-01.txt, January 2000, work in progress. [6] Debar, H., M. Huang, and D. Donahoo, "Intrusion Detection Exchange Format Data Model," draft-ietf-idwg-data-model-01.txt, December 15, 1999, work in progress. Mansfield/Curry Expires: July 19, 2000 [Page 23] Internet-Draft IDMEF SMI/XML Comparison January 20, 2000 [7] Case, J., R. Mundy, D. Partain, and B. Stewart, "Introduction to Version 3 of the Internet-standard Network Management Framework," RFC 2570, April, 1999. [8] Gupta, D., "IAP: Intrusion Alert Protocol," draft-ietf-idwg-iap-00.txt, September 15, 1999, work in progress. 7. Authors' Addresses Glenn Mansfield Cyber Solutions, Inc. 6-6-3 Minami Yoshinari Aoba-ku, Sendai 989-3204 Japan Phone: +81 22-303-4012 Email: glenn@cysols.com David A. Curry Internet Security Systems Emergency Response Service 148 Madison Avenue, 11th Floor New York, NY 10016 USA Phone: +1 212 592-5525 Email: davy@iss.net Full Copyright Statement "Copyright (C) The Internet Society (date). All Rights Reserved. This document and translations of it may be copied and furnished to others, and derivative works that comment on or otherwise explain it or assist in its implementation may be prepared, copied, published and distributed, in whole or in part, without restriction of any kind, provided that the above copyright notice and this paragraph are included on all such copies and derivative works. However, this document itself may not be modified in any way, such as by removing the copyright notice or references to the Internet Society or other Internet organizations, except as needed for the purpose of developing Internet standards in which case the procedures for copyrights defined in the Internet Standards process must be followed, or as required to translate it into languages other than English. The limited permissions granted above are perpetual and will not be revoked by the Internet Society or its successors or assigns. This document and the information contained herein is provided on an "AS IS" basis and THE INTERNET SOCIETY AND THE INTERNET ENGINEERING TASK FORCE DISCLAIM ALL WARRANTIES, EXPRESS OR IMPLIED, INCLUDING BUT Mansfield/Curry Expires: July 19, 2000 [Page 24] Internet-Draft IDMEF SMI/XML Comparison January 20, 2000 NOT LIMITED TO ANY WARRANTY THAT THE USE OF THE INFORMATION HEREIN WILL NOT INFRINGE ANY RIGHTS OR ANY IMPLIED WARRANTIES OF MERCHANTIBILITY OR FITNESS FOR A PARTICULAR PURPOSE. Mansfield/Curry Expires: July 19, 2000 [Page 25]