Internet Engineering Task Force syslog Internet Draft: Informational Chris Lonvick draft-ietf-syslog-syslog-00.txt Cisco Systems October 04, 2000 Expires: April, 2001 syslog Protocol draft-ietf-syslog-syslog-00.txt STATUS OF THIS MEMO This document is an Internet-Draft and is in full conformance with all provisions of Section 10 of RFC 2026. Internet-Drafts are working documents of the Internet Engineering Task Force (IETF), its areas, and its working groups. Note that other groups may also distribute working documents as Internet-Drafts. Internet-Drafts are draft documents valid for a maximum of six months and may be updated, replaced, or obsoleted by other documents at any time. It is inappropriate to use Internet-Drafts as reference material or to cite them other than as "work in progress". The list of current Internet-Drafts can be accessed at http://www.ietf.org/ietf/1id-abstracts.txt The list of Internet-Draft Shadow Directories can be accessed at http://www.ietf.org/shadow.html. This work is a product of the IETF syslog Working Group. More information about this effort may be found at http://www.ietf.org/html.charters/syslog-charter.html Comments about this draft should be directed to the syslog working group at the mailing list of syslog-sec@employees.org. When written in uppercase, The key words "MUST", "MUST NOT", "REQUIRED", "SHALL", "SHALL NOT", "SHOULD", "SHOULD NOT", "RECOMMENDED", "MAY", and "OPTIONAL" in this document are to be interpreted as described in RFC 2119. Expires April 2001 [Page 1] Draft syslog October 2000 1 Introduction Since the beginning, life has relied upon the transmission of messages. For the self-aware organic unit, these messages can relay many different things. The messages may signal danger, the presence of food or the other necessities of life, and many other things. In many cases, these messages are informative to other units and require no acknowledgement. As people created processes and machines, this same principal was applied to societal communications. As an example, severe weather warnings may be delivered through any number of channels - a siren blowing, warnings delivered over television and radio stations, and even through the use of flags on ships. The expectation is that people hearing or seeing these warnings would realize their significance and take appropriate action. In most cases, no responding acknowledgement of receipt of the warning is required or even desired. Along these same lines, operating systems, processes and applications were written to send messages of their own status, or messages to indicate that certain events had occurred. These event messages generally had local significance to the machine operators. As the operating systems, processes and applications grew ever more complex, systems were devised to categorize and log these diverse messages and allow the operations staff to more quickly differentiate the notifications of problems from simple status messages. The syslog process was one such system that has been widely accepted in many operating systems. Flexibility was designed into this process so the operations staff have the ability to configure the destination of messages sent from the processes running on the device. In one dimension, the events that were received by the syslog process could be logged to different files and also displayed on the console of the device. In another dimension, the syslog process could be configured to forward the messages across a network to the syslog process on another machine. The syslog process had to be built network-aware for some modicum of scalability since it was known that the operators of multiple systems would not have the time to access each system to review the messages logged there. The syslog process running on the remote devices could therefore be configured to either forward the message to a file, or to subsequently forward it to another machine. In its most simplistic terms, the syslog protocol is an event notification protocol that allows a machine to send event notification messages across IP networks to event message collectors -also known as syslog servers. Since each process, application and operating system was written somewhat independently, there is little uniformity to syslog messages. For this reason, no assumption is made upon the contents of the messages other than the minimum Expires April 2001 [Page 2] Draft syslog October 2000 requirements of its priority. The protocol is simply designed to transport these event messages. In all cases, there is one device that originates the message. The syslog process on that machine may send the message to a collector. No acknowledgement of the receipt is made. 1.1 Events and Generated Messages The writers of the operating systems, processes and applications have had total control over the circumstances that would generate any message. In some cases, messages are generated to give status. These can be either of a certain period of time, or at some other interval such as the invocation or exit of a program. In other cases, the messages may be generated due to a set of conditions being met. In that case, either a status message or a message containing an alarm of some type may be generated. The contents of a message have also been at the discretion of its creator. It has been considered to be good form to write the messages so that they are informative to the person who may be reading them. It has also been considered good practice to include a timestamp in the messages. However, neither of those is required. It should be assumed that any process on any device might generate an event message. This may include processes on machines that do not have any local storage - e.g. printers, routers, hubs, switches, and diskless workstations. In that case, it may be imperative that event messages are transported to a collector so that they may be recorded and hopefully viewed by an operator. 1.2 Operations of the Message Collector It is beyond the scope of this Internet Draft to specify how event messages should be processed. It was considered that the writers of the operating systems, processes and applications would quantify their messages into one of several broad categories. This was so that the operations staff could be presented with the more important and time sensitive messages quickly, while also having the ability to place status or informative messages in a file for later perusal. Expires April 2001 [Page 3] Draft syslog October 2000 2 Transport Layer Protocol syslog uses the user datagram protocol (UDP) [1] as its underlying transport layer mechanism. The udp port that has been assigned to syslog is 514. No restrictions are placed upon the source port of each message however, it is RECOMMENDED and has been considered good form that subsequent messages are from a single consistent port. 3 Packet Format and Contents The syslog message MUST be transmitted as ASCII alphanumerics and symbols. The code set is seven-bit ASCII in an eight-bit field. These are the ASCII codes as defined in "USA Standard Code for Information Interchange" [2] only using codes 32 through 126. The message starts with a leading "<" ('less-than' character), followed by a number, which is followed by a ">" ('greater-than' character). This is optionally followed by a single ASCII space. The remainder of the message follows this. There is no ending delimiter but the message length MUST be 1024 bytes or less, excluding the UDP overhead. The number described above is known as the Priority and represents both the Facility and Severity as described below. The Priority number consists of one, two, or three decimal integers. As examples, these are valid messages as they may be observed on the wire between two devices. Each message starts with the less-than character but has been indented, with line breaks inserted for readability. <37> Oct 11 16:00:15 mymachine su: 'su root' failed for lonvick on /dev/pts/8 <14>Use the BFG! <160> Aug 24 1987 03:24:00 AM CST mymachine.&.process_manager %% It's time to make the do-nuts. %% Ingrediants: Mix=OK, Jelly=OK # Devices: Mixer=OK, Jelly_Injector=OK, Frier=OK # Transport: Conveyer1=OK, Conveyer2=OK # %% <0> Oct 22 1990 08:22:59 That's All Folks! Expires April 2001 [Page 4] Draft syslog October 2000 3.1 The Priority Value: Facility and Severity The Facilities and Severities of the messages are numerically coded with decimal values. The operating system and some of the daemons and processes have been assigned a Facility parameter. Processes and applications that have not been assigned a Facility, or that have not been configured to use one of the "local use" Facilities SHOULD use the "user" Facility which has the numerical Facility code of decimal 8. All Facilities are shown in the following table along with their numerical code values. Numerical Facility Code 0 kernel messages 8 user-level messages 16 mail system 24 system daemons 32 security/authorization messages 40 messages generated internally by syslogd 48 line printer subsystem 56 network news subsystem 64 UUCP subsystem 72 clock daemon 80 security/authorization messages 88 ftp daemon 96 Reserved 104 Reserved 112 Reserved 120 Reserved 128 local use 0 (local0) 136 local use 1 (local1) 144 local use 2 (local2) 152 local use 3 (local3) 160 local use 4 (local4) 168 local use 5 (local5) 176 local use 6 (local6) 184 local use 7 (local7) Table 1. syslog Message Facilities Expires April 2001 [Page 5] Draft syslog October 2000 Each message Priority also has a decimal Severity level indicator. These are described in the following table along with their numerical values. Numerical Severity Code 0 Emergency: system is unusable 1 Alert: action must be taken immediately 2 Critical: critical conditions 3 Error: error conditions 4 Warning: warning conditions 5 Notice: normal but significant condition 6 Informational: informational messages 7 Debug: debug-level messages Table 2. syslog Message Severities The Priority code is calculated by summing the numerical values of the codes of the Facility and Severity. For example, A kernel message with a Severity of Emergency would have a Priority value of decimal 0, while a "local use 4" message with a Severity of Notice would have a Priority value of decimal 165. 3.2 Message Conveyance When an event message is forwarded by one message collector to another, the entire contents of the message MUST be kept intact. Additionally, the Priority level of the original message MUST be kept with the message. The retransmitting device MUST NOT modify any portion of the message. When the retransmitted message is received by the next collector, it must have the appearance of having been sent from the originating device. 4 Conventions Although Section 3 of this document specifies all requirements for the syslog protocol format and contents, certain conventions have come about over time for the inclusion of additional information within the syslog message. It must be plainly stated that these datum are not mandated but are usually included for completeness and to give the recipient some clue of their origin and nature. Expires April 2001 [Page 6] Draft syslog October 2000 It is now prevalent to include a time and date stamp within the message. It is also considered to be good practice to include some information about the process on the device that generated the message - such as the process name and process id - if that concept exists on the device. To readily identify the device that originated the message, it is a good idea to include the fully qualified domain name and/or its IP address. 5 Security Considerations An odor may be considered to be a message that does not require any acknowledgement. People tend to avoid bad odors but are drawn to odors that they associate with good food. The acknowledgement of the receipt of the odor or scent is not required and indeed it may be the height of discretion to totally ignore some odors. On the other hand, it is usually considered good civility to acknowledge the prowess of the cook merely from the ambiance wafting from the kitchen. Similarly, various species have been found to utilize odors to attract mates. One species of moths use this scent to find each other. However, it has been found that bolas spiders can mimic the odor of the female moths of this species. This scent will then attract male moths which will follow it with the expectation of finding a mate. Instead, when they arrive at the source of the scent, they will be eaten. [3] This is a case of a false message being sent out with inimical intent. In its local use, the syslog process places event notification messages into files on that system. This relies upon the integrity of the system for the protection of the messages. The subsequent configuration of the syslog process to use the syslog protocol to transport the messages to another collector was an extension of the delivery of event notification messages and exhibits the same trust of the network. As such there are some concerns about the applicability of this protocol in situations that require robust delivery. Along the lines of the analogy, computer event messages may be sent accidentally, erroneously and even maliciously. At the time of this writing, however, there have not been any reports of any machine consuming any other machine. Expires April 2001 [Page 7] Draft syslog October 2000 5.1 Packet Parameters As was described above, the message length MUST NOT exceed 1024 bytes. Attacks have seen where syslog messages are sent to a receiver that have message lengths greater than 1024 bytes. In some older versions of syslog, the receipt of syslog packets that had a message greater than 1024 bytes caused problems. syslog message receivers must not malfunction upon the receipt of packets where the message length is greater than 1024 bytes. If a message receiver does receive a message whose length is greater than 1024 bytes, it may log all of, or the source address and some of the contents of the message, or it may discard the message altogether. Devices MUST NOT retransmit messages whose received length exceeds 1024 bytes. Similarly, the receiver must rigidly enforce the correctness of the message body. syslog collectors must not malfunction if received messages do not have the less-than and greater-than characters around a valid Priority value. The receiver may locally log some or all of a received invalid message or they may discard it totally. Also, received messages must contain ASCII text in the message. Devices must not malfunction if they receive a message containing characters other than the ASCII characters described above. 5.2 Message authenticity The syslog delivery mechanism does not strongly associate the message with the message sender. Any device can generate any syslog message and send it to any other machine through the syslog protocol. The receiver of that packet will not be able to ascertain that the message was indeed sent from the reported sender, or if the packet was sent from another device. One possible consequence of this is that a misconfigured machine may send syslog messages to a collector representing itself as another machine. The administrative staff may become confused that the status of the supposed sender of the messages may not be accurately reflected in the received messages. The administrators may not be able to readily discern that there are two or more machines representing themselves as the same machine. Another consequence happens when the event messages are forwarded. Unless the identification of the device is contained within the body of the event message, the source of the message may be lost since it may only be self-identified by its IP address contained in the IP header. It should be noted that some cases of embedding the identity of a device may only have local significance and that may only be ephemeral. The inclusion of a fully qualified domain name in the message may give the administrators the best chance of identifying Expires April 2001 [Page 8] Draft syslog October 2000 the source of each message if it can always be associated with an IP address. However, if the device had obtained an IP address from a DHCP pool, then that association would not always hold true. Malicious exploits of this vulnerability have also been noted. An attacker may transmit syslog messages (either from the machine that the messages purport from which to be sent or from any other machine) to a collector. The attacks that have been noted run along these lines: - An attacker may perform a Denial of Service attack by filling the disk of the collector with false messages, or otherwise overwhelming the collector by sending more messages than it can receive or process. - An attacker may hide the true nature of an attack amidst many other messages. As an example, an attacker may start generating messages indicating a problem on some machine. This may get the attention of the system administrators who will spend their time investigating the alleged problem. During this time, the attacker may be able to compromise a different machine, or a different process on the same machine. - An attacker may generate false syslog messages to give untrue indications of status or of events. As an example, an attacker may stop a critical process on a machine, which may generate a notification of exit. The attacker may subsequently generate a false notification that the process had been restarted from another machine already under the control of the attacker. The system administrators may accept that misinformation and not verify that the process had indeed been restarted. 5.3 Sequenced delivery As a general rule, the forensics of a network anomaly rely upon reconstructing the sequence of events. In a perfect world, the messages would be received on the syslog collector in their order of generation from the other devices and anyone looking at these records would have an accurate picture of the sequence of events. Unfortunately, the syslog process and protocol do not ensure ordered delivery. This section details some of the problems that may be encountered from this. Expires April 2001 [Page 9] Draft syslog October 2000 5.3.1 Single Source to a Destination The syslog records are usually presented (placed in a file, displayed on the console, etc.) in the order in which they are received. This is not always in accordance with the sequence in which they were generated. As they are transported across an IP network, some out of order receipt should be expected. This may lead to some confusion as messages may be received that would indicate that a process has stopped before it was started. This may be somewhat rectified if the originating process had timestamped or numbered each of the messages before transmission. To be as effective as possible, both the source of the message and the syslog collector should both timestamp the messages. In this, both the sending device as well as the collector should utilize the same authoritative time source. It should be remembered, however, that not all devices are capable of receiving time updates, and not all processes timestamp their messages. 5.3.2 Multiple Sources to a Destination In syslog, there is no concept of unified event numbering. Single devices are free to include a sequence number within the event message but that can hardly be coordinated between multiple devices. In such cases, multiple devices may report that they are each sending message number one. Again, this may be rectified somewhat if the sending devices utilize a timestamp from an authoritative source in their messages. As has been noted, however, even messages from a single device to a single collector may be received out of order. This situation is compounded when there are several devices configured to send their syslog messages to a single collector. Messages from one device may be delayed so that messages from another device are received by the collector even though the messages from the first were generated before the messages from the second. If there is no timestamp or sequence number, then the messages may be presented in the order in which they were received which may give an inaccurate view of the sequence of actual events. Expires April 2001 [Page 10] Draft syslog October 2000 5.3.3 Multiple Sources to Multiple Destinations The plethora of configuration options available to the network administrators may further skew the perception of the order of events. It is possible to configure a group of devices to send the status messages -or other informative messages- to one collector, while sending messages of relatively higher importance to another collector. Additionally, the messages may be sent to different files on the same collector. If the messages do not contain timestamps from the source, it may be difficult to order the messages if they are kept in different places. An administrator may not be able to determine if a record in one file occurred before or after a record in a different file. This may be somewhat alleviated by placing marking messages with a timestamp into all destination files. If these have coordinated timestamps, then there will be some indication of the time of receipt of the individual messages. 5.3.4 Replaying Also, without any sequence indication or timestamp, messages may be recorded and replayed at a later time. An attacker may record a set of messages that indicate normal activity of a machine. At a later time, that attacker may remove that machine from the network and replay the syslog messages to the collector. The administrators would find nothing unusual in the received messages and their receipt would falsely indicate normal activity of the machine. 5.4 Reliable Delivery As there is no mechanism within either the syslog process or the protocol to ensure delivery, and since the underlying transport is UDP, some messages may be lost. They may either be dropped through network congestion, or they may be maliciously intercepted and discarded. The consequences of the drop of one or more syslog messages cannot be determined. If the messages are simple status updates, then their non-receipt may either not be noticed, or it may cause an annoyance for the system operators. On the other hand, if the messages are more critical, then the administrators may not become aware of a developing and potentially serious problem. Messages may also be intercepted and discarded by an attacker as a way to hide unauthorized activities. Expires April 2001 [Page 11] Draft syslog October 2000 5.5 Message Integrity Besides being discarded, syslog messages may be damaged in transit, or an attacker may maliciously modify them. In the case of a packet containing a syslog message being damaged, there are various mechanisms built into the link layer as well as into the IP [4] and UDP protocols which may detect the damage. A damaged IP packet may be discarded by an intermediary router [5]. Damage to a UDP packet may be detected by the receiving UDP module which may silently discard it. In any case, the original contents of the message will not be delivered to the collector. Additionally, if an attacker is positioned between the sender and collector of syslog messages, then he may be able to intercept and modify those messages in transit to hide unauthorized activities. 5.6 Message Observation While there are no strict guidelines pertaining to the event message format, most syslog messages are generated in human readable form with the assumption that capable administrators should be able to read them and understand their meaning. Neither the syslog protocol nor the syslog application has any mechanism to provide confidentiality of the messages in transit. In most cases passing the clear-text messages is a benefit to the operations staff if they are sniffing the packets off of the wire. The operations staff may be able to read the messages and associate them with other events seen from other packets crossing the wire to track down and correct problems. Unfortunately, an attacker may also be able to observe the human-readable contents of syslog messages. The attacker may then use the knowledge gained from those messages to compromise a machine or do other damage. 5.7 Message Prioritization and Differentiation While the processes that create the messages may signify the importance of the events through the use of the message Priority value, there is no distinct association between this value and the importance of delivery of the packet. As an example of this consider an application that generates two event messages. The first is a normal status message but the second could be an important message denoting a problem with the process. This second message would have an appropriately higher Severity value associated with the importance of that event. If the operators had configured that both of these messages be transported to a syslog collector then they would, in turn, be given to UDP for transmission. Under normal conditions, no distinction would be made between them and they would be transmitted in their order. Expires April 2001 [Page 12] Draft syslog October 2000 Again, under normal circumstances, the receiver would accept syslog messages as they are received. If many devices are transmitting normal status messages, but one is transmitting an important event message, there is no inherent mechanism within the syslog protocol to prioritize the important message over the other messages. On a case-by-case basis, device operators may find some way to associate the different levels with the quality of service identifiers. As an example, the operators may elect to define some linkage between syslog messages that have a specific Priority with a specific value to be used in the IPv4 Precedence field [2], the IPv6 Traffic Class octet [6], or the Differentiated Services field [7]. In the above example, the operators may have the ability to associate the status message with normal delivery while associating the message indicating a problem with a high reliability, low latency queue as it goes through the network. This would have the affect of prioritizing the essential messages before the normal status messages. Even with this hop-by-hop prioritization, this queuing mechanism could still lead to head of line blocking on the transmitting device as well as buffer starvation on the receiving device if there are many near-simultaneous messages being sent or received. In this same line, some implementations of the syslog application may have mechanisms for the prioritization of the more important messages within the transmission queue. This behavior is not unique to syslog but is endemic to all operations that transmit messages serially. There are security concerns for this behavior. Head of line blocking of the transmission of important event messages may relegate the conveyance of important messages behind less important messages. If the queue is cleared appropriately, this may only add seconds to the transmission of the important message. On the other hand, if the queue is not cleared, then important messages may not be transmitted. Also at the receiving side, if the syslog receiver is suffering from buffer starvation due to large numbers of messages being received near-simultaneously, important messages may be dropped indiscriminantly along with other messages. While these are problems with the devices and their capacities, the protocol security concern is that there is no prioritization of the relatively more important messages over the less important messages. Expires April 2001 [Page 13] Draft syslog October 2000 6 Conclusion The syslog protocol may be effectively used to transport event notification messages across a network. It is highly recommended that the network operators who choose to use this understand the characteristics of the protocol and its security implications. 7 Acknowledgements The following people provided content feedback during the writing of this draft: Jon Knight Magosanyi Arpad Balazs Scheidler Jon Callas Eliot Lear 8 Bibliography [1] Postel, J., "User Datagram Protocol", STD 6, RFC 768, USC/Information Sciences Institute, August 1980. [2] USA Standard Code for Information Interchange, USASI X3.4-1968. [3] Stowe, M., et al, "Chemical Mimicry: Bolas Spiders Emit Components of Moth Prey Species Sex Pheromones", Science, 1987 [4] Postel, J., "Internet Protocol", STD 5, RFC 791, USC/Information Sciences Institute, September 1981. [5] Baker, F., "Requirements for IP Version 4 Routers", RFC 1812, June 1995. [6] Deering, S. and R. Hinden, "Internet Protocol, Version 6 (IPv6) Specification", RFC 2460, December 1998. [7] Nichols, K., S. Blake, F. Baker, D. Black, "Definition of the Differentiated Services Field (DS Field) in the IPv4 and IPv6 Headers", RFC 2474, December 1998 Expires April 2001 [Page 14] Draft syslog October 2000 A Author's Address Chris Lonvick Cisco Systems 12515 Research Blvd. Austin, TX 78759 USA +1.512.378.1182 clonvick@cisco.com B Full Copyright Statement Copyright (C) The Internet Society (2000). All Rights Reserved. This document and translations of it may be copied and furnished to others, and derivative works that comment on or otherwise explain it or assist in its implementation may be prepared, copied, published and distributed, in whole or in part, without restriction of any kind, provided that the above copyright notice and this paragraph are included on all such copies and derivative works. However, this document itself may not be modified in any way, such as by removing the copyright notice or references to the Internet Society or other Internet organizations, except as needed for the purpose of developing Internet standards in which case the procedures for copyrights defined in the Internet Standards process must be followed, or as required to translate it into languages other than English. The limited permissions granted above are perpetual and will not be revoked by the Internet Society or its successors or assigns. This document and the information contained herein is provided on an "AS IS" basis and THE INTERNET SOCIETY AND THE INTERNET ENGINEERING TASK FORCE DISCLAIMS ALL WARRANTIES, EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO ANY WARRANTY THAT THE USE OF THE INFORMATION HEREIN WILL NOT INFRINGE ANY RIGHTS OR ANY IMPLIED WARRANTIES OF MERCHANTABILITY OR FITNESS FOR A PARTICULAR PURPOSE. Expires April 2001 [Page 15]