TICTOC D. Marlow, Internet Draft S. Knickerbocker, Intended status: Informational T. Plunkett, Expires: August 25, 2011 NSWC-DD February 25, 2011 Network Time Mechanisms for Improving Computer Clock Accuracy draft-marlow-tictoc-computer-clock-accuracy-00.txt Status of this Memo This Internet-Draft is submitted in full conformance with the provisions of BCP 78 and BCP 79. Internet-Drafts are working documents of the Internet Engineering Task Force (IETF), its areas, and its working groups. Note that other groups may also distribute working documents as Internet- Drafts. Internet-Drafts are draft documents valid for a maximum of six months and may be updated, replaced, or obsoleted by other documents at any time. It is inappropriate to use Internet-Drafts as reference material or to cite them other than as "work in progress." The list of current Internet-Drafts can be accessed at http://www.ietf.org/ietf/1id-abstracts.txt The list of Internet-Draft Shadow Directories can be accessed at http://www.ietf.org/shadow.html This Internet-Draft will expire on April 27, 2011. Copyright Notice Copyright (c) 2010 IETF Trust and the persons identified as the document authors. All rights reserved. This document is subject to BCP 78 and the IETF Trust's Legal Provisions Relating to IETF Documents (http://trustee.ietf.org/license-info) in effect on the date of publication of this document. Please review these documents carefully, as they describe your rights and restrictions with respect to this document. Marlow/Knickerbocker/Plunkett Expires August 25 2011 [Page 1] Internet-Draft Improving Computer Clock Accuracy February 2011 Abstract This draft describes network time synchronization mechanisms which may enable increased accuracy, beyond that possible with the current Network Time Protocol version 4 (NTPv4) standard, to time and frequency of computer clocks. The mechanisms considered are those that will provide improved estimates as to when a packet is put on the network, transferred across a network, or taken from the network. Potential standardization actions will be considered for the mechanisms considered, though no such actions are recommended at this time. Table of Contents 1. Introduction...................................................2 1.1. Motivation for Increased Performance......................3 1.2. NTP/PTP Commonality and Differences.......................4 2. Use Case Targeted..............................................4 2.1. Emerging Need for NTP and PTP Commonality.................5 3. Approach.......................................................5 4. Mechanisms Considered..........................................5 4.1. NTP Interleave............................................5 4.1.1. Standardization Considerations.......................6 4.2. Use of IEEE 1588 PTP and 802.1AS Mechanisms in the Underlying Network Service (e.g., Network Interface Controller [NIC]).........................................................6 4.2.1. Standardization Considerations.......................6 4.3. Use of IEEE 1588 PTP to Synchronize Computer Clocks.......6 4.3.1. Standardization Considerations.......................7 5. Early Experimentation..........................................7 5.1. Future Experimentation....................................8 6. Analysis of Results............................................9 7. Security Considerations.......................................10 8. Internet Assigned Numbers Authority (IANA) Considerations.....10 9. Conclusions...................................................10 10. Informative References.......................................11 11. Acknowledgments..............................................11 1. Introduction The IETF Timing over IP Connection and Transfer of Clock (TICTOC) Working Group was formed to investigate emerging needs to distribute highly accurate time and frequency information over Internet Protocol (IP) and Multiprotocol Label Switching (MPLS) Packet Switched Networks (PSNs). In this draft, new mechanisms beyond those identified in the NTPv4 standard (i.e., Request For Comment (RFC) Marlow/Knickerbocker/Plunkett Expires August 25, 2011 [Page 2] Internet-Draft Improving Computer Clock Accuracy February 2011 5905) are considered to provide increased time synchronization accuracy for computer (i.e., operating system) clocks' time and frequency. The mechanisms considered are those that will provide improved estimates as to when a packet is put on the network, transferred across a network, or taken from the network. This draft identifies a set of mechanisms that are candidates for experimentation. Standardization considerations will be described for the mechanisms identified. The purpose of this draft is to examine methods for improving NTPv4 time synchronization performance. The authors are requesting comments and contributions on the mechanisms described and on additional mechanisms that should be considered. It is hoped that discussions within the IETF TICTOC Working Group will motivate experimentation that will lead to standardization actions to enable better accuracy to those utilizing a future Network Time Protocol (NTP) specification. 1.1. Motivation for Increased Performance There are two reasons to improve upon the time synchronization performance that is currently available from NTP. Not only is the increased performance needed for existing product designs that would make use of the added performance if it were available, but it is expected that new uses will be identified that were not even possible until performance is improved. This is similar to how network speeds are increased every several years, and the uses for the increased network bandwidth soon follow. The current methods for achieving an increase in time synchronization performance involve use of a technology separate from the existing computer network (e.g., Inter-Range Instrumentation Group [IRIG] technology) or use of a technology like Precision Time Protocol (PTP), which is defined in the Institute of Electrical and Electronics Engineers (IEEE) 1588 standard. With PTP, the computer applications must interface with the installed PTP hardware in order to read time from its oscillator. There is a lot of resiliency built into NTP, which does not exist in the PTP hardware oscillator. It is unknown what happens to the time provided by the PTP hardware when a network switch in the network path to the time source is temporarily unavailable (i.e., the network switch gets rebooted). It would be beneficial to have the resiliency of the NTP algorithms be paired with the highly accurate PTP hardware-based time distribution. Marlow/Knickerbocker/Plunkett Expires August 25, 2011 [Page 3] Internet-Draft Improving Computer Clock Accuracy February 2011 1.2. NTP/PTP Commonality and Differences NTP and PTP are both packet-based protocols for exchanging time with a time server over a computer network. Both protocols are used to determine the offset between two independent clocks. Both implement a hierarchical tree structure for obtaining time from a master time source through intermediary time sources to time clients. Both assume network paths are symmetric, and both have their own methods for addressing network delays that are not symmetric. NTP uses its algorithms to determine which of several consecutive time measurements are most accurate and uses that measurement. PTP makes use of hardware means of measuring delays as packets traverse intermediate network devices and corrects its received time information based upon those measured delays. Because PTP has the ability to measure actual packet delays and to correct for them, PTP can provide the most accurate measurement of clock offset between two clocks. PTP does not define the method for synchronizing that clock once the highly accurate time measurements have been obtained. PTP is normally used to synchronize a hardware clock located on an interface card and does not synchronize the operating system clock. NTP, on the other hand, possesses the ability to synchronize the operating system clock based upon received clock offset measurements. The NTP algorithms have a lot of resiliency so that operating system clocks stay stable despite the conditions on the network. 2. Use Case Targeted The use case considered by this internet draft is a dense concentration of computing elements connected by a network. A satellite-based time source (e.g., Global Positioning System [GPS]) is used for synchronizing primary time servers. Secondary time servers and leaf computing elements are synchronized to the primary time servers via the network. In this use case, there are approximately 150 or so total computers where there are 3 to 4 levels of time servers. These time servers may have to communicate to each other through layer 2 and layer 3 network switches (could be 10 to 20 different layer 2 subnetworks). All of the computers are connected together through gigabit or faster network connections. In this environment, there will be some groups of computers that will need to synchronize to each other to within a microsecond, while other groups of computers only have to be synchronized to each other to within a millisecond. In this use case, there is one interconnected time synchronization scheme where NTP, PTP, or a combination of both is used to meet all time synchronization needs. Marlow/Knickerbocker/Plunkett Expires August 25, 2011 [Page 4] Internet-Draft Improving Computer Clock Accuracy February 2011 2.1. Emerging Need for NTP and PTP Commonality Due to its accuracy capabilities, PTP is beginning to replace NTP as the base protocol for time clients within dense computing sites. This results in an implementation in which some hosts use PTP while others use NTP within the same building and sometimes within the same room. Over time, hosts are being changed from NTP to PTP. This leads to an emerging need to provide similar approaches for basic time service functions for operational ease of managing the time distribution assets. Examples of functions where commonality is considered to be an emerging need include providing synchronization of the computer clock, providing management to time clients, and configuring timeservers. A standard means of synchronizing the computer clock for both protocols is in particular of interest because there appears to be no value in using different methods when the hosts that both protocols are supporting are often working within the same system. In addition, there are highly accurate PTP time clients that could serve as highly accurate secondary timeservers for the NTP time clients if this capability were supported in vendor products. 3. Approach The approach taken by the authors of this draft includes determining what the current accuracy capabilities are with NTPv4 and investigating additional mechanisms that may provide improvements in accuracy. Through experiments of those additional mechanisms, estimations of improvements can be calculated. Depending on the standardization difficulty and potential benefits offered, more than one standardization action may be recommended in the future. 4. Mechanisms Considered 4.1. NTP Interleave NTP Interleave is an extension of the NTPv4 protocol, which is included in the current NTP distribution [1]. It is designed to be backward compatible (i.e., not affecting NTP implementations that do not use the interleave extensions). It also utilizes the same NTP packet format as the current standard NTPv4. NTP Interleave uses an IEEE 1588 PTP-like feature that provides a follow-up packet with a better estimate of when a previous NTP packet was sent on the network and a message exchange sequence to determine network mean path delay. Marlow/Knickerbocker/Plunkett Expires August 25, 2011 [Page 5] Internet-Draft Improving Computer Clock Accuracy February 2011 This mechanism could be used by some of the primary time servers for synchronizing secondary (i.e., lower stratum) time servers and leaf computing elements, which have very accurate time synchronization requirements. Future experimentation may identify what gains are possible with this mechanism. 4.1.1. Standardization Considerations If the NTP Interleave investigation generates promising results, this may initiate a standardization proposal to add this capability to the NTPv4 standard. 4.2. Use of IEEE 1588 PTP and 802.1AS Mechanisms in the Underlying Network Service (e.g., Network Interface Controller [NIC]) The purpose of investigating this mechanism is to determine if using special capabilities in the underlying network service can improve the timestamp estimates when NTP packets are put on the network, transferred across networks, or taken from the network. 4.2.1. Standardization Considerations If the investigation of these mechanisms generates promising results, this may initiate a standardization proposal for additions to the NTPv4 standard to make use of these capabilities. Alternatively, a modification to the NTPv4 standard may be proposed, which would enable hardware assists to be incorporated into future NICs. 4.3. Use of IEEE 1588 PTP to Synchronize Computer Clocks The purpose of investigating this mechanism is to determine the viability of using the IEEE 1588 to synchronize computing elements, which require very accurate synchronization. This mechanism considers bringing the 1588 synchronization all the way to the computer clock through a standardized clock discipline algorithm. Computing elements synchronized by 1588 are candidates to be time servers (by the use of NTP) for computing elements not synchronized by 1588. Based on their respective strengths, the natural way to merge NTP and PTP would be to use PTP as the means of obtaining extremely accurate time information from across the network and to let the NTP algorithms use that time to keep local clocks synchronized. Marlow/Knickerbocker/Plunkett Expires August 25, 2011 [Page 6] Internet-Draft Improving Computer Clock Accuracy February 2011 4.3.1. Standardization Considerations If the investigation of this mechanism generates promising results, this may initiate a standardization proposal to specify a 1588 profile for use by NTP. It may be possible to replace the current NTP clock coordination services without affecting the NTP time management services or the clock access mechanisms used by each operating system. A variety of studies will be needed if this approach is pursued, including a study to determine if there are any issues for secondary time servers to run both NTP and PTP. If such issues are identified, standards activities may be needed in the IETF or in IEEE 1588. 5. Early Experimentation Some preliminary experiments were performed that tested the new Interleave mode available in NTP version 4.2.6. This mode mimics the operation of PTP defined by IEEE 1588 where an additional follow-up message is sent so that a more accurate transmission time can be used. In this experiment, seven workstations running Red Hat Enterprise Linux (RHEL) 5 were used. These workstations are 2-years old and make use of two dual-core processors. Since the GPS-based (i.e., stratum 1) time server does not currently have version 4.2.6 of NTP available, which supports the interleave mode, one of the seven workstations was synchronized to the stratum 1 time server; then, served as the stratum 2 time server to the other six stratum 3 workstations. The stratum 2 server and four of the six other workstations were upgraded to use NTP version 4.2.6, while the remaining two workstations were left running version 4.2.2 of NTP that was included with RHEL 5. Interleave mode was achieved by using the "xleave" option when either the broadcast mode or peer mode was used under NTP. The stratum 2 server was configured as a broadcast server, making use of the standard multicast address and using the "xleave" option. Two of the workstations running NTP 4.2.6 were configured as multicast clients so that the Interleave mode was utilized. The other two NTP 4.2.6 workstations were configured to synchronize to the stratum 2 server using standard client/server (unicast) mode. The two workstations running 4.2.2 of NTP were configured to be broadcast clients; however, they did not use Interleave mode since NTP 4.2.2 does not include Interleave support. All NTP polling intervals were configured to 16 seconds. Offset measurements were obtained between the six clients and the stratum 2 server using the ntpdate command with the "-q" option. Measurements were taken every minute over a period of approximately Marlow/Knickerbocker/Plunkett Expires August 25, 2011 [Page 7] Internet-Draft Improving Computer Clock Accuracy February 2011 four days. These workstations were not running any other major tasks, and NTP ran over a network with no discernable network load. All of the workstations were connected through the same Virtual Local Area Network on the same network switch (i.e., no routers involved). All of the network connections were 100 Mbit/sec Ethernet. Some results were obtained from the experiments where the average and the standard deviation of the absolute value of clock offset were measured. The worst behaved NTP Interleave client was able to stay synchronized with an average clock offset of 9 microseconds with a standard deviation of 8 microseconds. The worst behaved computer that synchronized using client/server mode was able to maintain an average clock offset of 11 microseconds with a standard deviation of 10 microseconds. The worst behaved broadband (without NTP Interleave) client stayed synchronized with an average clock offset of 49 microseconds and with a standard deviation of 58 microseconds. These results illustrate that broadcast with NTP Interleave does provide results that are better than having every client poll the server via unicast. However, the result is not significantly better (e.g., not an order of magnitude better). Preliminary experiments with hardware-based PTP have been performed in the past where the average offsets between PTP NICs and the PTP Grandmaster clock are in the hundreds of nanoseconds with standard deviations in the tens of nanoseconds. 5.1. Future Experimentation The results so far should be treated as preliminary, and further experiments are needed to draw conclusions. One concern is that the clock offsets are in the microsecond range. The use of ntpdate may not be a valid way to accurately measure clock offsets at this level since ntpdate makes measurements across the network and is susceptible to errors caused by variations in network delay. Further work is needed to ensure valid offset measurements. An out-of-band measurement technique, which is not affected by variations in network delay, needs to be investigated for use in future experiments. Another concern is that no load has been applied either to the processor or to the network carrying the NTP packets. Future experiments need to be performed to measure the resiliency of the NTP Interleave while under network and processor load and then make comparisons to the other time synchronization methods. This experiment was performed in a configuration in which the test Marlow/Knickerbocker/Plunkett Expires August 25, 2011 [Page 8] Internet-Draft Improving Computer Clock Accuracy February 2011 workstations were connected to the same network switch. Future experiments should be performed to determine how performance is affected when a more complex network configuration is used. Developing a standard to define time synchronization performance metrics would be beneficial by allowing different experimental efforts to be performed in a way that the results are comparable. 6. Analysis of Results In 2008, it was reported [2] that the synchronization of computer clocks using NTPv4 over a Local Area Network (LAN) can approach values on the order of ~145us (mean plus 3 sigma). This has been demonstrated in the laboratory and was achieved across a single stratum (e.g., stratum 1 to stratum 2). Recently, using newer hardware and a newer version of NTP, time synchronization values were measured on the order of ~40us (mean plus 3 sigma). The variation in the time synchronization comprises the majority of the 40us value. The results need to be analyzed in detail, but in a little over two years, the time synchronization of computer clocks over a LAN has improved; most likely due to hardware improvements and minor software enhancements. If this 40us synchronization can be maintained from stratum to stratum for all subsequent tiers in a well-engineered network with 3 to 5 stratums, a maximum theoretical time synchronization offset on the order of 120 to 200us could be achieved. A 50 percent improvement in the variability of clock synchronization from stratum to stratum would reduce this number to less than 125us. This does not imply that stratum-to-stratum accuracy should not be improved. On the contrary, by working on the accuracy and variability together all users of time synchronization will benefit. This needs to be accomplished without overloading the computer or the network. A 50 percent improvement appears to be a reasonable goal; however, if the stratum-to-stratum synchronization variability could be improved by an order of magnitude, it is reasonable to anticipate maximum theoretical time synchronization offsets of 50us or less in a well-behaved LAN and potentially in the hundreds of microseconds for a Wide Area Network. Follow-on discussions within the IETF TICTOC Working Group are needed to define what level of improvement in accuracy would warrant a standardization action to add an Interleave option to the NTP standard. Marlow/Knickerbocker/Plunkett Expires August 25, 2011 [Page 9] Internet-Draft Improving Computer Clock Accuracy February 2011 7. Security Considerations Security aspects of the aforementioned options will need to be considered in more detail. 8. Internet Assigned Numbers Authority (IANA) Considerations No IANA actions are required as a result of the publication of this document. 9. Conclusions Preliminary results tentatively indicate that an Interleave capability can provide a modest improvement to the accuracy achieved with NTPv4 and, as a result of these preliminary findings, the authors recommend conducting further testing to determine whether this should be added as an option to the NTPv4 standard. Preliminary results also tentatively indicate that Interleave will not provide accuracies in the range that PTP with hardware assists can; thus, the other two mechanisms described in this paper should be pursued in parallel. Assuming any of the proposed mechanisms in this draft or other mechanisms proposed through IETF TICTOC discussions provide a significant improvement in computer clock time synchronization, it is recommended that the IETF TICTOC Working Group initiate work on a standards-track targeted working group Internet-Draft. It would be of benefit to the IETF TICTOC Working Group to standardize the performance metrics used in describing the behavior of clock synchronization. Such a standard would enable better comparisons between the mechanisms considered. In addition to the same definitions of the metrics used, better agreement should be obtained in experiments being performed by different organizations. The authors are interested in contributions on the mechanisms described in this draft as well as on additional mechanisms that may improve the accuracy of computer clocks synchronized via a network. The authors request that other experimental results on mechanisms that can improve NTPv4 accuracy be shared. It is hoped that discussions on this topic within the IETF TICTOC Working Group will lead to standardization actions to enable better accuracy to those utilizing a future NTP specification. Marlow/Knickerbocker/Plunkett Expires August 25, 2011 [Page 10] Internet-Draft Improving Computer Clock Accuracy February 2011 10. Informative References [1] Mills, David, 1 September 2010. "Analysis and Simulation of the NTP On-Wire Protocols." http://www.eecis.udel.edu/~mills/onwire.html. Accessed 3 September 2010. [2] O'Donoghue, K., Glass, M., and Plunkett, T. (2008). "Next Steps In Network Time Synchronization For Navy Shipboard Applications" [Electronic Version]. 40th Annual Precise Time and Time Interval (PTTI) Meeting, pp. 187-195. 11. Acknowledgments The authors wish to thank Karen O'Donoghue (Internet Society) for her valuable comments. Approved for public release. Distribution is unlimited. This document was prepared using 2-Word-v2.0.template.dot. Authors' Addresses David Marlow NSWC-DD 17214 Avenue B Suite 126 Dahlgren, VA. 22448-5147 USA Email: david.marlow@navy.mil Sterling Knickerbocker NSWC-DD 17214 Avenue B Suite 126 Dahlgren, VA. 22448-5147 USA Email: sterling.knickerbocker@navy.mil Timothy Plunkett NSWC-DD Marlow/Knickerbocker/Plunkett Expires August 25, 2011 [Page 11] Internet-Draft Improving Computer Clock Accuracy February 2011 17214 Avenue B Suite 126 Dahlgren, VA. 22448-5147 USA Email: timothy.plunkett@navy.mil Marlow/Knickerbocker/Plunkett Expires August 25, 2011 [Page 12]