INTERNET-DRAFT Eric A. Hall Document: draft-hall-dns-data-04.txt March 2004 Expires: September, 2004 Category: Informational Considerations for DNS Resource Records Status of this Memo This document is an Internet-Draft and is in full conformance with all provisions of Section 10 of RFC 2026. Internet-Drafts are working documents of the Internet Engineering Task Force (IETF), its areas, and its working groups. Note that other groups may also distribute working documents as Internet- Drafts. Internet-Drafts are draft documents valid for a maximum of six months and may be updated, replaced, or obsoleted by other documents at any time. It is inappropriate to use Internet-Drafts as reference material or to cite them other than as "work in progress." The list of current Internet-Drafts can be accessed at http://www.ietf.org/ietf/1id-abstracts.txt The list of Internet-Draft Shadow Directories can be accessed at http://www.ietf.org/shadow.html. Copyright Notice Copyright (C) The Internet Society (2003). All Rights Reserved. Abstract This document discusses some common design considerations for DNS resource records and data models. Internet Draft draft-hall-dns-data-04.txt March 2004 Table of Contents 1. Introduction..............................................2 2. Prerequisites and Terminology.............................3 3. Architectural Principles and Inherent Limitations.........3 3.1. Resource Records.......................................3 3.2. Hierarchical Partitioning..............................5 3.3. Minimalist Messages....................................7 3.4. Built-In Record Caching................................9 3.5. Unreliable Hinting....................................10 3.6. World-Readable Data...................................11 3.7. Implementation Issues.................................11 4. Design Conclusion........................................12 5. Security Considerations..................................13 6. IANA Considerations......................................13 7. Normative References.....................................13 8. Acknowledgments..........................................13 Author's Address..............................................14 Full Copyright Statement......................................14 1. Introduction In terms of deployment, the Domain Name System (DNS) [STD13] is an extremely successful network service, having perhaps the widest usage of all Internet services. Unfortunately, the omnipresence of DNS makes it a frequent target for well-intentioned efforts to extend the service into roles that it is technically unsuited to provide, or which would impose excessive burdens on the Internet community as a whole if they were widely adopted. This document attempts to itemize some of these issues, so that planners and developers can try to avoid these concerns during their planning cycles. However, it should also be recognized that there are several modern DNS usage models which violate more than one of the considerations listed in this document, but which still provide significant value for the Internet community. As such, this document should not be considered as a governing device of any kind, and should not be used to reject any and all proposals for new usage models. Instead, this document is intended to be used to facilitate honest discussion about the kinds of problems that a particular proposal may be expected to encounter, or the burdens that it may impose on the Internet community as a whole if it were to be widely adopted. Hall I-D Expires: September 2004 [page 2] Internet Draft draft-hall-dns-data-04.txt March 2004 2. Prerequisites and Terminology Readers of this document are expected to be familiar with STD 13 [STD13], STD 3 [STD3], and RFC 2181 [RFC2181]. The key words "MUST", "MUST NOT", "REQUIRED", "SHALL", "SHALL NOT", "SHOULD", "SHOULD NOT", "RECOMMENDED", "MAY", and "OPTIONAL" in this document are to be interpreted as described in RFC 2119. 3. Architectural Principles and Inherent Limitations The current collection of DNS specifications define a lightweight and anonymous "lookup-by-name" service, with compact datagrams being relayed through a structured network of authoritative servers and caches, each of which provide access to specific database partitions and/or resource records. The Domain Name System is able to fulfill its primary responsibility as a fast and robust distributed naming service directly as a result of these design principles. However, DNS has several built-in critical limitations as a direct result of the highly-optimized lookup model. For example, DNS does not provide any functions to "search-by-value", nor does it provide any sort of mechanisms for user authentication, access control services, cache-validation, nor most of the other features which are typically associated with general-purpose databases or directories. Although DNS could be extended to provide some of these usages, such an effort would require a significant amount of engineering and deployment effort in order to preserve compatibility with existing DNS systems. Furthermore, there is a significant danger inherent in overloading DNS with excessive features such that the service itself becomes incapable of performing lightweight lookups quickly and efficiently, thereby precluding its primary purpose. Therefore, it is incumbent upon protocol developers and planners to recognize and accommodate the issues which govern DNS in its current form. This document describes the most common of those issues, so that some of the most common problems can be avoided early in the design cycle. 3.1. Resource Records Data stored in DNS uses a common resource record data-structure, consisting of six fields ["domain name", "type", "class", "time- Hall I-D Expires: September 2004 [page 3] Internet Draft draft-hall-dns-data-04.txt March 2004 to-live", "length" and "data" (the "data" field is further structured according to the kind of data being provided by the resource record itself)]. The domain name, type and class fields collectively form a unique identifier for each resource record, and allow clients to specifically identity the kind of data they want to retrieve for a specific named resource. Multiple resource records may share the same domain name, type and/or class values, but those resource records must have different data values to be considered unique. At a minimum, all lookup queries must explicitly identify the domain name of the resource record being requested. Queries may request all known types and/or classes associated with the named resource, but typically specify those fields as well. If a query results in multiple matches, then all of the matching resource records will be returned. Furthermore, DNS is only capable of issuing lookup-by-name queries, and does not provide any queries which would allow a resolver to search for resource records which contain a particular data value (although the original DNS specifications did provide a mechanism for searching a specific server for resource records with matching data-values, this feature was never widely deployed, and the query-type has since been officially deprecated). Meanwhile, DNS has never provided any means to search for all resource records of a particular type or class, without the client also specifying an exact domain name to match against, meaning that it is also not possible to query for all resource records of a specific type regardless of their name. In theory, it would be possible to create a super-index of all zones in the entire distributed database and issue these kinds of searches against that index, although nobody has ever built such a system. It is also possible to fake these kinds of searches on a per-zone basis by transferring the entire zone contents down to the client and then performing local searches against all of the resource record data. However, neither of these scenarios represent normal DNS lookup-by-name queries, and would not be representative of typical DNS transactions and client processes. In the absence of these mechanisms, designers must be aware that they can only issue queries against the name of a resource record unless they are willing to use something other than DNS. Hall I-D Expires: September 2004 [page 4] Internet Draft draft-hall-dns-data-04.txt March 2004 Since it is not possible for a query to specify anything other than the domain name, type and class fields, it is also not possible to explicitly request an exact instance from among a set, unless only one instance of the requested resource record exists at the specified domain name. However, it is not possible to guarantee that a particular resource record will only exist in the singular form at any given time. Although it is possible to demand that administrators "MUST NOT" create more than one instance of a particular resource record for any domain name, such demands are usually at the mercy of the administrators of those systems, and are generally unenforceable. Data models which depend on singular instances of a particular record should be designed with the recognition that multiple resource records may be returned anyway, and should be prepared to deal with that scenario. By the same token, it is also not possible to be sure that all of the resource records from a set will always be returned. For example, the original DNS specifications allowed each resource record in a set to have different time-to-live values, and this allowed (in theory) each resource record to be aged out of a cache at different times. Furthermore, there have been some secondary bugs in some implementations which have resulted in incomplete answer sets being returned and subsequently cached by other nodes. Although these problems have mostly been addressed over time, it is still not possible to guarantee with absolute certainty that all of the records in a set will always be returned. Data models which depend on spreading component data over multiple resource records in a set should be designed with this in mind. On a related point, many application designers are tempted to utilize the TXT resource record as a container for structured data. This is generally a terrible idea. For one thing, allowing one application to remodel the TXT record means allowing everyone to do so, resulting in all of the TXT resource records being returned whenever any of them are requested, with an increased potential for message overflow (see the discussion in section 3.3 for the problems this can cause). Furthermore, this model complicates message processing, in that the contents of the TXT resource record have to be analyzed and string-matched. These problems are avoided by the use of purposeful, task-specific resource records. 3.2. Hierarchical Partitioning From a high-level perspective, the DNS database is distributed across multiple partitions (called "zones"), each of which have Hall I-D Expires: September 2004 [page 5] Internet Draft draft-hall-dns-data-04.txt March 2004 ownership for the domain names within that partition. Zones are linked in a hierarchical tree, with the top-level zones having zones directly beneath them, and with some of those zones having subordinate zones, and so forth. Although the zones are arranged in a hierarchy, each zone acts as an independent partition and is usually only concerned with the records that it controls directly. The hierarchical zone structure is traversed by resolvers whenever a zone which is authoritative for a named resource record needs to be located (this usually only happens when the answer has not already been cached). In this regard, the domain name of a resource record acts like a lookup key, with resolvers matching the key value against the zone hierarchy until either an answer or an error is returned. Unfortunately, domain names are restricted to a maximum length of 255 characters. Since a domain name is the primary identifier for a resource record, and since the domain name also identifies the zone where a resource record is stored, the length restrictions of a domain name can be a significant limitation in some cases. For example, a domain name for a resource record in a zone that is nested several layers deep in the global hierarchy could face significantly tighter space constraints than domain names for resource records in a top-level zone, simply because there will be fewer octets left to work with in the lower-level zones. This can be a significant concern with applications which require the use of application-specific domain name sequences, especially when those sequences are relatively long. In some cases, it may simply be impossible to use those sequences in some zones, given the space restrictions. As such, the use of application-specific domain name sequences should generally be avoided. The use of the per-zone matching system also introduces certain complexities in feature-negotiation and error-recovery processes which are generally required in datagram transaction models. In particular, data-models which depend on issuing "fallback" queries in those cases where an earlier query has failed can impose significant burdens on the DNS infrastructure as a whole if the fallback mechanisms are triggered too frequently or if they are triggered at heavily-loaded servers. These problems can be slightly lessened somewhat if the fallback processing only occurs after the authoritative servers for the partition have already been learned, since the fallback queries will be sent directly to those servers, and will not impose Hall I-D Expires: September 2004 [page 6] Internet Draft draft-hall-dns-data-04.txt March 2004 additional burdens on the servers responsible for the root and top-level zones. However, the downside to this directed-fallback approach is that fewer systems in the query path will have the opportunity to cache the resulting answer data, which in turn will guarantee that a larger number of queries will almost certainly be needed in the common case. 3.3. Minimalist Messages The DNS protocol uses a highly-compact, binary message format which is specifically suited for fast and lightweight lookup transactions. There are very few spurious bits or fields in the DNS message (there is no "version" field, for example), with the ultimate objective being very small message sizes. By default, DNS uses UDP to transfer messages, avoiding the latency and processing costs that are typically associated with TCP sessions. However, there are some situations in which UDP cannot be used, and in those cases, DNS will typically use TCP in order to ensure that lookups succeed. Standard DNS messages sent over UDP have a maximum message size of 512 bytes. If a lookup results in a response message that exceeds the maximum message or datagram sizes, the query process must be restarted using TCP. Meanwhile, extended DNS (EDNS) [RFC2671] can carry messages up to 65,535 bytes over UDP, although the actual payload size is usually limited to 1280 bytes due to limitations in physical media capacity and problems that arise from fragmentation. If the size of the EDNS message exceeds the capacity of the end-to-end link, TCP will again be needed. However, DNS messages sent over TCP are themselves limited to a maximum size of 65,535 bytes, and messages which are larger than that size cannot be transferred over DNS at all. Furthermore, not all DNS servers support the use of TCP, and in those cases, messages which overflow the 512-byte limit for UDP will also be inaccessible. In short, messages which are larger than 512 bytes always cause performance problems and sometimes trigger catastrophic failures, while messages which are larger than 65,535 bytes always trigger catastrophic failures. In those cases where TCP works as expected, there can be several penalties from its use. For example, TCP session management typically consumes more resources than UDP datagrams, which can significantly limit the number of queries that a server is able to process at any given time. For a particularly busy server, Hall I-D Expires: September 2004 [page 7] Internet Draft draft-hall-dns-data-04.txt March 2004 processing a significant number of TCP transactions can mean that other transactions will have to be rejected. Meanwhile, the use of TCP also requires more round-trips, which can sometimes cause timers to expire while the query is still being processed, resulting in multiple duplicate queries going to that server (each of which will subsequently require TCP sessions), accelerating the negative affects. It's also important to recognize that TCP queries are often exchanged between the local resolver and the target server directly, and can bypass some parts of the caching infrastructure. As a result, answers returned over the TCP connection might not be cached by intermediary nodes, and the entire process would need to be repeated for each instance of the same query. For all of these reasons, planners and developers are strongly encouraged to limit resource record data to sizes that will not cause UDP messages to overflow. In those cases where this is unavoidable, they should be prepared for a variety of problems, including performance degradation and outright failure. Note that the DNS message format uses a protocol-specific compression technique which can be used to substitute label sequences with offset pointers to previous occurrences of those sequences, thereby saving some space in the message itself. However, this mechanism only works in a few instances, and is not as widely usable as many people hope. For example, older caches are not aware of newer resource record data-structures, so the compression mechanism cannot be used in the data fields of those resource records, but instead can only be used with the owner domain name of the resource record itself. This is not a problem for resource records which were defined as part of [STD13], since those resource records have well-known formats, but newer resource records are unable to utilize the standard compression mechanism. This is an especially important consideration to keep in mind when considering large data-structures. While it is tempting to believe that domain names can be compressed to save space in the message, this simply is not true as often as people would like. It is entirely feasible for newer resource records to define their own (record-specific) compression algorithms, although such schemes must be planned with legacy caches in mind -- those devices will not be able to expand the contents of those records, nor will they be able to apply any other kind of logic against those records. Hall I-D Expires: September 2004 [page 8] Internet Draft draft-hall-dns-data-04.txt March 2004 3.4. Built-In Record Caching DNS resolvers and servers are allowed to cache resource records that they have discovered as part of normal query processing. This allows subsequent queries for that information to be answered immediately from the cache, without requiring another batch of transactions for the same information. In turn, this ensures that lookups are answered in the shortest amount of time, that servers are not excessively burdened by unnecessary queries, and that the total number of transactions are kept to a minimum. Since DNS is optimized for lookups, the use of caching is generally considered a positive feature. However, caching can also be somewhat hostile towards certain usage models, especially since DNS does not provide any mechanisms for forcing a system to flush its cache of previously discovered records. In particular, caches prevent data from being validated against an authoritative source, in that a resolver or application cannot "demand" that a query be forwarded to an authoritative server (the client can do this on its own accord, but cannot request a proxy to do this on behalf of the client). While this is normally beneficial for lookup activities, it can be a devastating feature for data models that require data-integrity at all times. Although DNS servers can dictate the maximum length of time that a resource record is to be held in a cache, data models which require the use of low time-to-live settings are generally frowned upon by the DNS community, as these resource records place a disproportionate burden on the infrastructure. Furthermore, some DNS agents are known to apply their own minimum time-to-live values, regardless of the settings associated with the original data. As a result, DNS is generally considered to be inappropriate for data models which require full-time and instantaneous data integrity, and developers are generally encouraged to look towards other services if this level of responsiveness is needed, especially if the application is expected to be widely deployed. Another issue related to caching limitations is the amount of memory available to each particular cache. All systems have fixed amounts of available memory, and when that memory is consumed, some data will necessarily have to be flushed in order for any new data to be stored. If the flushed data is subsequently needed again, the query path will have to be reprocessed, and the cache will have to flush some other data in order to make room for the Hall I-D Expires: September 2004 [page 9] Internet Draft draft-hall-dns-data-04.txt March 2004 answers. In heavily loaded environments (such as a very busy ISP), this can result in a constant churning of the memory pool. This is obviously a good reason to limit the size of each resource record's data-structures, but it is also a good reason for limiting the total number of resource records in a set. Since each entry will have to consume memory in a cache somewhere, large resource record data blocks and large sets of resource records will both contribute to the potential for cache churning. 3.5. Unreliable Hinting DNS responses provide for the inclusion of hinting data, by way of the Additional-Data section of the response message. In the usual case, this section of the response message will contain resource records that are associated with the originally requested resource records (such as listing the IP addresses associated with a name server or mail server). However, due to certain design considerations, this data is often incomplete, and is almost always unreliable, and therefore must often be ignored. Specifically, DNS messages contain a "truncation" flag which indicates whether or not all of the answer data has been returned in the message (if the flag is enabled, the recipient system will need to retry the query via some other transport, as discussed in section 3.3). However, this flag does not apply to the Additional- Data section of the response message, and in the absence of any such flag (and there is none, so its absence is guaranteed), recipient systems must always assume that the Additional-Data section is incomplete. Meanwhile, caches often store resource records gleaned from the additional-data section, and then provide that data as answers to subsequent queries. In order to prevent caches from capturing and relaying incomplete resource record sets, authoritative servers should only provide them when the full set will fit within the Additional-Data section. Unfortunately, there are still several systems in use which do not conform to this behavior, and it is therefore possible that any given Additional-Data section will not contain a complete set. In theory, it is possible to examine the Authoritative Answer flag of a response message to determine whether or not the full set of resource records have been provided. Unfortunately, there are implementations which are known to have inheritance bugs, where resource records from the Additional-Data section are stored with Hall I-D Expires: September 2004 [page 10] Internet Draft draft-hall-dns-data-04.txt March 2004 the Authoritative Answer flag, even though that flag is only supposed to apply to the Answer section. As a result of these considerations, applications are generally encouraged to avoid the Additional-Data section entirely. Furthermore, if an application does make use of any resource records in this section, those applications are generally encouraged to issue new queries for those resource records if the data is absolutely critical, thus ensuring that the full answer set is always retrieved. Unfortunately, even that cannot be guaranteed, due to the prevalence of bugs. 3.6. World-Readable Data DNS is optimal for publishing anonymous and world-readable data, given the implicit collusion between several of the design factors in the DNS model. For example, the need for small DNS messages generally precludes the use of data such as access-control lists, while the use of in-line caching generally precludes any presumption of privacy. More specifically, DNS does not provide any mechanisms for authenticating users during the lookup process, nor does it provide any mechanisms for linking access controls to a resource record across the global network of servers and caches. Without these features, DNS is unsuitable for applications which require authenticated access to private data. Furthermore, although some products provide mechanisms for restricting query-level access to ranges of IP addresses or other filtered sources, it is important to recognize that once the resource records get into a cache outside of the protected scope, the information is only as secure as that system. In this regard, a cache which resides outside of a firewall will be just as informative as the DNS servers inside the firewall. In the end, there is no such thing as "private" data with DNS. Developers must treat all data as if it will eventually be made public, and are strongly encouraged to use some other service if higher levels of security are required. 3.7. Implementation Issues Most of the DNS resolvers which are provided with operating systems and TCP/IP stacks are purposefully optimized for the most Hall I-D Expires: September 2004 [page 11] Internet Draft draft-hall-dns-data-04.txt March 2004 common queries, usually only offering APIs for resource record types such as IPv4 addresses and a couple of others, but with little or no direct support for other resource record types. Similarly, some resolvers do not provide the kind of granularity that an application may require. For example, some resolvers have been known to provide only one resource record from a set to the application, and these resolvers can cause problems with applications which need to see the full response. Although there are usually several mechanisms that application developers can pursue to overcome these kinds of limitations, many developers are loathe to do so, especially when their applications are ported across multiple operating systems. Unfortunately, this reticence can represent a significant hurdle towards wide deployment of even the best designed resource records and usage models, and is frequently the single largest obstacle which must be overcome by protocol developers and planners. By no means should this be considered as a show-stopper, but it should be recognized as a potentially significant hurdle, and that the majority of the applications will likely be unable to immediately adopt any sufficiently new technologies. 4. Design Conclusion Due to the architectural tradeoffs inherent in the DNS lookup model, some usage models are better suited to DNS than others. In particular, DNS is highly efficient at lookups of compact, public and relatively stable data. Conversely, DNS is unsuitable for value-based queries or searches, restricted-access data, highly- dynamic data, or large records and arrays. Applications which require access to those kinds of data should investigate services such as LDAP or HTTP as being more appropriate. Generally speaking, planners and developers can usually define their own resource record types as part of a standards-track specification without interference from the DNS community, as long as the functional scope is limited to defining data-structures for those resource record types. However, there are some cases where it may be useful or necessary for the DNS community to be involved with the standardization of a particular resource record type. In particular, if a resource record type requires a server to perform some kind of extra processing other than piping data from a database into a message, then the DNS community should be consulted. Similarly, requiring that servers provide additional Hall I-D Expires: September 2004 [page 12] Internet Draft draft-hall-dns-data-04.txt March 2004 data outside the answer section of the response message should be vetted with the community. Moreover, if a specification requires special structuring of the message for the benefit of a single service, then the DNS community should definitely be involved in the discussion, since any changes to the highly-optimized message format could be disastrous in non-obvious ways. Requests to reserve portions of the namespace for the use of a single network service should also be brought to the DNS community for discussion. Finally, if a particular usage goes against more than two of the recommendations put forth in this document, then it would probably be a good idea to consult with the DNS community over any alternatives which may be available. In all cases, IANA must be involved in delegating resource record type codes and mnemonics. 5. Security Considerations This document does not create any security considerations. 6. IANA Considerations This document does not create any IANA considerations. 7. Normative References [RFC2181] Elz, R., and R. Bush, "Clarifications to the DNS Specification", RFC 2181, July 1997. [RFC2671] Vixie, P., "Extension Mechanisms for DNS (EDNS0)", RFC 2671, August 1999. [STD3] Braden, R., "Requirements for Internet Hosts - Application and Support", STD 3, RFC 1123, October 1989. [STD13] Mockapetris, P., "Domain names - concepts and facilities", STD 13, RFC 1034 and "Domain names - implementation and specification", STD 13, RFC 1035, November 1987. 8. Acknowledgments Funding for the RFC editor function is currently provided by the Internet Society. Hall I-D Expires: September 2004 [page 13] Internet Draft draft-hall-dns-data-04.txt March 2004 Significant feedback on this document was provided by Edward Lewis and Walt Howard. Author's Address Eric A. Hall ehall@ehsco.com Full Copyright Statement Copyright (C) The Internet Society (2003). All Rights Reserved. This document and translations of it may be copied and furnished to others, and derivative works that comment on or otherwise explain it or assist in its implementation may be prepared, copied, published and distributed, in whole or in part, without restriction of any kind, provided that the above copyright notice and this paragraph are included on all such copies and derivative works. However, this document itself may not be modified in any way, such as by removing the copyright notice or references to the Internet Society or other Internet organizations, except as needed for the purpose of developing Internet standards in which case the procedures for copyrights defined in the Internet Standards process must be followed, or as required to translate it into languages other than English. The limited permissions granted above are perpetual and will not be revoked by the Internet Society or its successors or assigns. This document and the information contained herein is provided on an "AS IS" basis and THE INTERNET SOCIETY AND THE INTERNET ENGINEERING TASK FORCE DISCLAIMS ALL WARRANTIES, EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO ANY WARRANTY THAT THE USE OF THE INFORMATION HEREIN WILL NOT INFRINGE ANY RIGHTS OR ANY IMPLIED WARRANTIES OF MERCHANTABILITY OR FITNESS FOR A PARTICULAR PURPOSE. Hall I-D Expires: September 2004 [page 14]