ADD A. Campling Internet-Draft 419 Consulting Limited Intended status: Informational N. Kowalewski Expires: January 14, 2021 Deutsche Telekom G. Scalone Vodafone C. Box BT Group A. Winfield Sky July 13, 2020 Practical Observations from Encrypted DNS Deployments by Network Operators draft-campling-operator-observations-00 Abstract The following document includes observations regarding a variety of implementations of recursive DNS capabilities that are important to network operators in terms of delivering DNS services to their (several tens of millions of) customers. It highlights some of the challenges that need to be addressed to allow the widespread adoption of encrypted DNS by the end-users of network operators. The information is intended to aid the development of discovery mechanisms for protocols such as DNS-over-HTTPS. It clearly defines problems that need technical solutions to allow the deployment of encrypted DNS by the largest number of operators to the largest number of users in the shortest possible timeframe with little or no disruption to the user experience. Status of This Memo This Internet-Draft is submitted in full conformance with the provisions of BCP 78 and BCP 79. Internet-Drafts are working documents of the Internet Engineering Task Force (IETF). Note that other groups may also distribute working documents as Internet-Drafts. The list of current Internet- Drafts is at https://datatracker.ietf.org/drafts/current/. Internet-Drafts are draft documents valid for a maximum of six months and may be updated, replaced, or obsoleted by other documents at any time. It is inappropriate to use Internet-Drafts as reference material or to cite them other than as "work in progress." Campling, et al. Expires January 14, 2021 [Page 1] Internet-DraPractical Observations from Encrypted DNS Deploym July 2020 This Internet-Draft will expire on January 14, 2021. Copyright Notice Copyright (c) 2020 IETF Trust and the persons identified as the document authors. All rights reserved. This document is subject to BCP 78 and the IETF Trust's Legal Provisions Relating to IETF Documents (https://trustee.ietf.org/license-info) in effect on the date of publication of this document. Please review these documents carefully, as they describe your rights and restrictions with respect to this document. Code Components extracted from this document must include Simplified BSD License text as described in Section 4.e of the Trust Legal Provisions and are provided without warranty as described in the Simplified BSD License. 1. Introduction The IETF has developed many protocols to improve the security and reliability of DNS over UDP or TCP (Do53) [RFC1035] including DNS over TLS (DoT) [RFC7858], DNS over HTTPS (DoH) [RFC8484] and DNS Security Extensions (DNSSEC) [RFC2535]. To enable the broadest adoption of these technologies, there are issues for consideration of any discovery solutions that are proposed to the Adaptive DNS Discovery [ADD] working group. Many network operators, including Internet Service Providers (ISPs), whether using fixed or mobile networks, would like to ensure that their encrypted DNS services can be seamlessly discovered and used by applications and operating systems that support encrypted DNS, particularly DoH, in order that encrypted DNS can be deployed to the widest possible community of users. They would particularly like to ensure that any proposed DNS discovery mechanisms take into account ISP use-cases such as DNS forwarders on CPE (Customer Premises Equipment or routers), the use of DNS for CDNs (Content Delivery Networks) with local content caches and the non-public nature of most ISP DNS services. This document has taken observations and experiences from a number of network operators that have been actively working on adding support for encrypted DNS to their networks. It is intended to make clear the requirements needed by any discovery mechanism developed by the ADD group. It collates and succinctly describes common problems faced by existing stakeholders in adopting encrypted DNS mechanisms. This document also presents some background information that is relevant to describing the issues and explains concerns around Campling, et al. Expires January 14, 2021 [Page 2] Internet-DraPractical Observations from Encrypted DNS Deploym July 2020 current proposed solutions. It should also be noted that, in many European countries, some regulations are specific to ISPs. One such requirement is that their customers should be able to connect to the Internet with any home router of their choice even if a router is provided by the ISP as part of its service. Therefore new proocols cannot be accommodated simply by requiring ISP customers to upgrade their routers. 2. Rationale This document is intended provide information to aid interested parties in developing discovery mechanisms for protocols such as DoH to allow their adoption with minimal disruption to the end user experience, maximising the number of users that can enjoy an easy upgrade path towards DNS encryption. The information provided will help interested paries develop discovery mechanisms that avoid the unnecessary exclusion of the majority of customers of a significant number of ISPs (including the major ones in Europe that serve several tens of millions of customers) from easy access to this new technology using the secure by design, same-provider auto-upgrade mechanisms. Such discovery choices will ensure that easy access to encrypted DNS is not dependent on the Internet access network architecture and on the ease of upgrade of any CPE. In addition, it will ensure that users are not forced to change their DNS resolver to a third party, potentially via manual configuration by the user, possibly losing functionality in the process. 3. The 'Same Provider Auto-Upgrade' Model Both Google Chrome and Microsoft Windows (and perhaps other client software in the future) currently deploy encrypted DNS through a 'same provider auto-upgrade' (SPAU) model. This approach results in the client not needing to prompt the user to change to a different resolver operator to enjoy an encrypted connection. Instead the client will rather try to determine whether an encrypted channel exists for communication with the same resolver operator that the user currently employs for unencrypted DNS resolution. If such a channel can be found, the client will automatically upgrade the connection from the original unencrypted one to the new encrypted one; otherwise, the client will continue sending DNS queries unencrypted. The current implementation of this model is as follows: Campling, et al. Expires January 14, 2021 [Page 3] Internet-DraPractical Observations from Encrypted DNS Deploym July 2020 o Out of band, the client software vendor discovers ISPs running DoH services (in the case of Google Chrome, ISPs will more likely apply for inclusion through Google's announced process). The vendor records the existing (Do53) resolver IP addresses, and adds them to a hard-wired table that maps those existing Do53 IP addresses to the DoH service that the vendor discovered to be associated with those resolver IPs. o When the client starts for the first time, and thereafter whenever it detects a network change, it checks the resolver configuration of the local host. If the configured resolver matches one of the IPs listed in the above table, the client auto-upgrades to use the DoH service instead. The above method ensures that users are only upgraded to DoH when the vendor is sure that the DoH service is the same service as the Do53 service already used. 4. The Problem with Auto-Upgrade and Forwarders Automatic upgrades that rely upon the user device being able to know and compare the address of the resolver that is serving the device can fail in some home network environments where the CPE is acting as a DNS proxy. To do this, the CPE will run software like DNSMASQ which acts as a proxy between the client and the DNS resolver, also providing DHCP services and performing DNS caching as well as forwarding. This is often paired with a home network architecture that uses RFC1918 [RFC1918] private IP addresses. In circumstances where private IP addresses are used, any auto- upgrade on the user device that compares the IP address of the device's DNS resolver against a list of known public DNS resolvers will fail because the client DNS resolver is a RFC1918 private address of the CPE device and not the public address of the actual DNS resolver operated by the network operator. As can be seen, the existing SPAU model doesn't work with the DNS- forwarder, private IP approach commonly used by network operators. Given that this approach allows for the implementation of the best privacy practices and best latency/engineering reqirements, it shouldn't change, therefore the SPAU model needs to be adapted to work with it. 5. Why DNS Discovery Needs to Support Forwarders Campling, et al. Expires January 14, 2021 [Page 4] Internet-DraPractical Observations from Encrypted DNS Deploym July 2020 5.1. Loss of Functionality if CPE Doesn't Support DNS Forwarders If the CPE is upgraded to announce the public resolver to clients, the following functionality will be lost o Caching/Proxy on the CPE - This leads to more load on the ISP's DNS platform because every client talks directly to the public resolvers (not only the clients which are auto-upgraded to DoH but also all other clients). o Local DNS routing and resilience - Some deployments segment the user base into regions, with CPE in each region receiving a different IPv4 and IPv6 address for the DNS server, improving latency and load balancing, as well as helping with cyber resilience compared to a single address for a typical anycast implementation. o Addressing local clients via their names - Often the CPE assigns the name configured to a client to the client's IP address on the CPE (for example, if the hostname is set to 'myhost' on a home network to reach this host on that network under that name). This will not work if the clients communicate directly with a resolver in the carrier network nor would it for auto-upgraded clients because, even if they fallback to Do53, they will still ask a resolver in the carrier network and not the CPE - and in both cases private network details will be leaked. o The CPE is the only network element that is aware of the local network topology. If the local network information is lost it is not possible to differentiate devices. The Discovery mechanism alone is not enough to solve this use case as additional logic is required on the DoH server to send back the request to the CPE. By using EDNS0 (Extension Mechanisms for DNS) [RFC2671] it is possible for a client running on the CPE to pass EDNS0 information to the ISP's DNS and provide, to the opted-in customers, information on the client that performed the request. This in turn allows the execution, for example, of parental controls on devices belonging to children (there are various ways of doing this that preserves privacy, for example by providing information only about the required filtering profile or by providing only a locally generated ID to distinguish between devices without necessarily identifying them). o Similar to the above use-case, some CPE can be configured to perform filtering directly, relying on a DNS forwarder's ability to intercept and modify DNS queries to do so. Moving queries to the network DoH server removes this capability, allowing more data Campling, et al. Expires January 14, 2021 [Page 5] Internet-DraPractical Observations from Encrypted DNS Deploym July 2020 to leave the local network, even if a resolver is available to perform similar filtering. 5.2. Why Not Just Upgrade the CPE to Stop Forwarding? It may seem easier to simply ignore the loss of functionality detailed above and just upgrade the CPE to stop DNS forwarding. However, such a software upgrade programme, or even the wholesale replacement of CPE, is not without its own challenges. The following is based on information from various large European ISPs, all of which use a DNS forwarder in their CPE. This configuration applies to operators in multiple countries, each of which has many millions of customers, so is a fair reflection of the envirnment in which any DNS discovery process needs to operate. o Non-standard CPE - Whilst many ISPs provide their customers with CPE, some customers will elect to use alternative equipment which will not accept software upgrades o Multiple hardware variants - ISPs typically endeavour to maintain support for legacy CPE. Upgrading the CPE software therefore requires complex and lengthy quality assurance processes to accommodate the many device variants, with some of the larger ISPs having 20-30 variants of devices. o Large, dispersed customer bases - Cycle times to replace CPE are lengthy due to the costs and numbers involved, and the outcome of any upgrade programme is uncertain due to the aversion of some customers to replace their existing equipment In summary, the timeframe for non-critical software updates of ISP- supplied CPE can be lengthy. In addition, any such upgrades will only apply to the ISP-supplied CPE so will at best only ever reach between 60-80% of the customer base for many of the largest European ISPs. A replacement programme will also take an extended period without a guaranteed outcome, and that is without considering the cost. 5.3. The Advantage of Supporting Forwarding The above is intended to illustrate why it is more effective to ensure that DNS discovery methods, including those that support the SPAU model, are developed that work with the hardware and software environments in common use by network operators. Campling, et al. Expires January 14, 2021 [Page 6] Internet-DraPractical Observations from Encrypted DNS Deploym July 2020 6. Alternative Solutions Some may be tempted to suggest that the simplest solution to address the issues noted in this document would be for users to connect to global DNS resolvers. Aside from the loss of functionality and significant reduction in user choice that this would imply, it would also result in the further, forced, centralisation of Internet infrastructure, a policy outcome which is out of scope for the ADD working group. It would also, of course, result in the personal data of very large numbers of users to shared with additional parties simply to provide encrypted DNS functionality. A better approach would be to address the underlying issues so that client software is able to auto-discover and connect to encrypted resolvers on existing network wherever these are available, giving users a seamless upgrade, maintaining full functionality and maximising choice. 7. Extending the Use Case TO DO The information in this document is largely based on input from a number of large European network operators, augmented with knowledge of the operations of others, mainly in Europe but with some from North America. It would be beneficial to extend this document with data from additional ISPs to complement the existing content and also to extend the narrative with examples of additional working practices relating to the operation of DNS where possible. This would provide valuable information to inform the development of DNS discovery approaches that will benefit a far broader set of users than would otherwise be possible. To this end, additional contributions are welcomed as these would ensure that the document is fully representative of the significant use cases globally. 8. Acknowledgements In addition to the authors, this document is the product of an informal group of experts including the following people: Andy Fidler, BT plc Neil Cook, Open-Xchange Nic Leymann, Deutsche Telekom Campling, et al. Expires January 14, 2021 [Page 7] Internet-DraPractical Observations from Encrypted DNS Deploym July 2020 Ralf Weber, Akamai Vittorio Bertola, Open-Xchange 9. Informative References [ADD] IETF, "Adaptive DNS Discovery (ADD) Working Group", February 2020, . [EDDI] EDDI, "Encrypted DNS Deployment Initiative", July 2020, . [RFC1035] Mockapetris, P., "Domain names - implementation and specification", STD 13, RFC 1035, DOI 10.17487/RFC1035, November 1987, . [RFC1918] Rekhter, Y., Moskowitz, B., Karrenberg, D., de Groot, G., and E. Lear, "Address Allocation for Private Internets", BCP 5, RFC 1918, DOI 10.17487/RFC1918, February 1996, . [RFC2535] Eastlake 3rd, D., "Domain Name System Security Extensions", RFC 2535, DOI 10.17487/RFC2535, March 1999, . [RFC2671] Vixie, P., "Extension Mechanisms for DNS (EDNS0)", RFC 2671, DOI 10.17487/RFC2671, August 1999, . [RFC7858] Hu, Z., Zhu, L., Heidemann, J., Mankin, A., Wessels, D., and P. Hoffman, "Specification for DNS over Transport Layer Security (TLS)", RFC 7858, DOI 10.17487/RFC7858, May 2016, . [RFC8484] Hoffman, P. and P. McManus, "DNS Queries over HTTPS (DoH)", RFC 8484, DOI 10.17487/RFC8484, October 2018, . Authors' Addresses Andrew J Campling 419 Consulting Limited Email: Andrew.Campling@419.Consulting URI: https://www.419.Consulting/ Campling, et al. Expires January 14, 2021 [Page 8] Internet-DraPractical Observations from Encrypted DNS Deploym July 2020 Normen B Kowalewski Deutsche Telekom Email: Normen.Kowalewski@Telecom.DE URI: https://www.Telecom.DE/ Gianpaolo A Scalone Vodafone Email: Gianpaolo-Angelo.Scalone@Vodafone.Com URI: https://www.Vodafone.Com/ Chris Box BT Group Email: Chris.Box@BT.Com URI: https://www.BT.Com/ Alister Winfield Sky Email: Alister.Winfield@Sky.UK URI: https://www.Sky.Com/ Campling, et al. Expires January 14, 2021 [Page 9]