INTERNET-DRAFT Danny McPherson Verisign, Inc. Dave Oran Cisco Systems, Inc. Expires: October 2011 April 27, 2011 Intended Status: Informational Architectural Considerations of IP Anycast Status of this Memo This Internet-Draft is submitted in full conformance with the provisions of BCP 78 and BCP 79. Internet-Drafts are working documents of the Internet Engineering Task Force (IETF). Note that other groups may also distribute working documents as Internet-Drafts. The list of current Internet- Drafts is at http://datatracker.ietf.org/drafts/current/. Internet-Drafts are draft documents valid for a maximum of six months and may be updated, replaced, or obsoleted by other documents at any time. It is inappropriate to use Internet-Drafts as reference material or to cite them other than as "work in progress." Copyright Notice Copyright (c) 2011 IETF Trust and the persons identified as the document authors. All rights reserved. This document is subject to BCP 78 and the IETF Trust's Legal Provisions Relating to IETF Documents (http://trustee.ietf.org/license-info) in effect on the date of publication of this document. Please review these documents carefully, as they describe your rights and restrictions with respect to this document. Code Components extracted from this document must include Simplified BSD License text as described in Section 4.e of the Trust Legal Provisions and are provided without warranty as described in the Simplified BSD License. McPherson, Oran [Page 1] INTERNET-DRAFT Expires: October 2011 April 2011 Abstract This memo discusses architectural implications of IP anycast, and provides some historical analysis of anycast use by various IETF protocols. McPherson, Oran [Page 2] INTERNET-DRAFT Expires: October 2011 April 2011 Table of Contents 1. Specification of Requirements. . . . . . . . . . . . . . . . . 4 2. Overview . . . . . . . . . . . . . . . . . . . . . . . . . . . 4 3. Anycast History. . . . . . . . . . . . . . . . . . . . . . . . 4 4. Use of Anycast in RFCs . . . . . . . . . . . . . . . . . . . . 5 5. Anycast in IPv6. . . . . . . . . . . . . . . . . . . . . . . . 6 6. DNS Anycast. . . . . . . . . . . . . . . . . . . . . . . . . . 7 7. BCP 126 Revisited. . . . . . . . . . . . . . . . . . . . . . . 8 8. Layering and Resiliency. . . . . . . . . . . . . . . . . . . . 9 9. Anycast Addresses as Destinations. . . . . . . . . . . . . . . 9 10. Anycast Addresses as Sources. . . . . . . . . . . . . . . . . 10 11. Regarding Widespread Anycast Use. . . . . . . . . . . . . . . 10 12. Service Discovery . . . . . . . . . . . . . . . . . . . . . . 10 13. Middleboxes and Anycast . . . . . . . . . . . . . . . . . . . 11 14. Transport Implications. . . . . . . . . . . . . . . . . . . . 11 15. Security Considerations . . . . . . . . . . . . . . . . . . . 12 16. Deployment Considerations . . . . . . . . . . . . . . . . . . 13 17. IANA Considerations . . . . . . . . . . . . . . . . . . . . . 13 18. Acknowledgments . . . . . . . . . . . . . . . . . . . . . . . 14 19. References. . . . . . . . . . . . . . . . . . . . . . . . . . 15 19.1. Normative References . . . . . . . . . . . . . . . . . . . 15 19.2. Informative References . . . . . . . . . . . . . . . . . . 15 20. Authors' Addresses. . . . . . . . . . . . . . . . . . . . . . 17 21. Appendix A: IAB Members . . . . . . . . . . . . . . . . . . . 17 McPherson, Oran [Page 3] INTERNET-DRAFT Expires: October 2011 April 2011 1. Specification of Requirements The key words "MUST", "MUST NOT", "REQUIRED", "SHALL", "SHALL NOT", "SHOULD", "SHOULD NOT", "RECOMMENDED", "MAY", and "OPTIONAL" in this document are to be interpreted as described in [RFC 2119]. 2. Overview IP anycast is used for at least one critical Internet service, that of the Domain Name System [RFC 1035] root servers. As of early 2009, at least 10 of the 13 root name servers were using IP anycast [RSSAC 29]. Use of IP anycast is growing for other applications as well. It has been deployed for over a decade for DNS resolution services and is currently used by several DNS Top Level Domain (TLD) operators. IP anycast is also used for other services in operational environments, to include Network Time Protocol (NTP) [RFC 1305]. Anycast addresses are syntactically indistinguishable from unicast addresses. Allocation of anycast addresses typically follows a model similar to that of unicast allocation policies. Anycast addressing is largely equivalent to that of unicast in multiple locations, and leverages unicast destination routing to deliver a packet to either zero or one interface among the interfaces asserting the address. The expectation of delivery is to the "closest" instance as determined by unicast routing topology metric(s). There is also an expectation of load-balancing that exists among equal cost routes. Unlike IP unicast, it is not considered an error to assert the same anycast address on multiple interfaces within the same or multiple systems. Some consider anycast a "deceptively simple idea". That is, many pitfalls and subtleties exist with application and transport, as well as for routing configuration and operation, when IP anycast is employed. In this document, we aim to capture many of the architectural implications of IP anycast. 3. Anycast History As of this writing, the term "anycast" appears in 126 RFCs, and ~360 Internet-Drafts (since 2006). McPherson, Oran Section 3. [Page 4] INTERNET-DRAFT Expires: October 2011 April 2011 The first formal specification of anycast was provided in "Host Anycasting Service" [RFC 1546]. The authors of this document did a good job of capturing most of the issues that exist with IP anycast today. One of the first documented uses of anycast was in 1994 for a "Video Registry" experiment [IMR 9401]. In the experiment, a UDP query was transmitted to an anycast address to locate a server, and TCP was used by the client to query the server, and then multicast was used to distribute the server database. There is also discussion that ISPs began using anycast for DNS resolution services around the same time, although no public references to support this are available. In 1997 the IAB clarified that IPv4 anycast addresses were pure "locators", and could never serve as an "identifier" (of a host, an interface, or anything else) [RFC 2101]. In 1998 the IAB conducted a routing workshop [RFC 2902]. Of the conclusions and output action items from the report, an Anycast section is contained in S 2.10.3. Specifically called out in the conclusions section is the need to describe the advantages and disadvantages of anycast, and the belief that local-scoped well-known anycast addresses will be useful to some applications. In the subsequent section, an action item was outlined that suggested a BOF should be held to plan work on progress, and if a working group forms, a paper on the advantages and the disadvantages of anycast should be included as part of the charter. In 1999, an Anycast BOF was then held at IETF 46. A number of uses were discussed. No firm conclusion was reached regarding use of TCP, but was useful for DNS, although costly. The use of global anycast was not expected to scale and hence was expected to be limited to a small number of key uses. In 2001, the Multicast and Anycast Group Membership (MAGMA) WG was chartered to address the initial authentication and access control issues associated with anycast group membership, but other aspects of anycast, including architecture and routing, were outside the group's scope. 4. Use of Anycast in RFCs SNTPv4 [RFC 2030] defined how to use anycast for server discovery. This was extended in [RFC 4330] as an NTP-specific "manycast" service, in which anycast was used for the discovery part. McPherson, Oran Section 4. [Page 5] INTERNET-DRAFT Expires: October 2011 April 2011 IPv6 defined some reserved subnet anycast addresses [RFC 2526] and assigned one to "Mobile IPv6 Home-Agents" [RFC 3775]. The original IPv6 transition mechanism [RFC 2893] made use of IPv4 anycast addresses as tunnel endpoints for IPv6 encapsulated in IPv4, but this was later removed [RFC 4213]. Carpenter's Relay Router [RFC 3056] scheme was augmented by a 6to4 relay anycast prefix [RFC 3068] aiming to simplify the configuration of 6to4 routers. Incidentally, 6to4 deployment has shown a fair number of operational and security issues [RFC 3964] that result from using anycast as a discovery mechanism. Specifically, one inference is that operational consideration is needed to ensure that anycast addresses get advertised and/or filtered in a way that produces intended scope (e.g., only advertise a route for your 6to4 relay to ASes that conform to your own acceptable usage policy), an attribute that can easily become quite resource (e.g., manpower) intensive. DNS use of anycast was first specified in "Distributing Authoritative Name Servers via Shared Unicast Addresses" [RFC 3258]. It is noteable that it used the term "shared unicast address" rather than "anycast address" for the service. Anycast was used for routing to rendezvous points (RPs) for MSDP and PIM [RFC 4610]. "Operation of Anycast Services" [RFC 4786] deals with how the routing system interacts with anycast services, and the operation of anycast services. "Requirements for a Mechanism Identifying a Name Server Instance" [RFC 4892] cites the use of anycast with DNS as a motivation to identify individual nameserver instances, and the NSID option was defined for this purpose [RFC 5001]. "Reflections on Internet Transparency" [RFC 4924] briefly mentions how violating transparency can also damage global services that use anycast. 5. Anycast in IPv6 Originally, the IPv6 addressing architecture [RFC 1884] [RFC 2373] [RFC 3513] severely restricted the use of anycast addresses. In particular, they provided that anycast addresses MUST NOT be used as a source address, and MUST NOT be assigned to an IPv6 host (i.e., only routers). These restrictions were later lifted in 2006 [RFC McPherson, Oran Section 5. [Page 6] INTERNET-DRAFT Expires: October 2011 April 2011 4291]. In fact, the recent "IPv6 Transition/Co-existence Security Considerations" [RFC 4942] overview now recommends: "To avoid exposing knowledge about the internal structure of the network, it is recommended that anycast servers now take advantage of the ability to return responses with the anycast address as the source address if possible." 6. DNS Anycast "Distributed Authoritative Name Servers via Shared Unicast Addresses" [RFC 3258] described how to reach authoritative name servers using anycast. It made some interesting points, for example this text from Section 2.3: "This document presumes that the usual DNS failover methods are the only ones used to ensure reachability of the data for clients. It does not advise that the routes be withdrawn in the case of failure; it advises instead that the DNS process shutdown so that servers on other addresses are queried. This recommendation reflects a choice between performance and operational complexity. While it would be possible to have some process withdraw the route for a specific server instance when it is not available, there is considerable operational complexity involved in ensuring that this occurs reliably. Given the existing DNS failover methods, the marginal improvement in performance will not be sufficient to justify the additional complexity for most uses." Other assertions included: o it asserted (as an advantage) that no routing changes were needed o it recommended stopping DNS processes, rather than withdrawing routes, to deal with failures, data synchronization issues, and fail-over, as provided in the quoted text above. o it argued that failure modes involving state were not serious, because: - the vast majority of DNS queries are UDP McPherson, Oran Section 6. [Page 7] INTERNET-DRAFT Expires: October 2011 April 2011 - large routing metric disparity among authoritative server instances would localize queries to a single instance for most clients - when the resolver tries TCP and it breaks, the resolver will move to a different server instance (where presumably it doesn't break) "Unique Per-Node Origin ASNs for Globally Anycasted Services" [UNIQUE_ORIGIN_AS] makes recommendations regarding the use of per- node unique origin ASNs for globally anycasted critical infrastructure services in order to provide routing system discriminators for a given anycasted prefix. The object was to allow network management and monitoring techniques, or other operational mechanisms to employ this new origin AS as a discriminator in whatever manner fits their operating environment, either for detection or policy associated with a given anycasted node. 7. BCP 126 Revisited "Operation of Anycast Services" (BCP 126) [RFC 4786] was a product of the IETF's GROW working group. The primary design constraint considered was that routing "be stable" for significantly longer than a "transaction time", where "transaction time" is loosely defined as "a single interaction between a single client and a single server". It takes no position on what applications are suitable candidates for anycast usage. Furthermore, it views anycast service disruptions as an operational problem, "Operators should be aware that, especially for long running flows, there are potential failure modes using anycast that are more complex than a simple 'destination unreachable' failure using unicast." The document primary deals with global Internet-wide services provided by anycast. Where internal topology issues are discussed they're mostly regarding routing implications, not application design implications. BCP 126 also views networks employing per-packet load balancing on equal cost paths as "pathological". McPherson, Oran Section 7. [Page 8] INTERNET-DRAFT Expires: October 2011 April 2011 8. Layering and Resiliency Preserving the integrity of a modular layered design for IP protocols on the Internet is critical to its continued success and flexibility. One such consideration is that of whether an application should have to adapt to changes in the routing system. Higher layer protocols should make minimal assumptions about lower layer protocols. E.g., applications should make minimal assumptions about routing stability, just as they should make minimal assumptions about congestion and packet loss. When designing applications, it would perhaps be safe to assume that the routing system may deliver each packet to a different service instance, in any pattern, with termporal re-ordering being a not-so-rare phenomenon. Stateful transport protocols (TCP, DCCP, SCTP), without modification, do not understand the properties of anycast and hence will fail probabilistically, but possibly catastrophically, when using anycast addresses in the presence of "normal" routing dynamics. 9. Anycast Addresses as Destinations Anycast addresses are "safe" to use as destination addresses for an application if: o A request message or "one shot" message is self-contained in a single transport packet o A stateless transport (e.g., UDP) is used for the above o Replies are always sent to a unicast address; these can be multi-packet since the unicast destination is "stable" * Note: this constrains the use of anycast as source addresses in request messages, since reply messages sent back to that address may reach a device that was not the source that initially triggered it. o The server side of the application keeps no hard state across requests o Retries are idempotent; in addition to not assuming server state, they do not encode any assumptions about loss of requests versus loss of replies. McPherson, Oran Section 9. [Page 9] INTERNET-DRAFT Expires: October 2011 April 2011 10. Anycast Addresses as Sources Anycast addresses are "safe" to use as source addresses for an application if: o No reflexive (response) message is generated by the receiver with the anycast source used as a destination unless the application has some private state synchronization that allows for the reflexive message arriving at a different instance. o The source anycast address is a bona fide interface address if reverse path forwarding (RPF) checking is on, or a service address explicitly provisioned to bypass RPF. 11. Regarding Widespread Anycast Use Widespread use of anycast for global Internet-wide services or inter- domain services has some scaling challenges. Similar in ways to multicast, each service generates at least one unique route in the global BGP routing system. As a result, additional anycast instances result in additional paths for a given prefix, which scales super- linearly as a function of denseness of inter-domain interconnection within the routing system (i.e., more paths result in more resources, more network interconnections result in more paths). This is why the Anycast BOF concluded that the use of global anycast addresses was not expected to scale and hence was expected to be limited to a small number of key uses." 12. Service Discovery Applications able to tolerate an extra round trip time (RTT) to learn a unicast destination address for multi-packet exchanges can safely use anycast destination addresses for service instance discovery. For example, "instance discovery" messages are sent to an anycast destination address. A reply is then sent from the unicast source address of the interface that received the discovery message, or a reply is sent from anycast source address of the interface that received the message, containing the unicast address to be used to invoke the service, which will avoid potential NAT binding issues for the relay packet. Subsequent exchanges use the unicast address McPherson, Oran Section 12. [Page 10] INTERNET-DRAFT Expires: October 2011 April 2011 provided. 13. Middleboxes and Anycast Middleboxes (e.g., NATs, firewalls) may cause problems when used in conjunction with anycast. In particular, a switch from anycast to unicast may require a new binding, and this may not exist in the middlebox. 14. Transport Implications UDP is the "lingua franca" for anycast today. Stateful transports could be enhanced to be more anycast friendly. It seems as though this was anticipated in Host Anycasting Services [RFC 1546], specifically: "The solution to this problem is to only permit anycast addresses as the remote address of a TCP SYN segment (without the ACK bit set). A TCP can then initiate a connection to an anycast address. When the SYN-ACK is sent back by the host that received the anycast segment, the initiating TCP should replace the anycast address of its peer, with the address of the host returning the SYN-ACK. (The initiating TCP can recognize the connection for which the SYN-ACK is destined by treating the anycast address as a wildcard address, which matches any incoming SYN-ACK segment with the correct destination port and address and source port, provided the SYN- ACK's full address, including source address, does not match another connection and the sequence numbers in the SYN-ACK are correct.) This approach ensures that a TCP, after receiving the SYN-ACK is always communicating with only one host." Multi-address transports (e.g., SCTP) might be more amenable to such extensions than TCP. Some similarities exist between what is needed for anycast and what is needed for address discovery when doing multi-homing in the transport layer. McPherson, Oran Section 14. [Page 11] INTERNET-DRAFT Expires: October 2011 April 2011 15. Security Considerations Anycast is often employed to mitigate or at least localize the effects of distributed denial of service (DDOS) attacks. For example, with the Netgear NTP fiasco [RFC 4085] anycast was used in a distributed sinkhole model to mitigate the effects of embedded globally-routed Internet addresses in network elements. "Internet Denial-of-Service Considerations" [RFC 4732] notes: that "A number of the root nameservers have since been replicated using anycast to further improve their resistance to DoS". "Operation of Anycast Services" [RFC 4786] cites DoS mitigation, constraining DoS to localized regions, and identifying attack sources using spoofed addresses as some motivations to deploy services using anycast. Multiple anycast service instances such as those used by the root name servers also add resiliency when network partitioning occurs (e.g., as the result of transoceanic fiber cuts or natural disasters). It should be noted that there is a significant man in the middle (MITM) exposure in either variant of anycast discovery (see Section 12: "Service Discovery") so the need for server authentication should be considered. Furthermore, as discussed in Section 4, operational consideration needs to be given to ensure that anycast addresses get advertised and/or filtered in a way that produces intended scope (for example, only advertise a route to your 6to4 relay to ASes that conform to your own AUP). This seems to be manpower intensive, and is often vulnerable to errors outside of the local routing domain, in particuar when anycasted services are deployed with the intent to scope associated announcements within some local or regional boundary. As previously discussed, [UNIQUE_ORIGIN_AS] makes recommendations regarding the use of per-node unique origin ASNs for globally anycasted critical infrastructure services in order to provide routing system discriminators for a given anycasted prefix. Network management and monitoring techniques, or other operational mechanisms may then employ this new discriminator in whatever manner fits their operating environment, either for detection or policy associated with a given anycasted node. Unlike multicast (but like unicast), anycast allows traffic stealing. Unlike unicast (but like multicast), the desire is to allow applications to cause route injection (either directly or as a side McPherson, Oran Section 15. [Page 12] INTERNET-DRAFT Expires: October 2011 April 2011 effect of doing something else). This combination is unique to anycast and presents new security concerns which are why MAGMA only got so far. The security concerns include: 1) Allowing route injection can cause DOS to a legitimate address owner. 2) Allowing route injection consumes routing resources and can hence cause DOS to the routing system and legit communications as a result. These are 2 of the core issues that were part of the discussion during [RFC 1884], the Anycast BOF, and the MAGMA chartering. Additional security considerations are scattered throughout the list of references provided herein. 16. Deployment Considerations This document covers issues associated with the architectural implications of anycast. Operators should heed these considerations when evaluating the use of anycast in their specific environments. 17. IANA Considerations No IANA actions are required. McPherson, Oran Section 17. [Page 13] INTERNET-DRAFT Expires: October 2011 April 2011 18. Acknowledgments Many thanks for Dave Thaler and Kurtis Lindqvist for their early review and feedback on this document. Thanks to both Brian Carpenter and Alfred Hoenes for their usual careful review and feedback as well. As for others, well, YOUR name could be here.. McPherson, Oran Section 18. [Page 14] INTERNET-DRAFT Expires: October 2011 April 2011 19. References 19.1. Normative References 19.2. Informative References [IMR 9401] "INTERNET MONTHLY REPORT", January 1994, http://mirror.facebook.com/rfc/museum/imr/imr9401.txt [RSSAC 29] "RSSAC 29 Meeting Minutes", December 2, 2007, http://www.rssac.org/meetings/04-08/rssac29.pdf [UNIQUE_ORIGIN_AS] McPherson, D. et al., "Unique Per-Node Origin ASNs for Globally Anycasted Services", April 2011, Internet-Draft, Work in Progress. [RFC 1035] Mockapetris, P., "DOMAIN NAMES - IMPLEMENTATION AND SPECIFICATION", RFC 1035, November 1987. [RFC 1305] Mills, D., "Network Time Protocol (Version 3) Specification, Implementation and Analysis", RFC 1305, March 1992. [RFC 1546] Partridge, C., Mendez, T., Milliken, W., "Host Anycasting Service", RFC 1546, November 1993. [RFC 1884] Hinden, R., Deering, S., "IP Version 6 Addressing Architecture", RFC 1884, December 1995. [RFC 2030] Mills, D., "Simple Network Time Protocol (SNTP) Version 4 for IPv4, IPv6 and OSI", RFC 2030, October 1996. [RFC 2101] Carpenter, B., Crowcroft, J., Rekhter, Y., "IPv4 Address Behaviour Today", RFC 2101, February 1997. [RFC 2119] Bradner, S., "Key words for use in RFCs to Indicate Requirement Levels", RFC 2119, March 1997. [RFC 2373] Hinden, R., Deering, S., "IP Version 6 Addressing McPherson, Oran Section 19.2. [Page 15] INTERNET-DRAFT Expires: October 2011 April 2011 Architecture", RFC 2373, July 1998. [RFC 2526] Johnson, D., Deering, S., "Reserved IPv6 Subnet Anycast Addresses", RFC 2526, March 1999. [RFC 2893] Gilligan, R., Nordmark, E., "Transition Mechanisms for IPv6 Hosts and Routers", RFC 2893, August 2000. [RFC 2902] Deering, S., Hares, S., Perkins, C., Perlman, R., "Overview of the 1998 IAB Routing Workshop", RFC 2902, August 2000. [RFC 3056] Carpenter, B., "Connection of IPv6 Domains via IPv4 Clouds", RFC 3056, February 2001. [RFC 3068] Huitema, C., "An Anycast Prefix for 6to4 Relay Routers", RFC 3068, June 2001. [RFC 3258] Hardie, R., "Distributing Authoritative Name Servers via Shared Unicast Addresses", RFC 3258, April 2002. [RFC 3513] Hinden, R., Deering, S., "Internet Protocol Version 6 (IPv6) Addressing Architecture", RFC 3513, April 2003. [RFC 3775] Johnson, D., Perkins, C., Arkko, J., "Mobility Support in IPv6", RFC 3775, June 2004. [RFC 3964] Savola, P., "Security Considerations for 6to4", RFC 3964, December 2004. [RFC 4085] Plonka, D., "Embedding Globally-Routable Internet Addresses Considered Harmful", RFC 4085, June 2005. [RFC 4213] Normark, E., Gilligan, R., "Basic Transition Mechanisms for IPv6 Hosts and Routers", RFC 4213, October 2005. [RFC 4291] Hinden, R., Deering, S., "IP Version 6 Addressing Architecture", RFC 4291, February 2006. [RFC 4330] Mills, D., "Simple Network Time Protocol (SNTP) Version 4 for IPv4, IPv6 and OSI", RFC 4330, January 2006. [RFC 4610] Farinacci, D., Cai, Y., "Anycast-RP Using Protocol Independent Multicast (PIM)", RFC 4610, August 2006. McPherson, Oran Section 19.2. [Page 16] INTERNET-DRAFT Expires: October 2011 April 2011 [RFC 4732] Handley, M., Rescorla, E., IAB, "Internet Denial-of- Service Considerations", RFC 4732, November 2006. [RFC 4786] Abley, J., Lindqvist, K., "Operation of Anycast Services", RFC 4786, December 2006. [RFC 4892] Woolf, S., Conrad, D., "Requirements for a Mechanism Identifying a Name Server Instance", RFC 4892, June 2007. [RFC 4924] Aboba, B., Davies, E., " Reflections on Internet Transparency", RFC 4924, July 2007. [RFC 4942] Davies, E., Krishnan, S., Savola, P., "IPv6 Transition/Coexistence Security Considerations", RFC 4942, September 2007. [RFC 5001] Austein, R., "DNS Name Server Identifier (NSID) Option", RFC 5001, August 2007. 20. Authors' Addresses Danny McPherson Verisign, Inc. Email: dmcpherson@verisign.com Dave Oran Cisco Systems Email: oran@cisco.com 21. Appendix A: IAB Members Internet Architecture Board Members at the time this document was published were: Bernard Aboba Marcelo Bagnulo Ross Callon Spencer Dawkins McPherson, Oran Section 21. [Page 17] INTERNET-DRAFT Expires: October 2011 April 2011 Vijay Gill Russ Housley John Klensin Olaf Kolkman Danny McPherson Jon Peterson Andrei Robachevsky Dave Thaler Hannes Tschofenig McPherson, Oran Section 21. [Page 18]