MPLS Working Group B. Mack-Crane Internet Draft L. Dunbar Intended status: Informational S. Hares Expires: April 2011 Huawei October 12, 2010 IPv6 Neighbor Discovery Scalability for Large Data Centers draft-mackcrane-armd-ipv6-nd-scaling-00.txt Status of this Memo This Internet-Draft is submitted to IETF in full conformance with the provisions of BCP 78 and BCP 79. Internet-Drafts are working documents of the Internet Engineering Task Force (IETF), its areas, and its working groups. Note that other groups may also distribute working documents as Internet- Drafts. Internet-Drafts are draft documents valid for a maximum of six months and may be updated, replaced, or obsoleted by other documents at any time. It is inappropriate to use Internet-Drafts as reference material or to cite them other than as "work in progress." The list of current Internet-Drafts can be accessed at http://www.ietf.org/ietf/1id-abstracts.txt The list of Internet-Draft Shadow Directories can be accessed at http://www.ietf.org/shadow.html This Internet-Draft will expire on April 12, 2011. Copyright Notice Copyright (c) 2010 IETF Trust and the persons identified as the document authors. All rights reserved. This document is subject to BCP 78 and the IETF Trust's Legal Provisions Relating to IETF Documents (http://trustee.ietf.org/license-info) in effect on the date of publication of this document. Please review these documents carefully, as they describe your rights and restrictions with respect to this document. Code Components extracted from this document must include Simplified BSD License text as described in Section 4.e of the Trust Legal Provisions and are provided without warranty as described in the BSD License. Dunbar Expires April 12, 2011 [Page 1] Internet-Draft IPv6 ND Scalability October 2010 Abstract Server virtualization allows one physical server to support many virtual machines (VMs) so that multiple hosts (20, 30, or hundreds) can be running from one physical platform. As virtual machines are introduced into a Data Center, the number of hosts within the data center can grow dramatically, which can have tremendous impact on the network and hosts. This document provides an analysis of the scalability of IPv6 Neighbor Discovery (RFC 4861) in data centers with a large number of virtual machines. Conventions used in this document The key words "MUST", "MUST NOT", "REQUIRED", "SHALL", "SHALL NOT", "SHOULD", "SHOULD NOT", "RECOMMENDED", "MAY", and "OPTIONAL" in this document are to be interpreted as described in RFC-2119 0. Table of Contents 1. Introduction................................................3 2. Network functions provided by IPv6 Neighbor Discovery.........3 3. Basic ND protocol message use................................4 3.1. Router Solicitation.....................................4 3.2. Router Advertisement....................................4 3.3. Neighbor Solicitation...................................5 3.4. Neighbor Advertisement..................................5 3.5. Redirect...............................................5 4. Some additional protocol activities..........................5 4.1. Duplicate Address Detection.............................5 4.2. Anycast and Proxy address resolution....................6 4.3. Neighbor unreachability detection.......................6 4.4. Host-based Load Spreading...............................6 4.5. Router-based Load Spreading.............................6 4.6. Holding packets while address resolution occurs..........7 5. Summary and conclusions......................................7 6. Manageability Considerations.................................7 7. Security Considerations......................................7 8. IANA Considerations.........................................8 9. Acknowledgments.............................................8 10. References.................................................8 Authors' Addresses.............................................8 Intellectual Property Statement.................................9 Disclaimer of Validity.........................................9 Mack-Crane Expires April 12, 2011 [Page 2] Internet-Draft IPv6 ND Scalability October 2010 1. Introduction Server virtualization allows the sharing of the underlying physical machine (server) resources among multiple virtual machines (VMs), each running its own operating system. While Server Virtualization is a great technology for flexible management of server resources, it does impose great challenges to networks which interconnect all the servers in data center(s). Large data centers may grow to support hundreds of thousands or even millions of hosts (VMs). Even though there may be enough link bandwidth to support the traffic volume from all those VMs, other issues associated with Layer 2, like frequent ARP broadcast by hosts, broadcast unknown, etc., can create problems for the network and hosts. This document presents an initial analysis of the scalability of IPv6 Neighbor Discovery (RFC 4861) protocols in the context of a large data center network. Two network cases are considered: 1) a single L2 VLAN connecting a very large number of hosts and a relatively small number of routers, and 2) a core VLAN connecting a large number of routers and few, if any, hosts. The analysis presented here is a rough assessment of which protocol behaviors should scale well and which may present some concern. It does not provide hard numbers and is not based on any measurements in live networks. 2. Network functions provided by IPv6 Neighbor Discovery The protocols described in RFC 4861 provide a variety of network functions used by IPv6 nodes to: - find routers and discover link and network parameters, - discover each other's presence, - determine each other's link-layer addresses, and - maintain information about the paths to active neighbors. These functions are accomplished using five ICMP messages: - Router Solicitation, - Router Advertisement, - Neighbor Solicitation, Mack-Crane Expires April 12, 2011 [Page 3] Internet-Draft IPv6 ND Scalability October 2010 - Neighbor Advertisement, and - Redirect. The first part of the analysis considers the basic ND protocol activities and how often each message is sent and to what L2 destination address to determine whether there is any concern that ND messages could take too much bandwidth or tax host processors with unnecessary work. The second part of the analysis considers whether there may be scalability concerns related to other protocol behaviors mentioned in RFC 4861 for ancillary purposes, for example duplicate address detection. 3. Basic ND protocol message use 3.1. Router Solicitation The Router Solicitation message is sent by nodes to discover routers on the LAN, effectively requesting routers to respond to the node with a Router Advertisement message. This message is sent to the all-routers multicast address and so is not seen by other hosts on the LAN. A Router Solicitation message is generally sent when a node is first attached to (or comes up on) the LAN. The frequency of these events should be low and so both the traffic and processing load for Router Solicitation messages is expected to be negligible. 3.2. Router Advertisement Router Advertisement messages are sent by routers periodically to the all-nodes multicast address to announce their presence on the LAN and advertise some link parameters. As long as there are not very many routers on the LAN this should not present much traffic or processing load. In the core case where the LAN is connecting many routers the traffic and processing load will increase with the number of routers and some measures may be needed to limit the traffic, either by reducing the transmission rate or disabling the protocol (if it is not needed in an all-router environment). Router Advertisement is also unicast to the requesting node in response to a Router Solicitation message and, as noted above, this should not present a significant load. Mack-Crane Expires April 12, 2011 [Page 4] Internet-Draft IPv6 ND Scalability October 2010 3.3. Neighbor Solicitation A Neighbor Solicitation message is sent by a node when that node has no (or a stale) cache entry for the L2 address for a particular next hop IPv6 address. This message is sent to a solicited-node multicast address which is manufactured from the next hop IPv6 address. A great advantage of using a solicited-node multicast address is that only the solicited neighbor node (or perhaps a very few more) will be subscribed to this address. Therefore the processing load for this message is restricted to a small number of nodes and is not likely to present a significant burden. In general the frequency of Neighbor Solicitation messages will be related to the number of each node's communicating peers on the LAN. Since this number is directly related to the amount of traffic the LAN must support for communications in general the fraction consumed by Neighbor Solicitation should be very small. 3.4. Neighbor Advertisement Neighbor Advertisement messages are sent in response to Neighbor Solicitation messages. They are unicast to the originator of the Neighbor Solicitation message and so the load presented in this case should, as with Neighbor Solicitation, be a small fraction of the traffic that must be supported on the LAN. Unsolicited Neighbor Advertisement messages may also be sent to the all-nodes multicast address; however, as this may be done when a nodes L2 address changes the frequency of these messages should be extremely low. 3.5. Redirect Redirect messages are sent by routers to nodes to change the next hop that node is using to reach a particular destination. Although the likelihood of redirect depends on the network topology and other factors, it is not expected to present a significant load on either the network or hosts. 4. Some additional protocol activities 4.1. Duplicate Address Detection Duplicate address detection as described in [ADDRCONF] involves sending a number of Neighbor Solicitation messages for the address to be checked (to that address's solicited-node multicast address). This is done before attempting to join the LAN using the address Mack-Crane Expires April 12, 2011 [Page 5] Internet-Draft IPv6 ND Scalability October 2010 being checked. Since this is an initialization procedure it is not expected to present a significant traffic or processing load during normal operation. It is also possible that address autoconfiguration will not be used in very large data centers. 4.2. Anycast and Proxy address resolution Address resolution for Anycast addresses or addresses for which nodes are acting as a Proxy may solicit multiple Neighbor Advertisement messages in response. In this case of Anycast addresses the responses are sent with random delay do that the requesting node does not see an unmanageable burst of responses. The response traffic in this case may be greater but not likely a problem, and the additional processing load is only on the requesting node (which is in control of the rate of solicitation). In a multi-site data center network it may be desirable to restrict the propagation of Anycast address resolution messages if it is desired that only responses local to the requesting node's site be delivered. 4.3. Neighbor unreachability detection Neighbor unreachability detection relies on hints from higher layers to determine whether or not a given neighbor is still reachable. In some cases when connectivity is suspect and no higher layer hints are available, a Neighbor Solicitation message may be used to verify continued connectivity. This is not expected to be a common occurrence between hosts or hosts and routers (since higher layer hints are most likely available). Between routers there may not be higher layer hints available but there are likely other means to detect connectivity to router peers across the LAN making use of Neighbor Solicitation messages unnecessary. 4.4. Host-based Load Spreading Host-based load spreading (e.g. RFC 4311) affects the selection of next hop router for particular packets. This may increase the number of routers a given host communicates with, but it is not expected to add significantly to neighbor discovery traffic or processing load. 4.5. Router-based Load Spreading Router-based load spreading (i.e. the use of a NULL SA in a Router Advertisement message) requires hosts to solicit a next hop router address. This increases the number of solicitations for router addresses, but this should not be significant if the number of Mack-Crane Expires April 12, 2011 [Page 6] Internet-Draft IPv6 ND Scalability October 2010 routers on the LAN is small. This mechanism may be inappropriate (and unneeded) in a core LAN interconnecting a large number of routers and therefore not a concern in that case either. 4.6. Holding packets while address resolution occurs In multi-site networks or virtualized networks in which the edge-to- edge delay may be increased over that in a normal (local) LAN, hold time for packets awaiting address resolution may increase significantly. This may be a concern depending on the percentage of packets that must wait for address resolution before being forwarded on the LAN. 5. Summary and conclusions The following summarizes the analysis presented: - IPv6 ND looks like it will scale well for the case of a large LAN with 1000s of hosts and a relatively small number of routers. - For the case of a core LAN connecting a large number of routers there are some ND protocol behaviors that may not scale well but these are either optional or not needed between routers (i.e., there are other mechanisms available to the routers to accomplish the same end). - Multi-site L2 networks may provide challenges for both holding time for packets while address resolution is carried out and address resolution for Anycast addresses (for example, if these are expected to select only local servers). - The impact of network virtualization (many VLANs and virtual routers) on platforms that support many virtual networks has not been analyzed and may present additional scaling challenges. 6. Manageability Considerations This document has no manageability considerations. 7. Security Considerations This document adds no security considerations since it does not define any new protocol behaviors. However, it may be worthwhile to consider whether or not the size of an L2 network (as discussed here) presents any new security challenges. No analysis in this area is provided in this draft. Mack-Crane Expires April 12, 2011 [Page 7] Internet-Draft IPv6 ND Scalability October 2010 8. IANA Considerations This document has no IANA considerations. 9. Acknowledgments This document was prepared using 2-Word-v2.0.template.dot. 10. References [RFC4861] Narten, T., Nordmark, E., Simpson, W., and H. Soliman, "Neighbor Discovery for IP version 6 (IPv6)", RFC 4861, September 2007. [RFC4862] Thomson, S., Narten, T., and T. Jinmei, "IPv6 Stateless Address Autoconfiguration", RFC 4862, September 2007. Authors' Addresses Ben Mackcrane Huawei Technologies 1700 Alma Drive, Suite 500 Plano, TX 75075, USA Phone: (630) 810 1132 Email: tmackcrane@huawei.com Linda Dunbar Huawei Technologies 1700 Alma Drive, Suite 500 Plano, TX 75075, USA Phone: (972) 543 5849 Email: ldunbar@huawei.com Mack-Crane Expires April 12, 2011 [Page 8] Internet-Draft IPv6 ND Scalability October 2010 Sue Hares Huawei Technologies 2330 Central Expressway, Santa Clara, CA 95050, USA Phone: Email: shares@huawei.com Intellectual Property Statement The IETF Trust takes no position regarding the validity or scope of any Intellectual Property Rights or other rights that might be claimed to pertain to the implementation or use of the technology described in any IETF Document or the extent to which any license under such rights might or might not be available; nor does it represent that it has made any independent effort to identify any such rights. Copies of Intellectual Property disclosures made to the IETF Secretariat and any assurances of licenses to be made available, or the result of an attempt made to obtain a general license or permission for the use of such proprietary rights by implementers or users of this specification can be obtained from the IETF on-line IPR repository at http://www.ietf.org/ipr The IETF invites any interested party to bring to its attention any copyrights, patents or patent applications, or other proprietary rights that may cover technology that may be required to implement any standard or specification contained in an IETF Document. Please address the information to the IETF at ietf-ipr@ietf.org. Disclaimer of Validity All IETF Documents and the information contained therein are provided on an "AS IS" basis and THE CONTRIBUTOR, THE ORGANIZATION HE/SHE REPRESENTS OR IS SPONSORED BY (IF ANY), THE INTERNET SOCIETY, THE IETF TRUST AND THE INTERNET ENGINEERING TASK FORCE DISCLAIM ALL WARRANTIES, EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO ANY WARRANTY THAT THE USE OF THE INFORMATION THEREIN WILL NOT INFRINGE ANY RIGHTS OR ANY IMPLIED WARRANTIES OF MERCHANTABILITY OR FITNESS FOR A PARTICULAR PURPOSE. Acknowledgment Funding for the RFC Editor function is currently provided by the Internet Society. Mack-Crane Expires April 12, 2011 [Page 9]