Network working group L. Yong Internet Draft L. Xia Category: Standard Track Huawei Q. Zu Ericsson Expires: December 2014 June 18, 2014 Network Virtualization Edge (NVE) draft-yong-nvo3-nve-04 Abstract This document specifies Network Virtualization Edge (NVE) data plane interoperability functionality for Network Virtualization Overlays (NVO3). These specifications are necessary for the interoperability between an NVE and its attached tenant systems and between the NVEs. Status of this Memo This Internet-Draft is submitted to IETF in full conformance with the provisions of BCP 78 and BCP 79. Internet-Drafts are working documents of the Internet Engineering Task Force (IETF), its areas, and its working groups. Note that other groups may also distribute working documents as Internet- Drafts. Internet-Drafts are draft documents valid for a maximum of six months and may be updated, replaced, or obsoleted by other documents at any time. It is inappropriate to use Internet-Drafts as reference material or to cite them other than as "work in progress." The list of current Internet-Drafts can be accessed at http://www.ietf.org/ietf/1id-abstracts.txt. The list of Internet-Draft Shadow Directories can be accessed at http://www.ietf.org/shadow.html. This Internet-Draft will expire on December 18, 2014. Yong, et al [Page 1] Internet-Draft Network Virtualization Edge (NVE) June 2014 Copyright Notice Copyright (c) 2013 IETF Trust and the persons identified as the document authors. All rights reserved. This document is subject to BCP 78 and the IETF Trust's Legal Provisions Relating to IETF Documents (http://trustee.ietf.org/license-info) in effect on the date of publication of this document. Please review these documents carefully, as they describe your rights and restrictions with respect to this document. Table of Contents 1. Introduction...................................................3 1.1. Conventions used in this document.........................3 1.2. Terminology...............................................3 2. NVE Design Principles..........................................3 3. Interconnecting Tenant Systems.................................4 3.1. Virtual Machines and Physical Servers.....................4 3.2. Network Service Appliances................................5 3.3. Gateways..................................................6 4. Network Virtualization Edge (NVE)..............................6 4.1. NVE Forwarding............................................6 4.1.1. L2 NVE...............................................6 4.1.2. L3 NVE...............................................8 4.1.3. L2/L3 NVE............................................9 4.2. Overlay Tunnel between NVEs...............................9 4.3. Multi-Tenancy Support....................................10 4.4. Distributed Gateway (dGW)................................10 4.5. Route Path Control.......................................13 4.6. Split-NVE................................................13 4.7. Multi-Homing Support.....................................14 4.8. OAM Tools on NVE.........................................15 5. Operation Considerations......................................16 5.1. VM Mobility..............................................16 5.2. Gateway vs. Distributed Gateway..........................16 6. Security Considerations.......................................18 7. Acknowledgements..............................................18 8. IANA Considerations...........................................19 9. References....................................................19 9.1. Normative References.....................................19 9.2. Informative References...................................19 Yong, et al [Page 2] Internet-Draft Network Virtualization Edge (NVE) June 2014 1. Introduction Network Virtualization Edge (NVE) is a component in Network Virtualization Overlays Technology. This component is described in the NVO3 framework [NVO3FRWK] and architecture [NV03ARCH]. This document specifies NVE data plane interoperate functionality. The functionality specifications are necessary for the interoperability between an NVE and its attached tenant systems and between the NVEs. The data plane functionality described in this document is independent of NVO3 control plane functionality. Thus, the control plane functionality is outside the scope of this document. However the specifications in this document can support any control plane implementation and are helpful in control plane protocol development. NVE data plane functionality essential is the packet forwarding. It receives a packet from a tenant system via a virtual access point (VAP), processes it, and sends it to the peer NVE via an overlay tunnel or forwards to a local VAP; it receives a packet from a peer NVE via an overlay tunnel, processes it, and sends it to a tenant system via a VAP. In the process, an NVE performs the table lookup, may modify the packet header and/or insert/remove the tunnel header on the packet prior to sending. The forwarding table is manipulated by the control plane. This document does not address the forwarding table update/lookup and does not specify tunnel encapsulation protocol but describe the usage at NVE. In order to make NVO3 data plane work properly, some configurations on NVEs are necessary. They can be done manually or automated. How these configurations are done is outside the scope of the document. 1.1. Conventions used in this document The key words "MUST", "MUST NOT", "REQUIRED", "SHALL", "SHALL NOT", "SHOULD", "SHOULD NOT", "RECOMMENDED", "MAY", and "OPTIONAL" in this document are to be interpreted as described in RFC-2119 [RFC2119]. 1.2. Terminology This document uses the terms defined in NVO3 framework [NVO3FRWK] and architecture [NVO3ARCH] documents. 2. NVE Design Principles NVE design principles are: Yong, et al [Page 3] Internet-Draft Network Virtualization Edge (NVE) June 2014 1. The solution supports multi-tenancy in a common underlying network. 2. The solution supports different types of tenant systems and requires no change on tenant system configuration and behavior. 3. The solution is agnostic to NVE location, i.e. regardless NVE is co-located with tenant systems on a server/device or physically separated from tenant systems. 4. No change on tenant system configuration and behavior whether a gateway or distributed gateway is used. 5. The solution must support tenant system, i.e. virtual machine (VM) mobility. 6. The solution must be scalable in supporting a VN having many NVEs each of which may have many attached tenant systems; and in supporting an NVE being the members of several VNs and having attached tenant systems that belong to the same or different VNs. Note that NVO3 architecture [NVO3ARCH] defines NVE and NVA entities; item 5 and 6 achievement depends on both NVE and NVA. This document only focuses on NVE data plane functionality. The interaction between NVE and NVA, between NVAs, and between hypervisor and NVE are outside the scope of the document. 3. Interconnecting Tenant Systems NVO3 provides network connectivity between the tenant systems locally attached to an NVE, and between local and remote tenant systems, i.e. on different NVE. NVE MUST be able to interwork with a tenant system. NVO3 architecture [NVO3ARCH] defines several types of tenant systems. Following sections describe these tenant systems in terms of their role and networking behavior. 3.1. Virtual Machines and Physical Servers Tenant system may be a virtual machine on a server or a physical server. For a virtual machine, Guest OS runs on the tenant system and application software runs on top of Guest OS. For a physical server, host OS runs on the server and application software runs on top of it. Here is the summary of such tenant system networking behavior: Yong, et al [Page 4] Internet-Draft Network Virtualization Edge (NVE) June 2014 . A tenant system (TS) is configured with a subnet (such as 10.1.1.0/24) and a default GW IP address, and TS MAC and IP address (manually or automatically e.g. [DHCP]). Note that TS IP address is an address in tenant subnet. How to assign and configure these addresses on Tenant System is outside the scope of document. . A tenant system learns the MAC address of the peer in the same subnet by using a protocol (e.g. ARP, NDP) and may learn the peer IP and MAC from the source address on the incoming packet as well. . A tenant system learns the GW MAC address from ARP or NDP protocol. The GW entity needs to support ARP or NDP protocol. . A tenant system may cache the interested destination IP and MAC address for the packet forwarding. . For intra subnet forwarding, a tenant system inserts the destination MAC address on the packet. . For inter subnet forwarding, a tenant system inserts the GW MAC address on the packet. . A tenant system may filter received packets and only accept the packet with the designation MAC address the same as its MAC address. NVE ability to support such host behavior will be described in section 4.1. 3.2. Network Service Appliances A network service appliance such as load balancer or firewall can act as a tenant system and provide a service to one or more VNs (via distinct VAPs). A network service appliance may be implemented on a physical device, a bare metal server, or a virtual machine on a server. A tenant system, as a network service appliance, may have different configuration and behavior as a host as described in section 3.1. Such tenant system attaches to an NVE via a VAP and acts as a middle box or service function in a VN. Typically, the configuration or policy on the VN determines which traffic or traffic flows in the VN is forwarded to this tenant system. In other words, the NVE may not forward the packets toward a tenant system based on the destination address on the packets. If a network service appliance provides a service for two VNs interconnection Yong, et al [Page 5] Internet-Draft Network Virtualization Edge (NVE) June 2014 such as GW, NAT. In this case, such tenant system may even modify the received packets header prior to send them. See next section. NVE ability to support such tenant system behavior will be described in section 4.5. 3.3. Gateways A gateway may be used to interconnect two VNs implemented by NVO3 (refer it as to NVO3 VN), between an NVO3 VN and other networks that may be virtual, physical, between an NVO3 VN and Internet, or a combination of these. A gateway may also interconnect two NVO3 VNs that are implemented with the same or different NVE service types. A gateway may be implemented on a physical network device, a physical server, or a VM. If it is a physical network device, the device often supports embedded NVE functions and acts as a network element in IGP. Note that a distributed gateway may be implemented for NVO3 VNs interworking. The distributed gateway means that a gateway function is implemented on NVEs so that the traffic between the VNs can be forwarded at the local NVE directly; as a result path optimization is gained. See section 4.4. It is often that a gateway integrates with several other network service appliances such as NAT, firewall and policy based forwarding for the interconnection need, which means that inter-VN traffic gets these special treatments. 4. Network Virtualization Edge (NVE) NVO3 framework [NVO3FRWK] defines three NVE service types: L2 service, L3 service, and L2/L3 service. A tenant network that is implemented by NVO3 can be implemented with one of NVE types or the combination w/ a gateway. Note the document uses ARP protocol to describe interoperability function between NVE and Tenant System. The use of NDP protocol is for next version. 4.1. NVE Forwarding 4.1.1. L2 NVE An L2 VN is implemented with L2 NVE service type and provides L2 broadcast domain to the tenant systems on the VN. The tenant system attaches to the NVE via a VLAN, directly attached port or virtual Yong, et al [Page 6] Internet-Draft Network Virtualization Edge (NVE) June 2014 port. For NVO3 architecture, upon receiving ARP request from a tenant system, an NVE should not forward the ARP request message. If the interested tenant system at a remote NVE or an associated VAP, the local NVE sends the ARP response with the requested MAC address back. An NVE can obtain the remote tenant system MAC address via NVA [NVO3ARCH]. The local tenant MAC is obtained from data plane learning, configuration or ARP announcement. If the NVE does not have the information about interested MAC in the receiving ARP request, it should query the NVA. If receiving an unknown MAC packet at VAP, NVE should change the packet to a known MAC packet prior to the forwarding. L2 NVE service supports a broadcast domain in the VN. Transporting tenant broadcast/multicast traffic among NVEs requires a way to map the VN broadcast/multicast traffic to an underlying IP multicast solution. IP network does not support a multicast solution yet and relies on Protocol Independent Multicast (PIM) to support multicast transport. An NVE can send the VN broadcast/multicast traffic to the remote NVEs by using unicast outer IP address on the packets, i.e. replicating in the underlying network. This method does not require the underlying network to support a multicast transport. How NVA conveys such mapping information to NVEs is outside the scope of the document. If underlying network supports PIM, mapping between a broadcast/multicast group in VN and an underlying IP multicast group need to be configured at NVE manually or automatically. In case that NVE is on a server, NVE uses IGMP [RFC3376] to join an IP multicast group in underlying IP network. Upon receiving mcast MAC packet, NVE encapsulates the packet and inserts underlying IP multicast address as outer IP address on the packet. To interconnect with external virtual or physical networks or an overlay or non-overlay virtual network, a gateway is necessary. The gateway as a tenant system attaches to an NVE and performs the traffic enforcement based on policy between an L2 VN and external networks. L2 NVE service may apply to non-IP and IP applications. However IP based application may be implemented in other ways.(see below) To use NVE data plane learns the mapping between remote tenant system MAC and remote NVE IP address without NVE not snooping and terminating ARP is for further evaluation. Yong, et al [Page 7] Internet-Draft Network Virtualization Edge (NVE) June 2014 4.1.2. L3 NVE An L3 VN is implemented with L3 NVE service type and provide L3 routing domain for the tenant systems in the VN. The VAP has an IP interface, attached port/vport, or a VLAN interface. An NVE acts as the first hop (or next hop if the TS has vR configured) router to the attached tenant systems. For the VLAN interface, i.e. Ethernet access interface, the NVE needs support ARP protocol, terminates all ARP request messages, and replies its MAC address to the tenant system. An NVE tracks tenant system IP and MAC address mapping if VAP is Ethernet interface unless the special configuration is done on the NVE.(see section 3.2). When a tenant system forwards packets to its attached NVE, the NVE receives either IP packets or Ethernet frames with NVE MAC address as the destination MAC on the packets depending on VAP type. The NVE performs an IP table lookup based on the destination IP address on the packets. If the packet needs to forward to another NVE, the NVE sends it via L3 overlay (NVE obtains the inner/outer address mapping from NVA). If the packet needs to be forwarded to a local VAP, and the VAP is Ethernet access, the NVE inserts the destination MAC address on the packet and its MAC address as source MAC prior to sending it to the tenant system; if the VAP is an IP interface, the NVE sends IP packet directly. NVE may learn local TS IP/MAC address via ARP or data plane learn or from NVA. The rule of thumb is that such method MUST not require any change on the tenant system side. The tenant systems connecting to an L3 VN can be on the same or different subnets. Typically, subnet or flow based policies may be configured for route constraint or route path control policies may be configured depending on the tenant network requirements. For example, one tenant system on an L3 VN may be an inter-subnet gateway. Packets from one subnet to another on the L3 VN may be sent to this gateway prior to reach the destination tenant system. If an L3 VN needs to interconnect with external networks, Internet, or another VN, a gateway MUST be used. To avoid address collision, either IP address space partition or IP address translation between two VNs at a gateway is used. The former, in turn, looks like one routing domain with one IP space shared between two virtual networks. Figure 1 illustrates an example that a gateway (GW) as a tenant system is used for interconnecting VNx, VNy, and external net. GW attaches to NVE2. VN interconnecting policy may be configured on GW. The distributed gateway may be used for VN interconnection, see section 4.4. Yong, et al [Page 8] Internet-Draft Network Virtualization Edge (NVE) June 2014 +-----------+ +----------+ | +------+ |-~-~-~-~-~-~-~-| +------+ | +----+ TS1---+-|L3VNIx+--{L3 Overlay(VNx)}-+L3VNIx|-+--+ | | +------+ |-~-~-~-~-~-~-~-| +------+ | | GW | | | | | | | | +------+ |-~-~-~-~-~-~-~-| +------+ | +--+-+ TS2---+-|L3VNIy+--{L3 Overlay(VNy)}-+L3VNIy|-+-/ | | +------+ |-~-~-~-~-~-~-~-| +------+ | ..+... +-----------+ +----------+ / \ NVE1 NVE2 | Ext. | | net. | \....../ Figure 1 Two VN interconnection via a GW Note: an L3 VN, as a route domain, does not support broadcast and multicast function. Applicability for MVPN [RFC4834] to NVO3 is for further study. It is obvious that L3 NVE service only supports IP based application. 4.1.3. L2/L3 NVE L2/L3 NVE service type is used when an NVE supports distributed L3 gateway function and multiple L2 VNs on the NVE are instantiated. See section 4.4. 4.2. Overlay Tunnel between NVEs NVE may implement L2 overlay or L3 overlay depending on NVE service types. A tunnel between two NVEs may be over one underlying network segment/domain or span across multiple network domains. Both NVEs need to use the same encapsulation protocol to encap./decap. packets to/from a tunnel in between. There are several encapsulation methods in the industry such as VXLAN [VXLAN], NVGRE [NVGRE]. If two NVEs do not support the same encapsulation method, an interworking gateway is needed for the encapsulation translation. An NVE may support more than one encapsulation method, in this case, two NVEs need to select the same encapsulation method. This can be done manually or via control plane negotiation. In NVO3 architecture, an NVE relies on the NVA to obtain the inner/outer address mapping and the underlying network supports the IP connectivity between two NVEs. How NVA obtains the mapping information is outside the scope of the document. The overlay mechanism requires NVE encapsulating the packets from tenant system, which adds overhead. To avoid packet fragmentation at Yong, et al [Page 9] Internet-Draft Network Virtualization Edge (NVE) June 2014 NVE, the tenant system packet MTU size MUST not exceed underlying MTU size minus overlay header bytes minus inner header bytes. NVE should drop the tenant packet when the packet size exceeds allowed MTU size and raise the alert. Tenant System MUST support MTU discovery.[RFC4821] The encapsulation process on an NVE also needs convey the packet characteristics in a VN to the underlying network, i.e. encode or translate the packet parameters in the inner header to the outer header, so that the packet can get the same treatment in the underlying network. Some examples are CoS value, and entropy information calculation. The detail will be updated in next version. 4.3. Multi-Tenancy Support It is very important for NVO3 solution to support multi-tenancy over a common physical infrastructure, and ensures independent address space in individual tenant networks that are not communicated directly, i.e. only communicate with address translation or via Internet, and traffic isolation among them. NVE MUST maintain separate forwarding tables to support address overlapping. Since a tenant network may have one virtual or multiple virtual networks, it is important for a tenant or DC operator to manage the address allocation for the virtual networks in a tenant network to avoid address collision. A tunnel between NVEs may carry the traffic belonging to different virtual networks. The VN ID in the overlay header serves the traffic segregation. 4.4. Distributed Gateway (dGW) Distributed Gateway function may be implemented on an NVE for inter- subnet, inter-VN gateway/forwarding for the local traffic. This will gain the path optimization, i.e. traffic in one VN can be routed to another VN at local NVE. Distributed Gateway implementation requires inter-VN forwarding policy to be configured at each NVE. To support TS mobility, all NVE use the same dGW address. Figure 2 illustrates one example using dGW. As shown, VNx and VNy present on NVE1 and NVE2; TS1 and TS3 connect to the L3 VNx and TS2 and TS4 to L3 VNy. The L3 overlay is used between the NVE1 and NVE2. Both NVE1 and NVE2 support distributed gateway (dGW). The packet from TS1 to TS2 will be processed at the dGW on NVE1. The packet from TS1 to TS4 will be processed at the dGW on NVE1 and forwarded over L3 Overlay (VNy) tunnel. The packet from TS1 to TS3 will be forwarded over L3 Overlay (VNx) tunnel without processing at any dGW. Yong, et al [Page 10] Internet-Draft Network Virtualization Edge (NVE) June 2014 +------------+ +-----------+ | +------+ |-~-~-~-~-~-~-~-~-~| +------+ | TS1---+-|L3VNIx+---{ L3 Overlay(VNx) }--+L3VNIx|-+---TS3 | +--+---+ |-~-~-~-~-~-~-~-~-~| +--+---+ | | +-+-+ | | +-+-+ | | |dGW| | | |dGW| | | +-+-+ | | +-+-+ | | +--+---+ |-~-~-~-~-~-~-~-~-~| +--+---+ | TS2---+ |L3VNIy+---{ L3 Overlay(VNy) }--+L3VNIy|-+---TS4 | +--+---+ |-~-~-~-~-~-~-~-~-~| +---+--+ | +------------+ +-----------+ NVE1 NVE2 Figure 2 L3 NVE Service w/dGW Model Figure 3 illustrates other two examples where L2/L3 NVE service type is used. Figure 3(a) shows two L2 VNs w/ distributed L3 gateway function on NVEs; Figure 3(b) shows L2 VNs and L3 VNs interconnected w/ distributed L3 gateway function on NVEs. In case (a), the VAP may be VLAN or port/vport based. Tenant systems on the same L2 VN or different L2 VNs may be on the same or different NVEs. The tenant systems on the same L2 VN are in a broadcast domain and can be communicated without a constraint. The implementation looks like L2 NVE described above. For the traffic across L2 VNs, i.e. from one L2 VN to another, the tenant system sends the packet with GW MAC address that is configured on the local NVE. The same MAC address for the same VN should be used on all NVEs at the ARP response to the TSs. (Use of different GW MAC address on NVEs is for further study). When an NVE receive a packet from a TS on a VN, say VNx, if the destination MAC on the packet is not L3dGW MAC address, NVE performs MAC table lookup and forward the packet to the tenant system on the same VN as described in section 4.1.1. If the destination MAC on the packet is L3dGW MAC address, NVE performs the IP table lookup and gets the destination VN, say VNy, and destination NVE. If the destination NVE is itself, the NVE sends the packet to the tenant system via the associated VAP; if the result is a remote NVE, the NVE encapsulates the packet (IP packet) with the destination VN ID prior sending to remote NVE. The remote NVE decapsulates the packet and performs IP lookup. For case (a); remote NVE inserts L3dGW MAC as the src MAC and found MAC address as dst MAC on the packet prior Yong, et al [Page 11] Internet-Draft Network Virtualization Edge (NVE) June 2014 sending to the tenant system. For case (b), remote NVE forward IP packet to tenant system directly. +----------+ +----------+ | +------+ |-~-~-~-~-~-~-~-~-~| +------+ | TSs---+-|L2VNIx+-{ L2 Overlay(VNx) }-+L2VNIx|-+---TSs | +---+--+ |-~-~-~-~-~-~-~-~-~| +---+--+ | | +--|--+ | | +--|--+ | | |L3dGW|-{L3 Overlay(VNx|VNz)}-|L3dGW| | | +--|--+ | | +--|--+ | | +---+--+ |-~-~-~-~-~-~-~-~-~| +----+-+ | TSs---+-|L2VNIz+-{ L2 Overlay(VNz) }-+L2VNIz| +---TSs | +------+ |-~-~-~-~-~-~-~-~-~| +------+ | +----------+ +----------+ NVE1 NVE2 (a) +----------+ +----------+ | +------+ |-~-~-~-~-~-~-~-~-~| +------+ | TSs---+-|L2VNIx+-{ L2 Overlay(VNx) }-+L2VNIx|-+---TSs | +---+--+ |-~-~-~-~-~-~-~-~-~| +--+---+ | | +--+--+ | | +-+---+ | | |L3dGW|-{L3 Overlay(VNx|VNz)}-|L3dGW| | | +--+--+ | | +-+---+ | | +---+--+ |-~-~-~-~-~-~-~-~-~| +--+---+ | TSs---+-|L3VNIz+-{ L3 Overlay(VNz) }-+L3VNIz| +---TSs | +------+ |-~-~-~-~-~-~-~-~-~| +------+ | +----------+ +----------+ NVE1 NVE2 (b) Figure 3 L3 distributed GW Examples A tenant network with multiple L2 VNs and L3 VNs interconnected w/distributed L3 gateway function MUST share the same IP and MAC address space. If the tenant network further interconnects with other tenant networks, external, or Internet via a gateway, either IP and/or MAC address partition or network address translation (NAT) MUST be used. Yong, et al [Page 12] Internet-Draft Network Virtualization Edge (NVE) June 2014 4.5. Route Path Control A tenant network may be implemented as a full mesh among NVEs and not have a policy on route path at all. As the result, single hop is between sender TS and destination TS. A tenant network may contain some tenant systems that are designated as a network service appliance. In the case, tenant may want some tenant traffic passing through some network service appliance prior to delivering to the destination tenant system and some are not. For example, Tenant network is implemented with three VNs in a DC. One L2 VN is for Web Tier, second L2 VN is for application tier; third L2 VN is for Database Tier. The policies are illustrated in the following figure. Traffic from the Web Tier to the App Tier MUST pass through tenant firewall on the Tenant System, say A; The traffic from App Tier to Web Tier can be directly routed; Traffic between App Tier and DB Tier MUST pass tenant firewall on tenant system, say B. No communication is allowed between the Web Tier and the DB Tier. Web Tier ----FW A----> App Tier ----FW B----> DB Tier <----------- <----FW B----- Figure 4 Polices on a Three-Tier Tenant Network In this case, An NVE attached by the firewall tenant system A is configured to forward all the traffic from Web Tier to the system A via a VAP. The system A processes the packets and may forward the packets to the App Tier via another VAP that is associated to App Tier on the NVE. The NVE forwards to the tenant systems in App Tier on behalf of the system A. The traffic from App Tier can be forwarded to the Web Tier directly. The NVE simply obtains the inner/outer address mapping and translate VN IDs on the packets prior to forwarding to the peer NVEs. Thus, in the pass-thru firewall case, the inner/outer address mapping that an NVE gets from NVA is not destination tenant address (inner)/its NVE address; the outer address is the address of the NVE which the system A attaches to. The NVO3 architecture, such route path control can be implemented in NVA, NVE, or both. 4.6. Split-NVE Split-NVE may be used in several use cases [NVO3ARCH]. Some NVE functions may reside on NVE spoke and some are on NVE hub. An overlay tunnel is used between NVE spoke and hub. One useful splitting structure in the data plane is to simplify the forwarding table on the NVE spoke, i.e. only maintains local forwarding entry; Yong, et al [Page 13] Internet-Draft Network Virtualization Edge (NVE) June 2014 and let the NVE hub maintain the complete forwarding table. An NVE spoke just sends the packets to the NVE hub if the receiving point is not at local. It is possible that an NVE hub does not have any direct attached TS but connects to many NVE spokes. In this case, all the NVE spokes and NVE hubs are the member of one VN. Another useful structure is Intra NVE and Inter NVE splitting. The Intra NVEs forwards the packets if the sender TS and receiver TS are in the same VN and forwards the packets to the Inter NVE if not. The Inter NVE forwards the packets between the different VNs. The Inter NVE is often seen as a gateway. Note that, this design may cause the packet hire-pinning if the sender and receiver TSs in two different VNs are on the same Intra NVE. See section 5.3. Split-NVE applies to all NVE service types. There is no configuration and behavior change between a TS and attaching NVE regardless if split-NVE is used or not. However, the network performance and the tenant network cost may differ. The splitting control plane functionality on an NVE is outside the scope of this document. 4.7. Multi-Homing Support Two multi-homing of NVEs scenarios are described in NVO3 architecture document [NVO3ARCH]. 1) One NVE may have more than one overlay paths in term of more than one reachable IP addresses. 2) When an NVE is physically separated from attached tenant systems, a tenant system may attach to more than one NVE via the VAPs. A design may use one of them or both together. For case 1), NVA may provide more than one inner/outer mapping to an NVE, the NVE may support some ECMP capability to distribute the traffic among the paths. For L2 NVE service, multi-homing may be configured with either active/active or active/standby to a tenant system in case 2). For active/active mode, the tenant system may distribute traffic per- flow or per-vlan. For L3 NVE service, NVEs are the first hop router for local tenant systems regardless of inter-subnet or intra-subnet traffic. The link/node redundancy mechanisms (e.g., ECMP, VRRP, etc) can provide various modes (i.e., active/active, active/standby) of multi-homing access for tenant systems. For L2/3 NVE service, the main extension to above 2 types is the internal distributed gateway function on NVEs. Since the gateway function is used for process the ingress traffic on individual NVEs, Yong, et al [Page 14] Internet-Draft Network Virtualization Edge (NVE) June 2014 multi-homing implementation is the same as of L2 NVE or L3 NVE service type. 4.8. OAM Tools on NVE It is necessary for an NVE to support some OAM tools. A tool can be turned on when a tunnel or VN is set up or dynamically turned on/off according to operation needs. The NVE implementation SHALL fulfill the OAM requirements described in [NVO3OAM]. Memo: followings are in considerations. This section is for the future study. OAM tools on NVE should be operated under the conditions: . Run various OAM tools along the same path as data frames of overlay network between a pair of NVEs . Run OAM tools between per-tenant NVEs to probe the status of tunnel or NVE entities; . Send fault notification from underlay network to overlay network for its fault handing and alarm suppression. NVE may support following tools but not limit to: . Connectivity Fault Detection: detect the tunnel connectivity fault between two or more NVEs that support the same virtual network; . Overlay Path Traceroute: trace the overlay path hops between two NVEs; . Underlying Path Traceroute: trace the underlay path hops between two NVEs; . Performance Monitoring: monitor various performance metrics such as packet loss, packet delay, packet delay variation, packet throughput, etc. . NVE Auto Discovery: dynamically discover other NVEs that support the same virtual network; . Send fault notification from underlay network to overlay network for its fault handing and alarm suppression. Yong, et al [Page 15] Internet-Draft Network Virtualization Edge (NVE) June 2014 5. Operation Considerations 5.1. VM Mobility VM Mobility provides some benefits for DC operator in term of resource optimizations and performance turning. If a tenant system on a VM runs guest OS and application software, it can be moved from one NVE to another NVE without the impact of the live application. When a VM moves, NVA will send new inner/outer mapping to NVEs. If the tenant systems running a network service appliances software, besides the mapping changes, some setting on old NVE also need to be configured on new NVE.(See section 4.5) A VN ID is used to segregate the traffic for different VNs in data plane, or say on "wire". NVE implementation may use a domain-wide global VN ID or egress NVE assigned local VN ID in the data plane. If use of local VN ID, when a VM moves from one NVE to another, a sender NVE not only has to obtain the new NVE address the VM moves to, i.e. the outer addresses, also has to obtain the new VN ID the new NVE allocating for the VN. In other words, the ingress NVE has to modify both overlay header and outer header when a VM moves from one NVE to another. If a domain-wide VN ID is used on NVEs, ingress NVE only need to modify the outer header when a VN moves. Although local allocated VN ID adds implementation complexity for VM mobility, it has advantage in associating VN ID with other context at egress NVE to facilitate egress NVE packet processing. Thus, either may be implemented for different use cases. 5.2. Gateway vs. Distributed Gateway A gateway is used, in general, to interconnect two networks. The gateway in NVO3 means that it interconnects a virtual network overlay with other networks. The other network can be a physical network, a virtual network, a virtual network overlay, or Internet, which are often called an external network. Distributed gateway is a gateway function that is implemented on the NVEs so the traffic between two tenant systems in different virtual networks can be routed on the local NVE directly. The main benefit to use the distributed gateway is path optimization. To interconnect two networks, a gateway may integrate with other network service appliances such as NAT, Firewall and handle policy enforcement. Note that a tenant often uses this as a rule in an application networking design. When implementing distributed gateway, that means that NVEs also need support such policy enforcement, which, sometimes, may become complex. Yong, et al [Page 16] Internet-Draft Network Virtualization Edge (NVE) June 2014 Should a tenant network uses a gateway or distributed gateway for two VN interconnection? Here are the general recommendations. . If a VN overlay interconnects to an external network that is a physical network, virtual network, or Internet, using a gateway is practical. In this case, it is easy to place a gateway on the traffic path. . If a VN overlay in a DC interconnects to another VN overlay in another DC, the inter-VN traffic will pass through DC GWs, i.e. traffic pattern is north-south, it is good to use a gateway. . If a VN overlay interconnects to anther VN overlay within a DC, i.e. traffic pattern is easy-west, and there is light policies for inter-VN traffic, using distributed gateway is better; otherwise use a gateway. A gateway and distributed gateway function on NVEs can further work together to provide the inter-VN connection in a tenant network. Figure 5 gives an example. A distributed GW is implemented on NVE1 for forwarding traffic between VNx and VNy on NVE1. The L2GW is used to interconnect VNx and L2 bridge network; VNx, VNy, and bridge network further connects to WAN network via DCGW. DCGW is member of VNx, VNy, and connect to WAN network. +----------+ +--------+ +--+. | +------+ |-~-~-~-~-~-~-~-~-| | / \ Physical TSs---+-|L2VNIx+-{ L2 Overlay(VNx) } L2GW |-|Bridge|-Servers | +--+---+ |-~-~-~-~-~-~-~-~-+----+---+ \ Net./ | | | {L2 Overlay} +---+ | +-+---+ |-~-~-~-~-~-~-~-~-+----+---+ .'^^.\ | |L3dGW+-{L3 Overlay(VNx/y)} DCGW +----->{ WAN } | +-+---+ |-~-~-~-~-~-~-~-~-+----+---+ .v.v./ | | | {L2 Overlay} | | | +----+-----+ | +--+---+ |-~-~-~-~-~-~-~-~-| +--+----+| TSs---+-|L2VNIy+-{ L2 Overlay(VNy) }-+L2VNIy |+--TSs | +--+---+ |-~-~-~-~-~-~-~-~-| +-------+| +----------+ +----------+ NVE1 NVE2 Figure 5 Example of Gateways and Distributed GW Usage Yong, et al [Page 17] Internet-Draft Network Virtualization Edge (NVE) June 2014 6. Security Considerations NVO3 networks may be deployed in various use cases [NVO3CASE]. Difference cases may have different level of security requirements for NVO3 networks. NVE is a key element for the security of NVO3 network. NVE should support the mutual or automated authentication with NVAs, other NVEs, and tenant systems, to guarantee its peer having valid identity and privilege to communicate. NVEs should also provide integrity, confidentiality, and origin Authentication protection for whether control or data traffic against the unsecure underlay network. A per-tunnel based signatures or digests may provide data origin authentication, non-repudiation, and integrity protection. In addition, an NVE itself need to tolerant the DoS attack. In the Split-NVE case, there are security risks that the NVE may be polluted by a compromised hypervisor with incorrect network updating information. However in this circumstance, the security damages can be limited to the hypervisor and the VNs attached to the compromised hypervisor. There are still ways to protect the attached NVE itself and mitigate the damages. When an NVE is in the hypervisor, there are additional security risks on the NVE if the hypervisor may be compromised. A compromised NVE may send data traffic of a VN which it is not supposed to send. It is very important for an NVE to prevent any security risk initiated from a compromised remote NVE. The NVE may use the inner- to-outer address mappings table to filter incoming data traffic to ensure the inner address sourced packet originated from a correct participating NVE address. If the tenant traffic privacy is a concern, cryptographic measures must be applied in addition. Confidentiality and integrity on the tenant data plane traffic could avoid the tenant traffic to be redirected, intercepted or modified by a compromised underlay network component. In additional, the NVE implementation shall fulfill the security requirements described in [nvo3-security-requirements]. 7. Acknowledgements Authors like to thank Qin Wu for the review and valuable comments. Yong, et al [Page 18] Internet-Draft Network Virtualization Edge (NVE) June 2014 8. IANA Considerations The document does not require any IANA action. 9. References 9.1. Normative References [RFC2119] Bradner, S., "Key words for use in RFCs to Indicate Requirement Levels", BCP 14, RFC2119, March 1997. [RFC4821] Mathis, M. and Heffner, J., "Packetization Layer Path MTU Discovery", RFC4821, March 2007 [RFC4834] Morin, T., "Requirements for Multicast in Layer 3 Provider-Provisioned Virtual Private Networks (PPVPNs)", RFC4834, April 2007 9.2. Informative References [NVO3ARCH] Black, D., Narten, T., et al, "An Architecture for Overlay Networks (NVO3)", draft-narten-nvo3-arch, work in progress. [NVO3CASE] Yong, L., et al, "Use Cases for DC Network Virtualization Overlays", draft-ietf-nvo3-use-case, work in progress [NVO3DPREQ] Bitar, N., et al, "NVO3 Data Plane Requirements", draft- ietf-nvo3-dataplane-requirements-01, work in progress [NVO3FRWK] LASSERRE, M., Motin, T., et al, "Framework for DC Network Virtualization", draft-ietf-nvo3-framework, work in progress. [NVO3OAM] Ashwood, P, et al, "NVO3 Operation Requirement", draft- ashwood-nvo3-operational-requirement, work in progress [NVGRE] Sridharan, M., et al, "NVGRE: Network Virtualization using Generic Routing Encapsulation", draft-sridharan-virtualization- nvgre, work in progress [VXLAN] Mahalingam, M., Dutt, D., etc, "VXLAN: A Framework for Overlaying Virtualized Layer 2 Networks over Layer 3 Networks", draft-mahalingam-dutt-dcops-vxlan, work in progress Authors' Addresses Yong, et al [Page 19] Internet-Draft Network Virtualization Edge (NVE) June 2014 Lucy Yong Huawei Technologies, USA Email: lucy.yong@huawei.com Frank Liang Xia Huawei Technologies Email: frank.xialiang@huawei.com Qiang Zu Ericsson Email: zu.qiang@ericsson.com Yong, et al [Page 20]