Lasserre et al [Page 1] Internet Draft Document Marc Lasserre Layer 2 VPN Working Group Xipeng Xiao draft-lasserre-l2vpn-vpls-ldp-applic-02.txt Riverstone Networks Yetik Serbest Cesar Garrido SBC Telefonica Marc Rapoport Completel Expires: Feb. 2005 Aug. 2004 VPLS Applicability draft-lasserre-l2vpn-vpls-ldp-applic-02.txt Status of this Memo This document is an Internet-Draft and is in full conformance with all provisions of Section 10 of RFC2026. Internet-Drafts are working documents of the Internet Engineering Task Force (IETF), its areas, and its working groups. Note that other groups may also distribute working documents as Internet- Drafts. Internet-Drafts are draft documents valid for a maximum of six months and may be updated, replaced, or obsoleted by other documents at any time. It is inappropriate to use Internet-Drafts as reference material or to cite them other than as "work in progress." The list of current Internet-Drafts can be accessed at http://www.ietf.org/ietf/1id-abstracts.txt The list of Internet-Draft Shadow Directories can be accessed at http://www.ietf.org/shadow.html. Abstract Virtual Private LAN Service (VPLS) is a layer 2 VPN service that provides multipoint connectivity in the form of an Ethernet emulated LAN, while usual L2 VPN services are typically point-to-point. Such draft-lasserre-l2vpn-vpls-ldp-applic-02.txt Aug 2004 Lasserre et al [Page 2] emulated LANs can span across metropolitan area networks as well as wide area networks. [VPLS-LDP] defines a method for signaling MPLS connections between member PEs of a VPN and a method for forwarding Ethernet frames over such connections. This document describes the applicability of such procedures to provide VPLS services. This document also compares the characteristics of this solution against the requirements specified in [L2VPN-REQ]. In summary, there are no architectural limitations to prevent the requirements from being met. But meeting certain requirements (e.g. QoS) is beyond the specification of [VPLS-LDP], and requires careful planning and precise implementation of the Service Provider (SP) networks. This document attempts to capture such issues, presents the potential solutions to these issues, and discusses the pros and cons of each alternative. This document does not cover the applicability of [VPLS-BGP]. Conventions The key words "MUST", "MUST NOT", "REQUIRED", "SHALL", "SHALL NOT", "SHOULD", "SHOULD NOT", "RECOMMENDED", "MAY", and "OPTIONAL" in this document are to be interpreted as described in RFC 2119 RELATED DOCUMENTS www.ietf.org/internet-drafts/draft-ietf-l2vpn-vpls-ldp-04.txt www.ietf.org/internet-drafts/draft-ietf-l3vpn-applicability- guidelines-00.txt Table of Contents 1. VPLS Overview........................................4 2. Operation of Control and Data Planes.................5 2.1. Control Plane.....................................5 2.1.1. Signaling......................................5 2.2. Data Plane........................................5 2.2.1. Ingress Processing.............................5 2.2.2. Egress Processing..............................6 2.2.3. Intermediate Node Processing...................6 3. VPLS vs. Alternative Approaches......................6 3.1. Ethernet Switching................................6 3.2. BGP/MPLS IP VPN...................................7 4. Provisioning.........................................7 4.1. PE Auto-Discovery.................................7 draft-lasserre-l2vpn-vpls-ldp-applic-02.txt Aug 2004 Lasserre et al [Page 3] 4.2. Operations and Maintenance........................8 5. Migration Impacts....................................8 5.1. Interconnecting Existing L2 Ethernet Islands with a VPLS Core..............................................9 5.2. Migrating an Existing L2 Ethernet Core to a VPLS Core 10 5.3. Interconnecting a new VPLS Network with Existing ATM/FR Networks.......................................11 5.4. Adding VPLS Support to an IP Routed Network......11 6. Multi-homing........................................11 7. Loop Prevention.....................................13 8. Packet Ordering.....................................13 9. Multi-Domain VPLS Service...........................14 10. Maximum Transmission Unit (MTU) Issues.............14 11. Interoperability and Interworking..................14 11.1. Interworking with BGP/MPLS IP VPN..............15 11.2. Interworking With Frame Relay & ATM Attachment Circuits..............................................15 12. Quality of Service.................................15 13. Security...........................................16 13.1. Customer Access Control and Authentication.....16 13.2. Traffic Separation between VPLS Instances......16 13.3. Protection of SP Networks......................16 13.4. Protection of User Data........................17 14. Scalability........................................17 14.1. Mesh topology..................................18 14.2. Signaling......................................18 14.3. MAC addresses and MAC learning.................18 14.4. Packet replication.............................18 14.5. Broadcast limiting.............................19 14.6. Multicast......................................19 15. Management.........................................19 16. Acknowledgments....................................20 17. References.........................................20 18. Authors' Addresses.................................21 Intellectual Property Considerations This document is being submitted for use in IETF standards discussions. Full Copyright Statement draft-lasserre-l2vpn-vpls-ldp-applic-02.txt Aug 2004 Lasserre et al [Page 4] Copyright (C) The Internet Society (2001). All Rights Reserved. This document and translations of it may be copied and furnished to others, and derivative works that comment on or otherwise explain it or assist in its implementation may be prepared, copied, published and distributed, in whole or in part, without restriction of any kind, provided that the above copyright notice and this paragraph are included on all such copies and derivative works. However, this document itself may not be modified in any way, such as by removing the copyright notice or references to the Internet Society or other Internet organizations, except as needed for the purpose of developing Internet standards in which case the procedures for copyrights defined in the Internet Standards process must be followed, or as required to translate it into languages other than English. The limited permissions granted above are perpetual and will not be revoked by the Internet Society or its successors or assigns. This document and the information contained herein is provided on an "AS IS" basis and THE INTERNET SOCIETY AND THE INTERNET ENGINEERING TASK FORCE DISCLAIMS ALL WARRANTIES, EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO ANY WARRANTY THAT THE USE OF THE INFORMATION HEREIN WILL NOT INFRINGE ANY RIGHTS OR ANY IMPLIED WARRANTIES OF MERCHANTABILITY OR FITNESS FOR A PARTICULAR PURPOSE. 1. VPLS Overview The primary motivation behind Virtual Private LAN Services (VPLS) is to provide connectivity between geographically dispersed customer sites across MAN/WAN network(s), as if they were connected using a LAN. The intended applications for the end-user can be divided into the following two categories: - Connectivity between customer routers - Connectivity between customer Ethernet switches In addition, VPLS can also be used by the service providers to deliver voice, video , and data services (i.e., triple play, aggregation of IP services over Ethernet access) to connected end-users. Unlike L3 VPNs such as BGP/MPLS IP VPNs [2547bis] where traffic exchanged between customers and service providers must be IP, VPLS only requires traffic to be Ethernet over which any protocol can be transported, e.g. Netbios or IPX. The Service Provider Network is a packet switched network (PSN). The PEs are assumed to be fully meshed (note that the mesh can be broken with HVPLS) with transport tunnels over which customer frames that belong to a specific VPLS instance are encapsulated and forwarded. IP-in-IP, L2TPv3, GRE, and MPLS are examples of transport tunnels. draft-lasserre-l2vpn-vpls-ldp-applic-02.txt Aug 2004 Lasserre et al [Page 5] Specific labels used to identify end-to-end paths over such transport tunnels, and these end-to-end paths, which are known as pseudo wires (PW), are established via targeted LDP [VPLS-LDP]. VPLS defines the bridging rules required for PEs to provide an emulated Ethernet LAN service. In particular, it defines how a loop- free topology must be built, the forwarding rules between PEs, and the signaling method to set up PWs between PEs. The resulting service provides a unique broadcast domain per VPN, with the ability to send unicast, multicast and broadcast traffic (as well as flooding of unknown unicast traffic). 2. Operation of Control and Data Planes 2.1. Control Plane 2.1.1. Signaling As with [PWE3-ETHERNET], [VPLS-LDP] specifies the use of targeted LDP for the signaling of PWs. PWs are established between PEs that are part of the same VPLS instance. 2.2. Data Plane 2.2.1. Ingress Processing VPLS provides an Ethernet emulated LAN service and hence customer frames are capsulated as Ethernet frames (Ethernet DIX or 802.1). Note that such Ethernet frames can be carried over various access transport technologies (Frame Relay, ATM, etc). Ingress PEs will determine which Forwarding Information Base (FIB) to look up based on the port, VLAN or port/VLAN combination where frames come from. This port to FIB mapping is performed at provisioning time. The destination MAC address is then looked up to determine on which PW this address has been learned from. If the lookup fails, i.e. if this MAC address has not been learned yet, the frame needs to be sent on all the PWs that are part of the corresponding VPLS instance. If the address is known, the frame is sent only over the associated PW. Before actually transmitting the customer frame, it needs to be encapsulated as defined in [PWE3-ETHERNET], and is further encapsulated with the appropriate transport header (e.g. MPLS or GRE). draft-lasserre-l2vpn-vpls-ldp-applic-02.txt Aug 2004 Lasserre et al [Page 6] 2.2.2. Egress Processing Once the tunnel header has been removed, the egress PE determines from the PW label which FIB to look up to determine the egress interface, i.e., VLAN or port/VLAN combination. The original Ethernet frame is then encapsulated with the proper transmission header if necessary (e.g. Frame Relay header) and sent over the corresponding port. MAC addresses are learned dynamically as traffic is exchanged. New source MAC addresses are learned on a per PW label per VPLS instance basis. An aging timer is used to remove such bindings after a period of time. When user topology changes occur, MAC withdrawal messages in the signaling plane may be used to unlearn MAC addresses to improve convergence time. Egress PEs might also be configured to perform specific egress encapsulation functions (e.g. VLAN translation). 2.2.3. Intermediate Node Processing Intermediate nodes (P routers) only act as pure forwarders based on the outer tunnel header. Hence, they do not participate in any VPLS related processing. Only PE routers maintain VPN specific information. This improves the scalability of VPLS service. 3. VPLS vs. Alternative Approaches 3.1. Ethernet Switching Ethernet can be used to provide multipoint connectivity within small geographical areas such as small metropolitan networks. Pure Ethernet based solutions have scalability issues (e.g. STP limitations, 4095 VLAN limitations). Some enhancements such as QinQ, STP extensions (RSTP, MSTP) provide additional scalability. VPLS overcomes several limitations of Ethernet based solutions by supporting large numbers of VPNs, better traffic engineering, transport link layer independence and better quality of service. It is not uncommon for VPLS networks to be complemented with Ethernet switched networks as an aggregation layer. draft-lasserre-l2vpn-vpls-ldp-applic-02.txt Aug 2004 Lasserre et al [Page 7] 3.2. BGP/MPLS IP VPN In metropolitan area networks (MANs), BGP is usually not enabled. MANs provide a transport service to end-users. When multiple sites need to be connected within a metro, VPLS offers the appropriate multipoint transport solution. It is expected that a VPLS instance supports up to O(10^2) site interfaces. When multipoint connectivity is required for a higher number of interfaces sites, with a various range of interface types (e.g. dial-up access, IPSec Tunnels), BGP/MPLS IP VPNs can be more appropriate. Section 11.1 describes how VPLS and BGP/MPLS IP VPNs can be complementary. The following sections compare the characteristics of LDP-based VPLS solution against the requirements specified in [L2VPN-REQ]. Key deployment issues that require careful planning and precise implementation of SP networks are highlighted. 4. Provisioning To provision a VPLS service for a customer, the first step is to create a Virtual Switching Instance (VSI), and assign the customer attachment circuit (AC) (e.g. port, port/VLAN, ATM VC with 1483b encapsulation, etc.) and PWs (including H-VPLS spokes) to it. The PWs interconnect VSIs at different PEs and MTUs together to form an emulated LAN for the customer. One challenge in doing this is, when a VPLS site needs to be added or removed at a PE, in addition to configuring that particular PE, the network operator needs to find out which other PEs participate in that VPLS instance, and re-configure those PEs. PE auto-discovery can automate this process. The pros and cons of several auto-discovery approaches are discussed in 4.1. 4.1. PE Auto-Discovery Currently there are several proposals for PE auto-discovery: the BGP- based approach [VPLS-BGP], the RADIUS-based approach [RADIUS-DIS], and the Provisioning System-based approach. The BGP and RADIUS-based approaches mandate the use of BGP or RADIUS in every PE, and rely on it to propagate the information of which PEs participate in a VPLS instance (Signaling can automatically happen after the other PEs belonging to the same VPLS instance are discovered). The pros of both approaches are reduced provisioning work and no need for a provisioning system. The con is BGP/RADIUS has to be in every PE, which may not be the case in reality. draft-lasserre-l2vpn-vpls-ldp-applic-02.txt Aug 2004 Lasserre et al [Page 8] With the Provisioning System-based approach, network operators do not configure the PEs. Instead, they specify which PEs participate in which VPLS instances at the Provisioning System. The Provisioning System then translates such service information into PE configuration commands and telnet/ssh to the PEs to execute such commands. Because all information related to every VPLS instance is centralized at the Provisioning System, PE auto-discovery is automatically achieved. To add or remove a PE for a VPLS instance, a network operator simply specifies it at the Provisioning System which will then configure the PEs accordingly. For VPLS deployments that span across multiple domains, because the ASBRs (autonomous system border routers) of other domains can be treated as CEs of the current domain, these auto-discovery approaches can all work in the multi-domain case. However, the built-in scalability mechanism in BGP makes the BGP-based auto-discovery more scalable in this scenario [VPLS-BGP]. 4.2. Operations and Maintenance To meet the service level agreement (SLA) with their customers, SPs also need to provision the following: - Traffic management throughout the network and on customer facing ports in particular - Traffic Engineering - Traffic protection (e.g. Fast reroute) - Service management (e.g. SLA measurement, OAM, accounting, billing, etc) Manual provisioning for these tasks can be tedious. A provisioning system is highly desirable. If a provisioning system is used, PE auto-discovery may be integrated into it. 5. Migration Impacts Migration in this document means replacing, or more often, supplementing, an existing metro Ethernet or ATM/Frame Relay network with a VPLS network. There are four likely scenarios: - Interconnecting existing L2 Ethernet islands with a VPLS core - Migrating an existing L2 Ethernet core to a VPLS core; - Interconnecting a new VPLS network with existing ATM/FR networks - Adding VPLS support to an IP routed network Migration impacts may be mitigated through the use of careful planning when building the network. Also, consideration must be taken when integrating with protocols such as STP/MSTP and how control packets (BPDUs) are handled. In addition, one must also consider ongoing draft-lasserre-l2vpn-vpls-ldp-applic-02.txt Aug 2004 Lasserre et al [Page 9] standards efforts within various standards bodies such as the IEEE [802.1ad] and the Metro Ethernet Forum to assess future impact of any changes within the provider network. 5.1. Interconnecting Existing L2 Ethernet Islands with a VPLS Core Today, many existing metro Ethernet networks are relatively small and cover only specific districts in a metro area. Such networks may simply backhaul traffic to a routing backbone and not interconnected at L2. When metro Ethernet service grows and these networks need to be interconnected at L2, one approach that may be used for a migration strategy is to effectively utilize existing L2 (possibly 802.1Q based or QinQ) networks as "islands" attached to an MPLS based VPLS core network. In this particular case, the L2 network uses predetermined Provider 802.1Q tags (P-tags) to transport a given customers traffic. This P-tag is then utilized as a service delimiter that is then stripped prior to being transported across the MPLS cloud. The service delimiting P-tag is used to identify the VPLS instance to which the traffic should be mapped. ----CE1 ------- ------- / -------- CE2- / \ / PE1 / \ \ / \ / \ / \ ---| QinQ \ / MPLS/ \ / QinQ | | Domain PE VPLS PE Domain | \ / \ Domain / \ /\ \ / \ / \ / \ ------- ---------- -------- --CE3 In this scenario, one issue that SPs needs to address is that different sites of a customer may have a mismatch of 802.1Q tags (either different .1q tags on each side, or .1q tag on one side, and untagged on the other side). For example, customer connectivity at one site will be tied to a port on a VPLS PE/MTU that will utilize a PW for tunneling this packet through the network. Customer connectivity at another site will be interconnected to a port on a QinQ switch that will utilize QinQ techniques (see the figure below). By mandating that "the Ethernet packet that traverses a VPLS is always a customer Ethernet packet", the [VPLS-LDP] solution naturally accommodates this need. ----- / A1 \ ---- ----CE1 | / \ ------- ------- / | | | A2 CE2- / \ / PE1 \ / draft-lasserre-l2vpn-vpls-ldp-applic-02.txt Aug 2004 Lasserre et al [Page 10] \ / \ / \ / \ ----- ---- ---| QinQ \ / MPLS/ | | Domain PE2 VPLS | \ / \ Domain / ----- \ / \ / |QinQ|_/ ------- ------- -| | ---- / ------ ---- / \/ \ / \ CE = Customer Edge Router | A3 CE3 --C4 A4 | PE = Provider Edge Router \ / \ / ---- ---- 5.2. Migrating an Existing L2 Ethernet Core to a VPLS Core Providers that have already deployed VLAN based core may choose to overlay an MPLS edge on top of this existing L2 domain. In this method, provider 802.1Q tags maybe assigned to MPLS backbone links that are then used for carrying VPLS traffic. While this approach may allow for a simple transition to solve some immediate deficiencies of a pure L2 network, it still does not solve some of the underlying problems associated with protocols such as spanning tree. In this case, although MPLS may provide some scaling advantages, the limitations associated with spanning tree can still pose potential problems to the overall infrastructure. CE1 ------------------- ------ / / \ -|VPLS| / / \ / | PE |- / \ ------ / \ | 802.1Q/ | | QinQ | \ / ----- \ /\ ------ |VPLS|_/ \ / \ |VPLS| -| PE | \ / -| PE |- / ------ ------------------- ------ \ / \ \ CE3 --CE4 CE2 Alternatively, a parallel VPLS core is built and connected to the existing 802.1Q/Q-in-Q core. The 802.1Q/Q-in-Q core is effectively treated as a super-island. Then one by one, each individual Ethernet access island is disconnected from the existing core (i.e. super- island) and connected to the VPLS core. The migration issues then become similar to those described in 5.1. draft-lasserre-l2vpn-vpls-ldp-applic-02.txt Aug 2004 Lasserre et al [Page 11] A third approach consists in configuring a second VPLS control plane in the existing QinQ PE, hence implementing two virtual networks over a single physical infrastructure. Once the PE/MPLS control plane is running, each customer can be separately migrated through a reconfiguration of its corresponding access ports. For increased stability, the dual control plane approach might require dedicating some links or PEs to the MPLS/VPLS network. 5.3. Interconnecting a new VPLS Network with Existing ATM/FR Networks If interworking at L2 is needed, the existing ATM/FR networks would need to carry bridge-encapsulated traffic. VPLS can support ATM and Frame Relay (FR) attachment circuits with Ethernet bridge encapsulation. Once the FR/ATM encapsulation has been stripped off, the resulting Ethernet frames can be processed as if they came from an Ethernet link. Therefore, interworking can be naturally achieved. If the existing ATM/FR networks do not carry bridge-encapsulated traffic, then interworking can only happen at L3. For example, if both VPLS and ATM/FR carry IP traffic, then an IP router can be used to interconnect the two networks. 5.4. Adding VPLS Support to an IP Routed Network In such a scenario, if existing PEs can support VPLS, then they can continue to serve as PEs. Otherwise, new VPLS PEs need to be added and existing IP routers will serve as Ps or as Layer3-only PEs. Depending on whether the existing IP routers support MPLS or not, MPLS or some other tunneling mechanism such as GRE can be used. 6. Multi-homing Multi-homing is necessary in order to remove a VPLS PE as a single point of failure for all devices attached to it. There are two instances of multi-homing that apply to VPLS: When a CE device is connected to more than one PE, 1. In the case of hierarchical VPLS - when an MTU-s device is connected to more than one PE-rs. In both of these cases, the concern is that a particular MAC address will appear as a source on more than one PE device, causing other PE devices to continuously change their FIBs with regard to the true location of the MAC. This will cause constant table thrashing on the remote PEs, a behavior akin to a Layer 2 switch which participates in a loop. draft-lasserre-l2vpn-vpls-ldp-applic-02.txt Aug 2004 Lasserre et al [Page 12] It is therefore required that any Layer 2 loops, created by multi- homing of a CE or an MTU-s, be resolved within the group of devices participating in that loop. This group includes the multi-homed CE or MTU-s, and all PEs to which it is attached. The PEs involved in such a loop are connected with a full mesh of pseudowires per VPLS instance. There are two approaches to resolving the loops created by the multi- homed devices: 1. Running an MSTP instance between all devices in the group. In this case, the PEs within the group will need to utilize a P-VLAN for the purposes of running MSTP in the group. This P-VLAN can be re-used on non-overlapping groups of multi-homed CE (or MTU-s) and its PEs. It must be clear that the MSTP process discussed here is a completely different and independent instance of STP than any STP the customer may be running. Such customer STP is always tunneled through the VPLS network, and is never acted upon by the PE or MTU-s devices. 2. The MTU-s or the CE can designate its link to one of the PEs it connects to as primary and only send packets for this particular VPLS instance over that link. In this case the MTU-s (CE) is responsible for monitoring the state of that link and for switching to an alternate link if the primary fails. No action is required from the PEs participating in the group, though there should be an indication given from the MTU-s to its connected PEs as to whether the PE is connected to the primary or backup link. This is a very lightweight approach, which is quite useful given the simple and known topology between the CE (MTU-s) and its PEs. With this approach the operator must ensure that PWs in the core remain up, as long as the ingress PE they start from is up. This can typically be ensured with MPLS TE tools, such as fast re-route or back-up LSPs. If there is no available path between the ingress PE and the Egress PEs, a mechanism that monitors the status of the pseudowires to force the access connection to go down when the pseudowires are down might be useful. If PWs in the core go down while their ingress PE is up and accepting customer traffic, black-holes can occur. In each case, the PE nodes are most likely in two different physical locations in the provider network providing network element protection, last mile protection, fiber diversity and provider facility backup. Customer STP traffic is always tunneled through the provider network, and is never acted upon by the PE or MTU-s devices. Lastly, it should be observed that, since VPLS services provide Ethernet switch-like transport level services, the customer is free to connect any device they desire as a CE. This could be anything from a simple host, hub, L2 switch, or a router. The operator has to be cognizant of the different capabilities of each of those devices to ensure loop-free environment when multi-homed. draft-lasserre-l2vpn-vpls-ldp-applic-02.txt Aug 2004 Lasserre et al [Page 13] 7. Loop Prevention Loops in the core VPLS network are prevented by creating a full mesh of transport circuits between PEs and by applying a split-horizon rule. The split-horizon approach prevents a frame received from the backbone network from being sent out anything other than the customer facing ports belonging to that VPLS instance on the receiving PE. The frame MUST not be forwarded out other PW connecting the receiving PE to other PEs participating in the VPLS instance. This provides the necessary protection, network bandwidth optimization and scalability in the carriers' network as it does not rely on link blocking technologies, like spanning tree type protocols. This forwarding mechanism allows PEs to effectively protect the core network from data loops. Customer networks need to be able to transparently transport the protocol information that allows their network to properly converge. However, the provider should consider loop protection schemes between the CE and PE that do not affect the customer functions. This would be in addition to spanning tree when the PE connects to a VLAN based L2 metro or when the customer is directly connected to multiple PE nodes. Methodologies providers can use to avoid loops when multi-homing CE devices have been discussed in the previous section. Some of these mechanisms involved running STP (or MSTP) between groups of PEs. The provider should look at deploying a loop protection scheme that would intervene automatically when it detects a loop condition. This loop protection scheme serves as an additional line of defense against protocol failures or misconfigurations, which can result in data loops. The concern is that a particular MAC address will appear as a source on more than one PE device, causing other PE devices to continuously update their tables. An external loop protection scheme adds a level of insurance above the customer link protection schemes. Its function is to reduce unnecessary core bandwidth usage when a loop condition occurs in an adjacent network and provide an extra level of protection to multi-homed networks. It is a compliment but not a replacement for traditional loop protection mechanisms, like spanning tree. Such a loop protection scheme could be based on the monitoring of the number of Mac addresses moving from one attachment circuit/pseudowire to another circuit/pseudowire. With directly connected customers, careful consideration needs to be given to backdoor connections. Backdoor connections provide an alternate path around a single provider. If a loop detection scheme is invoked here the customer may be forced to traverse a link that is not desired. 8. Packet Ordering draft-lasserre-l2vpn-vpls-ldp-applic-02.txt Aug 2004 Lasserre et al [Page 14] Normally there is only one transmission path towards a destination with VPLS. So there is no packet re-ordering issue. But if some load sharing mechanism is enabled for traffic inside an LSP, or LSPs carrying VPLS traffic are rerouted, packets may be re-ordered inside the PSN. As reordering can be avoided if the load sharing algorithm is based on flows this algorithm should be carefully chosen. Flows can be identified through a number of identifiers in the packet, including MPLS labels, MAC addresses, IP addresses, and UDP/TCP ports. VPLS data packets use the encapsulation mechanism defined in [PWE3- ETHERNET]. An optional control word which contains a sequence number field can be used to assist in-order delivery. If the userÆs applications are sensitive to packet re-ordering, this option may be used. However, enabling sequencing usually causes forwarding performance degradation. Another alternative is to avoid load sharing for traffic inside a LSP and pin down LSPs to avoid rerouting. 9. Multi-Domain VPLS Service As the use of VPLS grows, it is expected that customers will require a single VPLS service delivered by different providers (e.g. either for redundancy or because none of the SPs has the presence to support all the sites of a customer). Different providers would then need to interconnect their VPLS domains for these customers. [VPLS-LDP] has provision for such a requirement, utilizing a full mesh of LSPs among the VPLS gateways of these domains. However, experience of such interconnection is not yet available. 10. Maximum Transmission Unit (MTU) Issues Because of the encapsulation and transport headers, the MTU for user applications will be smaller than the smallest MTU of all the physical links. In responding to path MTU discovery message, each network device must deduct the total header size from a physical linkÆs MTU. Since path MTU discovery is not always used, SPs must clearly communicate the potential MTU issue to their customers and ask for their cooperation. In reality, most applications will work fine but a small number of them may be affected. This is by no means specific to VPLS. Any networks that put additional header(s) on customerÆs packets will have the same issue. 11. Interoperability and Interworking Interoperability should be ensured by proper implementation of the published standards. draft-lasserre-l2vpn-vpls-ldp-applic-02.txt Aug 2004 Lasserre et al [Page 15] 11.1. Interworking with BGP/MPLS IP VPN When interworking VPLS with BGP/MPLS IP VPN, a BGP/MPLS IP VPN (in the backbone) is typically used to interconnect VPLS domains in multiple metros, with such VPLS domains acting as Ethernet aggregation networks for the IP service. In this type of scenario, the BGP/MPLS IP VPN will carry inter-metro traffic whereas VPLS will handle intra-metro traffic. A useful method for interconnecting a VPLS with a BGP/MPLS IP VPN is to use a "link" to interconnect the VSI and the VRF. Such a "link" can be a physical port, a VLAN spanning across one or multiple physical hops, or 2 LSPs with one in each direction, etc. Analogously, this is like interconnecting a L2 switch with a router, with the VSI as the switch and the VRF as the router. Access/transport networks such as VPLS can also be interconnected with BGP/MPLS IP VPNs using various mechanisms such as CarrierÆs Carrier as defined in [RFC-2547]. 11.2. Interworking With Frame Relay & ATM Attachment Circuits Frame Relay (FR) and ATM attachment circuits with Ethernet bridged encapsulation can be terminated within VPLS PEs. The resulting Ethernet frames (i.e. once the FR/ATM encapsulation has been stripped off) are processed as standard Ethernet frames. In order to support a complete interworking model between FR and Ethernet or between ATM and Ethernet, mapping service profiles and OAM traffic from one to the other are necessary. Additionally, circuit management (e.g. LMI to PW state mapping) between the various technologies are required. Such standards are being defined by other standard organizations such as the MPLS-FR-ATM Alliance. 12. Quality of Service The provision of appropriate QoS capabilities may require any combination of the following: - QoS in the access network. - Admission control by the PE router on the ingress access links. - Classification by the PE, for traffic arriving from the CE. Once the PE classifies a user packet, this classification needs to be preserved in the encapsulation (MPLS EXP or IP DSCP) used to send the packet across the backbone. - Traffic conditioning (policing or shaping) by the PE router on the ingress access links. draft-lasserre-l2vpn-vpls-ldp-applic-02.txt Aug 2004 Lasserre et al [Page 16] - DSCP/EXP-based queuing and WRED in the VPLS network - Traffic engineering in the VPLS network. - Fast reroute in the VPLS network None of these features are VPLS specific. The ability to support them depends on whether the features are available on the edge and core devices. It is up to the SPs to decide how to use such mechanisms to provide QoS. Such mechanisms can be used to support either the "hose model" or the "pipe model", although the hose model is a more natural fit and is usually the support model by default. 13. Security 13.1. Customer Access Control and Authentication Control of the customer access can be achieved by controlling physical access to the CEs, the PEs and the links between them. If multiple customers use service delimiting tags in the same trunk link to access VPLS service, and the tags are put on by the customers themselves, ACLs should be used to ensure that each customer only puts on the tag that it is supposed to put on - Packets with other tag(s) must be dropped. If the CE device is a router, then 802.1x may be used for CE device authentication. 13.2. Traffic Separation between VPLS Instances VPLS instances maintain separation of broadcast domains between themselves. Traffic entering a given VPLS instance at a given PE device does not, under any circumstances, cross the boundaries of the VPLS into another instance. VPLS devices (PEs and MTU-s) ensure that by maintaining a FIB table on a per-VPLS instance basis. The above statement is correct regardless of the learning mode employed by a particular VPLS instance (qualified or unqualified), or whether or not VLANs are treated as broadcast domain identifiers, or simply as circuit IDs which have no significance in determining the broadcast domain. In either of these cases, the VPLS instance is the outer-most "envelope" which ensures that traffic within it does not "leak" into another VPLS instance. 13.3. Protection of SP Networks Two types of DoS attacks are of concern with VPLS: draft-lasserre-l2vpn-vpls-ldp-applic-02.txt Aug 2004 Lasserre et al [Page 17] 1. Attacks against VPLS devices 2. Attacks against other devices, for which the VPLS network is a transport. Attacks of the first type are naturally of greater concern for a VPLS operator, because they can destabilize the VPLS network as a whole, and affect multiple customers. The tunneling nature of VPLS by itself limits the possibilities for attacks via the data plane, simply because such attacks will be tunneled through the VPLS network, and will create the same load on the VPLS equipment as legitimate traffic will. Operators must watch for exception packet handling in VPLS equipment. In many cases, exception packets are sent to the control plane for handling. If that is the case, the operator must ensure that such exception packets can be rate-limited in a fashion that guarantees that the control plane will not be significantly burdened by them. A SP should limit the amount of traffic that a customer can flood. The second type of DoS attacks, which use the VPLS network as a transport, are not really a threat to the VPLS devices themselves but are to devices behind them. VPLS PEs may be configured with rate- limiting and rate-shaping capabilities which permit them to limit the amount of traffic allowed into a particular VPLS instance. This prevents a VPLS customer from consuming excessive amount of network resources and from starving other customers. For example, it might be useful to limit the multicast/broadcast/unknown traffic of the customer, considering that the replication of this traffic will create a load in the core proportional to the number of PEs participating to the VPLS instance. Optionally, they can also be tasked with advanced processing of the traffic they tunnel. For example, they may impose access lists which deny traffic from particular sources or protocols. Such approaches however are highly vendor-specific and outside the scope of [VPLS-LDP]. In addition, they may have significant design and operational repercussions. Alternative approaches which hand-off DoS protection activities to non-VPLS devices (such as customer equipment) are a possibility. 13.4. Protection of User Data VPLS does not have special provisioning for ensuring user data security. If a customerÆs traffic is IP traffic, that customer may provide its own user data security by using IPsec. In fact, VPLS is compatible with any use of security by the customer, as long as a clear text Ethernet header is passed from CE to PE. 14. Scalability draft-lasserre-l2vpn-vpls-ldp-applic-02.txt Aug 2004 Lasserre et al [Page 18] As per [L2VPN-REQ], a large SP may eventually require support of up to O(10^4) VPLS instances. In addition, some of these VPLS instances may need to support O(10^2) sites and O(10^3) users/MACs. This section describes the key scalability challenges and how VPLS-LDP addresses them. 14.1. Mesh topology A full mesh of tunnel LSPs, over which PWs are established û resulting in a full mesh of PWs, is created between participating PEs. When using hierarchical VPLS constructs, the size of this full mesh can be reduced to hub PEs aggregating point-to-point spokes as described in section 10 of [VPLS-LDP]. This reduces the number of tunnels and PWs from O(N*N) to O(N). 14.2. Signaling Using HVPLS constructs also allows the total number of targeted LDP sessions to be reduced from O(N*N) to O(N). 14.3. MAC addresses and MAC learning Depending on the type of CE devices used, i.e. switches or routers, the total number of MAC addresses to be learned by VPLS PEs can vary from one address per site to a large number of MAC addresses. When Ethernet networks exceed a large number of MAC addresses (e.g. hundreds), routers are introduced to limit the size of such broadcast domains. This reduces the total number of MAC addresses to learn to such routers only. In the case of large flat Ethernet networks, ingress PEs must be able to limit the number of MAC addresses that can be learned on a per VPLS basis. 14.4. Packet replication With VPLS, broadcast, multicast and unknown destination frames get replicated by the ingress PEs, i.e. close to the source of the frame. Ideally such frames should be replicated as close to the destination as possible to minimize bandwidth consumption. With hierarchical VPLS, the replication process is distributed between several ingress and egress MTUs and PEs. This helps not only minimizing bandwidth resources but also improving multicast performance and reducing latency. draft-lasserre-l2vpn-vpls-ldp-applic-02.txt Aug 2004 Lasserre et al [Page 19] 14.5. Broadcast limiting Ingress MTUs or PEs may be able to rate limit the amount of broadcast/multicast/unknown traffic generated by end users in order to protect core resources and to prevent a few users from using all the bandwidth available. 14.6. Multicast In order to optimize the replication of multicast traffic, it is highly desirable for PEs to support multicast snooping techniques in order to only forward traffic where needed. In the case where the CE device is an L2 switch, IGMP snooping would be required, however, if the CE device is a router PIM snooping would be more applicable. 15. Management Five major areas in management are: Fault, Configuration, Accounting, Provisioning, and Security. They are discussed below. VPLS introduces new configurations related to creation and removal of VSIs, etc. VPLS also introduces new provisioning challenges because the service needs to be delivered end-to-end and therefore many things such as access control, QoS, etc need to be provisioned accordingly. Achieving these via manual CLI configuration can be error prone. Therefore, it is advisable to use a provisioning system for configuration and provisioning. Although VPLS-specific MIBs are still under development, accounting information can usually be achieved via [IF-MIB] and [LSR-MIB]. The important point is that accounting information should be available per service basis. Such information can then be processed by an accounting application to produce the accounting records. Security can be achieved by the measures described in Section 13. Managing fault with VPLS involves multi-point connectivity verification and locating the fault if there is one. Such mechanism is sometimes referred to as "VPLS OAM" and is discussed below. Although VPLS OAM is still being defined, one of the approaches has gained momentum. This approach proposes applying Ethernet OAM mechanism that is being standardized by ITU, IEEE and the Metro Ethernet Forum (MEF) to an VPLS environment for L2 connectivity verification and fault locating, and applying MPLS OAM mechanism such as [LSP-PING] or [BFD] or [VCCV] to MPLS connectivity verification and fault locating. Of course, if IP tunnels (e.g. GRE) are used, IP ping and traceroute can be used in the place of MPLS OAM. VPLS OAM is draft-lasserre-l2vpn-vpls-ldp-applic-02.txt Aug 2004 Lasserre et al [Page 20] therefore achieved by integrating OAM mechanisms at different layers together. With VPLS OAM, ideally the OAM packets should always follow the same path as the VPLS data packets. However, because the Ethernet MAC layer has no TTL support, something needs to be added to the OAM packets to achieve the traceroute capability unless existing network equipment can be enhanced with new OAM processing capability (which is unlikely). As such, traceroute packets may not always follow the same path as the VPLS data packets. Nevertheless, VPLS OAM achieves the practical purpose of verifying VPLS connectivity and locating fault to a good extent. In summary of this section: management of VPLS services involves many things and can be tedious. A complete suite of management software including EMS, NMS and a provisioning system can therefore be highly desirable. 16. Acknowledgments The authors wish to thank the following people for their constructive contributions to the text in this document: Javier Antich Ian Cowburn Richard Foote Rob Nath Nick Slabakov Some text was adapted from the Applicability Statement for BGP/MPLS IP VPNs [AS2547] document. 17. References [AS2547] "Applicability Statement for BGP/MPLS IP VPNs", draft-ietf- l3vpn-as2547-05.txt, Work in progress, May 2004. [BFD] D. Katz and D. Ward, ôBidirectional Forwarding Detectionö, draft-ietf-bfd-base-00.txt, Work in progress, Jul. 2004. [IF-MIB] "The Interfaces Group MIB using SMIv2", McCloghrie, Kastenholtz, RFC 2233, November 1997 [LSP-PING]K. Kompella, P. Pan, et al, "Detecting Data Plane Liveliness in MPLS", , work in progress, Feb. 2004. [LSR-MIB] "MPLS Label Switch Router Management Information Base", Srinivasan, Viswanathan, Nadeau, draft-ietf-mpls-lsr-mib-14.txt>, November 2003 draft-lasserre-l2vpn-vpls-ldp-applic-02.txt Aug 2004 Lasserre et al [Page 21] [L2FRAME] "L2VPN Framework", draft-ietf-ppvpn-l2-framework-05, Work in progress, Jun. 2003. [L2VPN-REQ] "Service Requirements for Layer 2 Provider Provisioned Virtual Private Networks", draft-ietf-ppvpn-l2vpn-requirements- 01.txt, Work in progress, Feb. 2004. [PWE3-CTRL] "Pseudowire Setup and Maintenance using LDP", draft-ietf- pwe3-control-protocol-08.txt, Work in progress, February 2003. [PWE3-ETHERNET] "Encapsulation Methods for Transport of Ethernet Frames Over IP/MPLS Networks", draft-ietf-pwe3-ethernet-encap- 02.txt, Work in progress, Jul. 2004. [RADIUS-DIS] "Using Radius for PE-Based VPN Discovery", Work in progress, Feb. 2004 [VCCV] T. Nadeau et al "Pseudo Wire Virtual Circuit Connectivity Verification (VCCV)", draft-ietf-pwe3-vccv-03.txt, Work in progress, Jun. 2004 [VPLS-LDP] "Virtual Private LAN Services over MPLS", draft-ietf- ppvpn-vpls-ldp-03.txt, Work in progress, Apr. 2004 [VPLS-BGP] "Virtual Private LAN Service", draft-ietf-ppvpn-vpls-bgp- 02.txt, Work in progress, May 2004 [Y.17ethoam] "OAM mechanisms for Ethernet based networks", ITU-T, SG13, Jul. 2003 [802.1ad] "IEEE standard for Provider Bridges", Work in progress, December 2002. [802.1ag] "IEEE Connectivity Fault Management", Work in progress. 18. Authors' Addresses Marc Lasserre Riverstone Networks Email: marc@riverstonenet.com Xipeng Xiao Riverstone Networks Email: xxiao@riverstonenet.com Yetik Serbest SBC Communications Yetik_serbest@labs.sbc.com Cesar Garrido, Telefonica draft-lasserre-l2vpn-vpls-ldp-applic-02.txt Aug 2004 Lasserre et al [Page 22] cesar.garridosanahuja@telefonica.es Marc Rapoport Completel m.rapoport@completel.fr draft-lasserre-l2vpn-vpls-ldp-applic-02.txt Aug 2004