Internet Engineering Task Force CY Lee INTERNET DRAFT S Cirkovic Alcatel Informational M Suzuki NTT A Iwata June 2003 NEC Hybrid Virtual Private LAN Service Status of this Memo This document is an Internet-Draft and is in full conformance with all provisions of Section 10 of RFC2026. Internet-Drafts are working documents of the Internet Engineering Task Force (IETF), its areas, and its working groups. Note that other groups may also distribute working documents as Internet-Drafts. Internet-Drafts are draft documents valid for a maximum of six months and may be updated, replaced, or obsoleted by other documents at any time. It is inappropriate to use Internet-Drafts as reference material or to cite them other than as "work in progress." The list of current Internet-Drafts can be accessed at http://www.ietf.org/ietf/1id-abstracts.txt. The list of Internet-Draft Shadow Directories can be accessed at http://www.ietf.org/shadow.html. Abstract This draft describes a solution to enable an emulated LAN Service or Virtual Private LAN Service (VPLS) by decoupling the bridging and tunneling of Ethernet traffic into Customer Located Equipment (CLE) or Customer Edge (CE) and Provider Edge (PE), respectively. This decoupling of functions allows an operator to provision point-to- point transport of Ethernet services for customer on PEs, while leveraging bridging functions at CLEs to forward traffic towards the appropriate point-to-point Ethernet service leading to the destination node. This decoupling of functions also allow providers to leverage existing network infrastructures to enable VPLS while easing migration to new network infrastructures. In addition, it allows a Expires December 2003 Informational [Page 1] Internet Draft Hybrid VPLS June 2003 provider network to transition and co- exist existing point-to-point Ethernet services with multipoint Ethernet services, using the same network infrastructures and Provider Edge devices and leveraging existing OAM support and OSS. This approach scales as the number of customers and sites increases, and as customers' emulated LAN sites span over a large wide area network. It is also the intention to show that an interface like Ethernet can be offered to customers and emulated LAN services can be enabled, but it is not mandatory to introduce multipoint Ethernet switching in the provider's network. In addition, existing and heterogeneous access technologies can be supported in a common network infrastructure. Abbreviations AC Attachment Circuit [L2VPN-FW] CE Customer Edge (customer managed CPE, in PPVPN the term CE and CLE is not differentiated) CLE Customer Located Equipment (provider managed CPE). It performs IEEE 802.1d/q bridging over physical and virtual ports. It has one VSI. It may use IEEE 802.1d/q VLAN switching for multiple VLANs within the one VSI. CPE Customer Premises Equipment DLCI Data Link Connection Identifier hub CLE A CLE that bridges over more than one virtual port (on the network side) FR Frame Relay L2PE Layer 2 Provider Edge MTU Multi-tenant Unit p2p point-to-point p2p tag A tag indicating the virtual port a frame is being received from or sent to PE Provider Edge PSN Packet Switched Network PW Pseudo-Wire virtual port A virtual interface being used as a bridge port as defined in IEEE 802.1d VPL Virtual Private LAN VSI Virtual Switching Instance [L2VPN-FW] 1. Introduction This draft describes a solution to enable Virtual Private LAN Service (VPLS) by decoupling the bridging and tunneling of Ethernet traffic into CLEs/CLEs and PEs, respectively. This decoupling of functions allows an operator to provision point to point transport of Ethernet frames for customer on PEs, while leveraging bridging functions at CLE to figure out which point to point Ethernet connection to forward Ethernet traffic (destined to a node in a VPL) to. Expires December 2003 Informational [Page 2] Internet Draft Hybrid VPLS June 2003 This Internet Draft only describes how Ethernet over IP/MPLS is used in Hybrid VPLS, other types of Ethernet over X transport technologies (being defined by other standardization bodies] can be used in a similar manner but is not within the scope of this ID. For e.g. when the provider network is an Ethernet network, the customer Ethernet traffic is transported in a p2p Ethernet connection across the provider Ethernet network, and hence multipoint switching of customer traffic is not required in the Ps (Provider core) or PEs equipment. P2P Ethernet connection across a provider Ethernet network is described in [802.1ad]. The implications of decoupling the bridging and tunneling of Ethernet traffic are: a) a PE need not learn and store customers' Ethernet addresses and switch customers' Ethernet traffic (as in PE-based solutions such as [VPLS]) b) a CLE need not be configured with addresses of other CLEs to enable the CLE to tunnel traffic to other CLEs of a VPL (as in a CE- based VPL [CE-VPL]), thereby eliminating the need to signal CLE to CLE tunnel and have common network addressing from CLE to CLE. c) when new sites are added to a VPLS, provisioning at hub CLEs are reduced (as a result of (b)) or eliminated if signaling is used from PE to CE to setup additional virtual ports at a CLE 2. Target Applications The target application of this proposal is for VPLS with mostly traffic from/to a large number of leaf sites to/from a smaller number of hub sites. This covers the majority of the known application requirements for L2VPN. 3. Reference Model The following diagram show a Hybrid VPLS reference model, PE devices transporting customer's traffic over PWs and CLE devices bridging traffic over the PWs. CE1, CE2 and CE3 belong to one VPLS instance. The diagram shows a hub and spoke topology although the proposal is not restricted to such a topology. Additional PWs may be setup for redundancy or failover but the forwarding path used is a tree. [Note: Although a tree is not optimal in forwarding traffic when all sites need to communicate with all other sites frequently, the majority of known L2VPN application requirements can be met with tree topologies and does not require all sites to be connected directly with PWs. A tree may seem like a fragile topology compared to a full-mesh, since it partitions when a PW cannot forward traffic, but it is this property that allows a tree to emulate a LAN correctly and more robustly than a full-mesh of PWs when there is a link failure] A PE, PE2 maps an attachment circuit AC2a to a PW setup between PE2 Expires December 2003 Informational [Page 3] Internet Draft Hybrid VPLS June 2003 and PE1, to transport frames received on AC2a to CLE1 via the single attachment circuit of PE1 to CLE1. Similarly, PE2 maps AC2b to another PW setup between PE2 and PE3, which in turn is mapped to the single attachment circuit of PE3 to CE3. A Hybrid VPLS can contain a single IEEE [802.1q] VLAN or multiple VLANs. PEs are concerned with switching p2p Ethernet PWs. CLEs bridge traffic over different attachment circuits. An attachment circuit may be a physical interface or/and multiplex over a physical interface. If the Ethernet traffic is encapsulated or tagged over the attachment circuit, the encapsulation or tagging should be terminated at the PE. The PE encapsulates the Ethernet frame in a p2p PW to a remote PE. The remote PE decapsulates the Ethernet frame as specified for PW and send it out the appropriate attachment circuit (encapsulating or tagging the Ethernet frame as required). Attachment circuits multiplexed over a physical interface may use : i) a reserved range from IEEE 802.1q VLAN tag not used by customers (used when ii) is not available yet) or a reserved range from 802.1ad Provider Tag. ii) a new EtherType with the same format as IEEE 802.1q VLAN or 802.1ad Provider tag or iii) MPLS label encapsulation The above tag or label is referred to as a p2p tag (to differentiate it from a VLAN tag, which has different semantics) and indicates the virtual port a frame is being received from or sent to. Expires December 2003 Informational [Page 4] Internet Draft Hybrid VPLS June 2003 In the example below, since there is no p2p tag from CLE1, traffic from CLE1 whether multiplexed (tagged/labeled or not) is not of concern to PE1. PE1 simply maps it to one PW. Similarly for traffic from CE3. On CLE2, p2p tags are used (i.e. more than one ACs). If the customer's (CE) traffic is already VLAN tagged then: a) Provider Tag or MPLS label should be used as a p2p tag to multiplex the different ACs at CLE2 or b) if VLAN tag is used as a p2p tag then additional procedures described in "Forwarding" are required. If the customer's traffic is not already VLAN tagged, then CLE2 may use VLAN tag as a p2p tag. ................................................... . . . . . +-----+ +-----+ . CE1----+ CLE1+--+ |CLE2 |----CE2 . +-----+ | <---EthoPSN PW-----> +-----+ . . | +----+ +----+ AC2a| | . . | | | | |------+ | . . +--| PE1|--Routed---| PE2|--------+ . . +----+ Backbone +----+ AC2b . . | ^ . . | |EthoPSN . . | |PW . . +----+ | . . | |<------+ . . | PE3| . . +----+ . . | . . | . . | . . | . . | . .......................|........................... | Emulated LAN CE3 Figure 1 Note : It is not necessary to have a CLE between CE1 and PE1 or CE3 and PE3 if a site does not need more than one direct PW to other sites, I.e a CLE never need to be a hub CLE The multiplexed traffic from CLE2 is mapped to the appropriate PWs, setup towards remote sites. CLE2 performs the bridging of traffic Expires December 2003 Informational [Page 5] Internet Draft Hybrid VPLS June 2003 across the different ACs. The next section outlines a pragmatic migration strategy for CLE2. Expires December 2003 Informational [Page 6] Internet Draft Hybrid VPLS June 2003 CLE1 PE1 PE2 CLE2 +-----------+ +------+ +------+ +-----------+ |\ /| | | | | |\ /| | \ Relay / | | | | | | \ Relay / | | \ / | | | | | | \ / | | \ / | | | | | | \ / | | \ / | | | | | | \ / | | V | | | | | | V | | MAC | MAC | | E| |E | | VP | MAC | +--+--+--+--+ +------+ +------+ +--+--+--+--+ | | | | | | | |LAN | | | | | | | |port | | | | PSN | | | | | +----------+ +------------+ +----------+ | | AC |<-----PW------>| AC | | | | | ...|........................................................|.... . | | . . | | . ...|........................................................|.... | Emulated LAN | | | Figure 2 The above figure shows an AC on CLE2 as a Virtual Port(VP) of CLE2. An AC is service multiplexed over the access link (CLE-PE), as described before. The Encapsulation (E) Entity, is responsible for encapsulating frames to be transported over the PSN. The Virtual Port (VP) provides MAC service support similar to the MAC entity. The Internal Sublayer Service (ISS) and Enhanced Internal Sublayer Service (E-ISS), i.e. the indication and request primitives provided by a MAC entity to the MAC Relay Entity within a CLE is as defined in 802.1d and 802.1q. The Relay is a bridge as defined in 802.1d and optionally 802.1q. CLEs process BPDUs, as defined in 802.1d and optionally 802.1q, and MAC control frames as defined in IEEE specifications. Expires December 2003 Informational [Page 7] Internet Draft Hybrid VPLS June 2003 3.1 Pragmatic migration steps for CLE In the case where a CLE is not capable of bridging to the different virtual ports (using the defined p2p tags above) I.e. does not have the functions as defined above, an existing CLE which maps customer's Ethernet traffic coming from different physical ports on the customer facing side, to different VLAN tags (used as p2p tags) may be used. For e.g. : customer's +-----+ traffic | | -----CE--------| | VLAN/Stacked VLAN tagged traffic | | | | | +-------| CLE |---------------- to provider's network | | | +---------| | | | +-----+ Figure 3 CE MUST be a bridge with multiple ports. The number of remote sites that can be interconnected shall be limited by the number of ports available on the CE and CLE. In this example, the CE bridges traffic to different physical ports corresponding to remote sites, the CLE merely tags (using VLAN tags as p2p tags) traffic coming in from the different ports on the customer facing side. No protocol changes are required at CE or CLE. 3.2 Major technical differences from PE-based VPLS or Provider Bridges [802.1ad] - PEs or Ps are not bridging customers' (internal) MAC addresses - one level of Spanning Tree among CLEs - a CLE processes the customer's BPDUs (customer's BPDUs are transparent to PEs and Ps) 4. Applicability to L2PE The functions defined for a CLE is applicable to a L2PE where the number of VPLS instances supported are smaller, or if the number of VPLS instances can be supported by [802.1ad]. In this case, the Relay in Figure 2 is the Relay defined in Provider Bridges [802.1ad]. The functions defined in a PE in the "Reference Model" is applicable here as well. Depending on how 802.1ad evolves from 802.1d/q additional interface functions may need to be specified. Expires December 2003 Informational [Page 8] Internet Draft Hybrid VPLS June 2003 5. Rationale for this architecture 5.1 Robustness In this architecture when a PW in an emulated LAN does not work, CE routers or bridges will continue to operate properly. If alternate or failover link is available, all sites of the emulated LAN should still be able to reach other; if a failover link is not available, the emulated LAN is partitioned. In either case, higher layer protocols e.g., CE routers and bridges will still be able to function properly over the recovered or partitioned emulated LAN. In contrast in a full-meshed split horizon architecture, if a PW does not work or is missing (due to misconfiguration), CE devices (routers or bridges) will not function properly because the service no longer behaves like an emulated LAN. In particular for the case where CE bridges and xSTP are used in the customer network, there may be loops introduced in the end customer LAN when the PW comes back up again. In the case of CE routers, if IS-IS or OSPF is used as the routing protocol in the customer network, CE routers may black hole traffic when a PW is not working and the problem becomes difficult to troubleshoot because routes for the traffic being black-holed are present in CE routers and CE routers may not be able to use alternate working routes. In Hybrid VPLS, if a PW A-B is broken in a tree spanning C-A-B, the tree is partitioned, but router A and C (and other routers in the emulated LAN can still reach each other). If there is a redundant link between C and B, when this link is active in the spanning tree, A, B and C can reach each other again. In full-meshed split horizon, each PW used between routers would need a backup PW or underlying backup tunnel in case of a PW failure on any PW, i.e. more PWs are needed for functional purposes (not just for redundancy) and the architecture is less robust. To overcome the above problem of emulating a LAN, a suggested method is - the emulated LAN service provided by one of the PEs that is connected by the failed PW must be disabled. To disable an emulated LAN/VPLS on a PE when one or more PWs is missing may require a LAN failure emulation protocol (e.g., a mechanism to exchange information to determine if every node has the same view of PWs or how to partition the the VPLS/emulated LAN). The appropriate PE's emulated LAN should be disabled before any of the BPDUs transmitted has an effect on bridge protocols. Note that during the addition or disconnection of a PE to/from a mesh, the emulated LAN service provided by the PE must be disabled until configurations for all Expires December 2003 Informational [Page 9] Internet Draft Hybrid VPLS June 2003 relevant PWs are completed. If a large number of PWs operational status changes, it is not clear if a PE is able to coordinate the exchange of information of PWs of each VPLS instance, and then report accordingly to the bridge entity, within a short time. Hence the robustness of the VPLS may be impacted by a single PW or subset of PW(s) or PE(s) misbehavior. 5.2 Scalability 5.2.1 Address Learning and BPDU processing This architecture does not attempt to scale the learning of VPN/VPLS ID & MAC address pair which cannot be scaled on a global basis. Instead it constrains the MAC address learning to provider managed CPE devices (CLE) such that PEs do not have to deal with the VPLS ID and MAC address pair and its inherent non-scalability. Hence the MAC addresses table size required at CLE is no more than that supported by existing Ethernet switches for end-customers. Similarly the Spanning Tree BPDU processing required shall be for a smaller number of VLANs. 5.2.2 N^2 scalability For applications which requires only a few PEs to be full-meshed for a VPLS instance, N^2 meshed PWs is not an issue for one single VPLS, but may be an issue for simultaneous broadcast or multicast traffic for many VPLS instances Support of broadcast and multicast on each PE may also be more challenging, when VPLS is used as a backbone. 5.2.3 Provisioning In Hybrid VPLS, less than N^2 provisioning of PWs is required since a full-meshed of PWs is not required. It is only necessary to provision the PEs attached to the few hub sites with the parameters required to setup PWs to the PEs attached to leaf sites and have LDP/L2TP signaling over IGP routes. It is not necessary to provision PWs over specific protected tunnels, as in a full-meshed architecture. 5.2.4 Optimality in forwarding traffic The target application of this architecture covers VPLS with frequent traffic from/to a large number of leaf sites to/from, respectively, a smaller number of hub sites. This represents the majority of the known application requirements for L2VPN. The discussion below considers other potential application requirements. Expires December 2003 Informational [Page 10] Internet Draft Hybrid VPLS June 2003 Although the forwarding of traffic is less optimal using a tree rather than a full-mesh architecture, for other potential applications which requires full-meshed connectivity among bridges in the emulated LAN, a tree is still relatively more scalable when there are a large number of full-meshed of devices in the emulated LAN. For instance, a PE may not be able to multicast effectively in a full- meshed topology. On the other hand, the number of direct devices to multicast to can be reduced in a tree topology by trading off forwarding optimality (i.e., traffic may have to travel more than one hop to reach another device on the emulated LAN). This trade off of forwarding optimality for scalability is similar to multi-hop IP routing. In general, a partial mesh architecture may provide more flexibility in balancing scalability and optimality of forwarding traffic. A partial mesh of PWs can be used in Hybrid VPLS, but a spanning tree will only use the subset of PWs forming a tree for forwarding to avoid loops. The non-active links may be used for failovers, but otherwise is not used to forward traffic normally. In comparison, full-meshed VPLS uses all the links in a full-mesh for forwarding traffic (but since each PW must be protected, the overall backup resources required may be more than Hybrid VPLS); and IP routing may use all the links in a partial mesh topology. In general a tree and full-meshed topology can be used for a relatively smaller number of nodes in a network e.g., a LAN/VLAN or subnet, and for a larger network or VPN these subnets can be interconnected with routing devices over a partial mesh topology. When there are a large number of sites to be interconnected, grouping the CE sites into different subnets (emulated LAN) would allow the VPN to be scaled better (See later section on interconnecting CE devices on different subnets). 5.3 Transparency of service This architecture provides a transparent LAN service such that existing routers and bridges can be connected to the service as if they are connected to a LAN and is transparent to higher layer protocols such as IP and IEEE bridging. It may not be feasible to provide this type of transparency in a full-meshed architecture. If for any reason a PE is not able to transmit frames from one VPLS site/CE to another, but other VPLS sites connectivity are not affected, the service provided is no longer a transparent LAN service, unless additional mechanisms are specified to overcome this problem. The connectivy is more like an NBMA network, resulting in connectivity problems like in an NBMA network, as described in "Robustness" above. Expires December 2003 Informational [Page 11] Internet Draft Hybrid VPLS June 2003 The consequences are dependent on the customer network configuration, and may be difficult to troubleshoot. 5.4 Topology constraints All nodes involved in multipoint Ethernet switching (i.e. CLE, CE bridges, client bridges) for a VPLS must participate in the same spanning tree algorithm to eliminate loops. "Islands" architectures which allow the splicing of independently constructed trees (without a common loop prevention algorithm executed at all nodes involved in multipoint Ethernet switching for a VPLS) cannot prevent loop freedom in the forwarding path. If the islands have additional connectivity to each other via for e.g. other VPLS network or 802.1ad network and if the provider BPDU in each island are not tunneled through the VPLS "backbone", bridges in each islands would not be aware of additional connectivity or loops in the topology. To overcome this problem, additional topology constraints are mandated, as proposed in [Spanning the World with Ethernet]. It is not clear if this type of topology constraints which does not allow a bridge island (operating independent spanning tree) to be connected to more than one VPLS can be mandated in operational networks. 5.5 Protection Racing If a PW fails, it may be restored using IP or MPLS independently of bridging protocols. In this case, there is a possibility that the recovery may occur at the same time as failure detection and recovery of xSTP, leading to race protection and unnecessary lower or upper layer service recovery. Hence the recovery time of the lower layer, IP or MPLS should be of a lower order of magnitude than the bridging protocol. In Hybrid VPLS, a bridge is able to failover to a redundant link when a PW fails and hence is not dependent on IP or MPLS recovery. If PW recovery is used independently, the recovery time of IP or MPLS and bridging protocol should be coordinated as described. Otherwise, a redundant recovery may disrupt service temporarily but CE routers and bridges are still able to function as defined. In contrast, the full-meshed architecture mandates the use of IP/MPLS recovery and recovery must occur before the loss of connectivity results in the consequences described in "Robustness". 5.6 Inter-AS Since bridging is restricted to CLEs only, no inter-AS bridging or multipoint swithing is required here. The bridging protocols running among CLEs in different ASes are still within one VPLS instance. Expires December 2003 Informational [Page 12] Internet Draft Hybrid VPLS June 2003 5.7 Conclusions Hence it is anticipated that in the long term, to build VPLS that scales for many customers and can span many different AS or providers : - bridging of customer traffic is confined to the very edge of the network (preferably the CLE, hence MAC address table size requirement is the same as for a customer LAN, and not a global scalability issue and not dependent on the number of MAC addresses in many customers' LANs. Similarly the amount Spanning Tree BPDU processing is also significantly less than if a device need to support many customers' LAN. L2PEs can be used and scaled for a smaller number of customers edge to edge, if the expectations are that the service will be used for a smaller number of customers). - this allows customer traffic to be tunneled over more scalable network architectures (like routing) which can span many AS and/or providers, rather than doing multipoint Ethernet switching over a large global LAN. The latter is inherently difficult to scale unless the fundamentals of Ethernet switching are changed to parallel existing more scalable network architectures. - the full-meshed architecture is inherently more fragile and expensive to deploy and operate with mandatory requirements for full- meshed protection and frequent monitoring of PWs, solely for functional purposes. 6. Control Plane 6.1 Signaling (optional) 6.1.1 PE to PE To setup PWs, [MARTINI-CTL] or/and [ETH-L2TP] may be used. For e.g. some PWs for a VPLS instance may be setup using [MARTINI-CTL] within a domain and using [ETH-L2TP] to a different domain which may not support MPLS. Using [MARTINI-CTL], the PW is signaled with either the PWid FEC or Generalized ID FEC. The PW Type is "Ethernet". The Generalized ID FEC when used in the same way as described for CLE to PE signaling can reduce provisioning required at PEs. In this case only PEs attached to hub CLEs of a VPLS need to be provisioned and the PW setup is triggered from these PEs. 6.1.1.1 PE to PE Inter-AS Expires December 2003 Informational [Page 13] Internet Draft Hybrid VPLS June 2003 When a PW is signal across domains, it is more likely that the "PE" in another domain would be a border Network Element (NE) and the border NE setup another PW to a PE (facing customers' CEs) independently. The VPLS ID and provider ID, e.g. Avpl.providerY.com may be included in the signaling from a PE in Provider X to a border NE/"PE" in Provider Y network. That way the border NE in Provider Y network can identify which Provider Y internal PW (belonging to Avpl) at the border NE, the external PW should be cross-connected with. Further details shall be specified if there is interest in this area. Note that since bridging is restricted to CLEs only, no inter-AS bridging is required here. The bridging protocols running among CLEs are still within one VPLS instance. 6.1.2 PE and CLE 6.1.2.1 p2p tag - MPLS label If the p2p tag used at an AC is an MPLS label, [MARTINI-CTL] may be used to signal a PW that span the CLE and PE. The purpose of introducing signaling here is to reduce provisioning at CLEs. The goal is to allow a provider to connect a hub CLE to a new site by provisioning at the attaching PEs only, and triggering the CLE-PE PW setup from the PE. No additional provisioning is required at CLEs. This implies a PW setup from PE can indicate to CLE to setup a PW back to the PE. A bi-directtional LSP can then be setup between PE- CLE, by triggering PW setup from a PE. [MARTINI-CTL] is being considered for this purpose. The PE has knowledge of the CLE and initiates the setup of the LSP in the incoming (CLE-->PE) direction by sending a Label Mapping message containing the FEC type, Generalized ID FEC. The FEC element includes the SAII, AGI and TAII. For e.g. the SAII is set to "ETHCL100", the TAII is set to "ETH1L200" and AGI is set to NULL. When the CLE accepts the Label Mapping message, if an LSP in the opposite direction (PE -->CLE) is not setup yet, the CLE shall initiate a Label Mapping message to PE, with the same FEC Type, but the SAII and TAII is now "ETH1L200" and "ETHCL100", respectively, i.e. they are reversed. The PW Type is "Ethernet" unless VLAN tag is used as a p2p tag and the customer traffic is already VLAN tagged (as described in "Reference Model"), in which case the PW Type is "Ethernet VLAN". The latter allows the CLE and PE to indicate to each other via the "Interface Parameters" field of the FEC, the VLAN ID value to be rewritten at egress of CLE or PE. 6.1.2.2 p2p tag - VLAN/Provider Tag Expires December 2003 Informational [Page 14] Internet Draft Hybrid VPLS June 2003 If the p2p tag is an VLAN or Provider Tag, the above or other signaling means e.g. based on GMPLS or LDP/RSVP-TE may be considered in future. No LSP is setup in this case, the signaling is used to indicate the p2p tag values to use at each end, and traffic is forwarded using VLAN or Provider Tags, not MPLS labels. 6.2 Discovery (optional) The purpose of the mechanisms described here is to reduce the provisioning required to enable VPLS. 6.2.1 Reducing Provisioning at CLEs 6.2.1.1 Using signaling As described in "Control Plane" signaling between PE and CLE can be used to reduce the provisioning required at CLE. For e.g. when a new site is added no provisioning is required at CLEs if signaling as described in that section is used, only provisioning at the PEs attached to hub CLEs are required. 6.2.1.1 Using a convention of consecutive multiplexing (p2p tag) IDs A convention of assigning multiplexing ID at a CLE and PE and the mapping of an AC to the appropriate PW can be used to reduce the provisioning required on CLEs and PEs. CLEs can be pre-provisioned with a range of reserved multiplexing IDs, used to multiplex traffic to different remote sites. For e.g. the CLEs in Figure 1 have reserved 100 VLAN tags, 2001-2100 each for this purpose. The customer wants to have two p2p from CLE2 to CLE1 and CE3. The provider may use LMI or other means to provision CLE2 or have the customer configure CLE2 via a web interface to connect to two remote CLEs. This is the only configuration required at the CLE when remote CLE(s) that this CLE should connect to, are added or removed. CLE2 may then allocate VLAN tag 2001 and 2002 for these two connections. PE2 expects CLE2 to use VLAN tag 2001 and 2002 and shall map VLAN tag 2001 to the first PW and 2002 to the second PW, the mapping of VLAN tags to PWs can be arbitrary if there is no specific requirements for the PWs e.g. different SLAs requirements. If a new CLE4 is added, the number of remote CLEs to connect to at CLE2 shall be configured to 3 and VLAN tag 2003 shall be allocated for the new p2p connection to the CLE4. If CLE3 is removed, the number of remote CEs to connect to at CLE2 shall be changed to two, and the virtual port association (VLAN 2003) for the third p2p connection shall be changed to VLAN 2002. Note that this change of VLAN tag to virtual port association should not affect the status of the virtual port. Expires December 2003 Informational [Page 15] Internet Draft Hybrid VPLS June 2003 Using the above convention it is not necessary to use specific protocols to reduce the provisioning of CLEs. 6.2.1.3 LMI The number of remote CLEs to connect to at each CLE may be updated by an LMI approach. After provisioning the required PWs, LMI may be used to inform CLE2 that the PWs are up. When a CLE is added or removed, and the corresponding PW is established or removed, LMI may be used to inform the remote CLE(s) that the p2p connection is up or down, respectively. 6.2.2 Reducing Provisioning at PEs To reduce provisioning at PEs, network management tools which reduces provisioning steps or the BGP mechanism described in [L2VPN-KOMPELLA] may be used. DNS [DNS-VPLS] or the BGP mechanism described in [L2VPN-KOMPELLA] may be used to discover the remote PEs and cross-connect information to setup the PWs. 7. Forwarding 7.1 PE If the p2p tag used at an AC is VLAN/Provider Tag, the VLAN/Provider Tag if present is removed (or the VLAN tag is rewritten if Section 3(b) is used) first and the Ethernet traffic from an AC is encapsulated in a PW as described in [MARTINI-ENCAP] or [ETH-L2TP]. Ethernet traffic received from a PW on the network facing side is decapsulated and forwarded to an AC as described in [MARTINI-ENCAP] and [ETH-L2TP]. If the p2p tag used is MPLS label, the PW between the CLE and local PE is spliced with the PW between the local PE and remote PE [ROSEN- SIG]. In this case, the label of the "ingress" PW is replaced by the label of the "egress" PW, and no decapsulation or encapsulation need to be performed at the local PE (the splicing point). 7.2.1 Inter-AS PW A PW spanning a PE and a border Network Element may also be spliced with another PW from the border Network Element to a PE in another AS at the border Network Element, in a similar manner as described above. The advantage of a p2p PW across ASes is multipoint switching Expires December 2003 Informational [Page 16] Internet Draft Hybrid VPLS June 2003 or bridging across AS is not required. 7.2 CLE If the p2p tag used is VLAN or Provider Tag, an Ethernet frame from a port on the customer facing side is tagged as defined in [IEEE 802.3] and being defined in [IEEE 802.1ad], respectively. If the p2p tag used is VLAN Tag, the following additional procedures are required. The customer VLAN tag in traffic from CLE2 (which indicates VLAN membership and has a different semantics than p2p tag) is swapped with a p2p tag at CLE2 and the customer VLAN tag restored at PE2. Similarly customer VLAN tagged traffic decapsulated from a PW at PE2, is swapped with the appropriate p2p tag before being sent to CLE2. At CLE2, the p2p tag is rewritten with the customer VLAN tag. A separate Spanning Tree is required for each customer VLAN in this case. Additional parameter signaling between CLE and PE would help reduce the provisioning required in this case as described in "Signaling". Note that this is not a preferred approach as a separate PW is required for each customer VLAN and it may not preserve the ordering of frames (of the different customer VLAN) from the customer LAN, and should only be used if Provider Tag or MPLS label cannot be used. If the p2p tag used is MPLS label, an Ethernet frame from a port on the customer facing side is encapsulated in a PW as described in [MARTINI-ENCAP]. Ethernet traffic received in a PW from a PE is decapsulated and forwarded to an AC as described in [MARTINI-ENCAP]. The tagged or encapsulated frame is bridged over virtual ports. The learning, bridging, filtering and forwarding procedures are as defined in [802.1d] and [802.1q] (and [802.1ad] in the case of a L2PE), except that a virtual port on a CLE can be a bridge port. For instance, a CLE learns MAC addresses from the customer facing ports and the virtual ports (corresponding to remote sites of a VPLS). When a new MAC address is learned, the MAC address is associated with the virtual port where the frame arrives. When a frame with the cached MAC address is received, the CLE knows which virtualport to forward the frame to. When a frame with a new MAC address is received, a CLE floods the frame to all other ports (including virtual ports), except the port where the frame is received from. 8. Deployment Scenarios 8.1 Point-to-Point Ethernet Service The provider provides and provision point-to-point Ethernet service between CEs. [Note: CEs and CLEs functions described above may be Expires December 2003 Informational [Page 17] Internet Draft Hybrid VPLS June 2003 collapsed into CE devices]. The different sites requiring the service are specified by customers. The customer provisions the number of remote CEs, and the set of multiplexing IDs to be used. To reduce provisioning, a provider and the customer may agree apriori the set of multiplexing IDs range to be used for the p2p services. The Section "Discovery" describes different means for a provider to reduce the provisioning required by the provider and the customer. If signaling from PE to CE is desired to setup additional p2p tags, UNI signaling may be considered instead. The customer manages the bridging aspects of the emulated LAN on CEs, while the provider manages the point-to-point Ethernet service. 8.2 Emulated LAN Service The provider provides Emulated LAN service. Any device with Ethernet interface (e.g. host, routers, bridges) can be connected to the emulated (bridged) LAN as if the devices are connected to a LAN segment or bridged LAN. P2P Ethernet PWs are provisioned in the provider network from PE to PE. The provider pre-provisions the set of multiplexing IDs (p2p tag IDs) that can be used for the p2p Ethernet service on CLEs if signaling from PE is not used to provision the CLE from the PE side. The Section "Discovery" describes different ways to reduce provisioning on CLEs and PEs. New virtual ports can be instanstiated at a CLE without direct provisioning by: i) triggering the setup of an LSP between a CLE and a PE from the PE or ii) updating the number of remote CLEs to connect to, at a CLE, using an LMI method. CLEs perform the bridging among different sites. 9. Interconnecting sites with different access media This section describes how Hybrid VPLS can be used with existing and heterogeneous access technologies within an emulated LAN. This section shall be separated into a different ID in future. It describes how a CE with point-to-point links such as Frame Relay or ATM access can appear as a node on a VPLS/emulated LAN, thereby allowing a CE to work with other CEs as if it is connected to the same LAN as the other CEs. Only one DLCI is required at a CE router with FR access to allow it to peer with other CE routers with Ethernet or FR/ATM access. This reduces the amount of provisoning required by end customers, e.g. instead of provisioning m point-to- Expires December 2003 Informational [Page 18] Internet Draft Hybrid VPLS June 2003 point DLCIs and m subnets for CE routers to peer, an end customer only need one DLCI or Ethernet interface and IP addresses for one subnet for routers to peer with other routers on the emulated LAN. When a new site is added, only the new router need to be provisioned and only one DLCI or one Ethernet interface is required. Alternatively, if existing FR CE devices are configured with routed encapsulation and it is not feasible to reconfigure the FR CE devices, some of the FR CE and Ethernet CE devices can be connected to different existing subnets instead, while still requiring only one DLCI or Ethernet interface at each site for all CEs to be interconnected. Note that this does not prevent additional access interfaces to be used for redundancy. The following sub-sections describe the different alternatives which can be used to enable an end customer to interconnect sites with different Layer 2 access technologies (e.g. Ethernet, Frame Relay or ATM). In general, 3.1 and 3.2.2 which allow CEs to peer over one subnet reduces overall provisioning required at CEs, reduces link states in routing protocols and the flooding of link states compared to using a partial or full-mesh of point-to-point links with 3.2.1. 3.1 Bridged Encapsulation is the preferred method and scale well when used with Hybrid VPLS while 3.2.1 and 3.2.2 should only be used (sparingly in a provider network since the latter two methods require caching IP and MAC addresses in the network and do not scale well) when it is not feasible to configure the FR/ATM interface to use bridged encapsulation. When there are a large number of sites to be interconnected, grouping the CE sites into different subnets (emulated LAN) would allow the VPN to be scaled, while still reducing provisioning and link states within one subnet. 9.1 Bridged encapsulation This method allows CEs connected to different access technologies to peer on the emulated LAN, since traffic (including routed traffic e.g. IP) are encapsulated in Ethernet frames, hence Layer 3 traffic shall be able to work as if the devices are connected over a LAN. Traffic from different access technologies are decapsulated or demultiplexed and Ethernet frames are transported over PWs from PE to PE and bridged in CLEs as described earlier in the draft. Expires December 2003 Informational [Page 19] Internet Draft Hybrid VPLS June 2003 For e.g. in Figure 4a, traffic from CE4 is bridged encapsulated as defined in [FR-MP] or [ATM-MP], the ATM/FR headers are decapsulated at PE3 and transported over Ethernet over PSN PW (over MPLS or IP as defined in [MARTINI-ENCAP] or [ETH-L2TVP]) to PE2. At PE2, the Ethernet frames are decapsulated as defined in [MARTINI-ENCAP] and sent to CLE2, and CLE2 bridged traffic as described in earlier sections. Similarly the Ethernet traffic from CE2 is bridged in CLE2, decapsulated at PE2 and transported over Ethernet PW as defined in [MARTINI-ENCAP] to PE3. At PE3, the Ethernet frames are decapsulated as defined in [MARTINI-ENCAP] and bridged encapsulated as defined in [FR-MP] or [ATM-MP] before being sent to CE4. .................................................... . . . . Eth +-----+ Eth +------+ Eth CE1-----+ CLE1+--+ | CLE2 |-----CE2 . +-----+ | <----EthoPSN-----> +------+ . . | +----+ +----+ | | |Eth. . +--| | | |-AC2a-+ | | . . | PE1|----PSN----| PE2|-AC2b---+ | . . +----+ Backbone +----+-AC2c-----+ . . | ^ ^ . . | EthoPSN| |EthoPSN . . | | | . . +----+ | | . . | |<-----+ | . . | PE3|<-------+ . . +----+ . . | | . . | +------+ . . | | . . | | . . | | FR . . Eth| | . ......................|.......CE4.................... | Emulated LAN CE3 Figure 4a - CE devices connected over an Emulated LAN This method when used with Hybrid VPLS requires less changes in a provider network and scales better than the other methods described here in a provider network. However, it must be feasible to reconfigure the FR/ATM CEs where needed, to use bridged encapsulation for traffic from customer site(including routed traffic e.g. IP). Expires December 2003 Informational [Page 20] Internet Draft Hybrid VPLS June 2003 9.2 Routed encapsulation 9.2.1 Routed Encapsulation over a point-to-point link This method allows a CE with FR/ATM access to peer with a CE with Ethernet access over a different subnet than the emulated LAN used by CEs with Ethernet access, allowing an FR/ATM CE to maintain the same configuration as before. If the number of end customer sites are large, grouping sites into different subnets/emulated LAN (even for CEs with Ethernet access only, I.e. this is independent of the reasons or need to use routed encapsulation over FR/ATM access) would be the better approach to scale the Virtual Private LAN or VPN design, while reducing the provisioning required by peering some L3 devices over a subnet or emulated LAN. Eth AC1a +------+ Eth CE1--------------+ | CLE2 |-----CE2 --------+ | <----EthoPSN-----> +------+ | | +----+ +----+ | | |Eth | +--| | | |-AC2a-+ | | | | PE1|----PSN----| PE2|-AC2b---+ | +--------+----+ Backbone +----+-AC2c-----+ AC1b ^ | ^ ^ | | EthoPSN| |IPoPSN IPoPSN | | | | | +----+ | | +-->| |<-----+ | +---------| PE3|<-------+ | +----+ | | | | Eth| +------+ | +-----+ | | | CLE3| | | +-----+ | FR FR | Eth| | | | CE4 | | CE5 CE3 Figure 4b - CE IP Devices connected over multiple subnets Note: CE1, a router has 2 IP interfaces (VLAN tagged) a) AC1a is on the same subnet as CE2,CE3,CE4 (emulated LAN service) b) AC1b is on the same subnet as CE5 (p2p IP over Ethernet service) The disadvantage is some CEs with Ethernet access would need to be Expires December 2003 Informational [Page 21] Internet Draft Hybrid VPLS June 2003 configured to peer with multiple FR/ATM sites on separate subnets instead of with one subnet (as with other CEs with Ethernet sites), even for a VPN with a relatively smaller number of sites, as shown in Figure 4b. The next alternative overcomes this issue but introduces additional L3 configuration changes in L3 CE devices). Expires December 2003 Informational [Page 22] Internet Draft Hybrid VPLS June 2003 9.2.2 Routed Encapsulation over a broadcast network This method allows all CEs to peer over the same emulated LAN or subnets, but require configuration changes in L3 protocols on FR/ATM CEs (e.g. OSPF Interface Type is changed to broadcast mode) and hence may not be feasible for some deployment. It allows CE devices which for whatever reason are not able to use bridged encapsulation instead of routed encapsulation, but wished to peer over the same emulated LAN, instead of over different subnets as in the previous alternative. .................................................... . . . . Eth +-----+ Eth +------+ Eth CE1-----+ CLE1+--+ | CLE2 |-----CE2 . +-----+ | <----EthoPSN-----> +------+ . . | +----+ +----+ | | |Eth. . | | | | |-AC2a-+ | | . . +--| PE1|----PSN----| PE2|-AC2b---+ | . . +----+ Backbone +----+-AC2c-----+ . . | ^ ^ . . | EthoPSN| |IPoPSN . . | | | . . +----+ | | . . | |<-----+ | . . | PE3|<-------+ . . +----+ . . | | . . Eth| +------+ . . +-----+ | . . | CLE3| | . . +-----+ | FR . . Eth| | . ......................|.......CE4.................... | Broadcast network CE3 Figure 4c - CE IP Devices over a Broadcast network Note: CE1,CE2,CE3,CE4 are all on the same subnet Expires December 2003 Informational [Page 23] Internet Draft Hybrid VPLS June 2003 9.3 Additional mechanisms required for routed encapsulation [ARP-MEDIATION] specifies how a point-to-point PW with two different access media can interwork, when carrying IP traffic. One MAC device is assumed at the Ethernet end. This proposal consider that there may be more than one MAC device/ address at the Ethernet end (although there is only one device at the FR/ATM end). Even though the transport of traffic on the PW is point-to-point, the MAC layer is still a multipoint service at the Ethernet end. For e.g. a customer may use a p2p Ethernet service carrying IP trafic, but the customer may have more than one MAC device on the MAC layer at the Ethernet end. [Note that the bridged encapsulation method is also able to handle this case] For routed encapsulation over a broadcast network, this draft allows the CE FR/ATM device to peer over an emulated LAN service as described earlier in the draft. [IPLS] uses a mesh of PWs and may not emulate LAN connectivity in the case where CE A is not able to see another CE B while other CEs can see CE A and CE B. This issue is similar to the full-meshed architecture. 9.3.1 Mechanisms required at FR/ATM PE for routed encapsulation The additional mechanisms required at the FR/ATM PE for both point- to-point link and broadcast networks are described in this section. The common problem for both cases is one end of the service is point- to-point in nature and the other end is a shared media, and there are no Ethernet names/addresses (as in bridged encapsulation) provided from the point-to-point end, when the traffic is routed encapsulation is used. 9.3.1.1 Discovering MAC address of a network device at Ethernet site To illustrate the problem, Fig 4c shows CE4 connected to the emulated LAN, traffic from CE4 is routed encapsulated at the FR/ATM access link. Only the network address is available when the routed encapsulated traffic from CE4 is decapsulated at PE3. PE2 needs to determine the corresponding MAC address of the network address on the shared media end. It is possible that there are other MAC devices on the same subnet as CE2. Similarly in Fig 4b, where CE1 and CE5 are connected to the same subnet. In the case of IP traffic, if CE routers support Proxy ARP, when PE2 receives an IP packet from the IPoPSN PW for an unknown address, PE2 (as a Proxy ARP) shall send an ARP request for the MAC address of the IP address of the packet to the emulated LAN via AC2c. PE2 shall cache the MAC address learned in an ARP reply, for the corresponding Expires December 2003 Informational [Page 24] Internet Draft Hybrid VPLS June 2003 IP address. [Note: If the IP address belongs to a device on the subnet, PE3 should receive a response from the IP device itself, if the IP address belongs to a device on another subnet, the Proxy ARP in Ethernet CE routers shall respond with its own MAC address] In the case of [IS-IS or] where an IP CE router support [IRDP], PE2 shall snoop for router advertisement messages on AC2c to discover a router to forward an Ethernet frame to. If the router selected is not the optimal router on the subnet, a redirect message shall be sent towards the source of the packet to inform it of the optimal router. PE2 shall snoop for redirect message. [In the case of IS-IS, the optimal router MAC address is indicated] in IP, the optimal router IP address is indicated. In IP, an additional step is required, I.e. PE2 shall send an ARP request of the IP address of the optimal router to learn the corresponding MAC address] PE2 shall cache the destination network address the redirect message is meant for and corresponding MAC address of the optimal router and forward subsequent packets with that MAC address. When PE2 receives a packet from the IPoSPN PW and has a corresponding learned MAC address, it shall encapsulate the packet in an Ethernet frame with the learned MAC address and send the Ethernet frame towards CE2 via AC2c. From there on, CLE2 shall bridge the Ethernet traffic as described for emulated LAN service earlier. 9.3.1.2 Discovering MAC address of network devices at FR/ATM site To discover the MAC address of network address of devices at Site 5 (CE5) as shown in Figure 4b, the following mechanism is required. The device sending the packet at Site 1 (CE1) shall use already defined specifications for the routed protocol. For e.g. an IP device shall send an ARP request for the IP destination address via AC1b. PE1 shall act as a Proxy ARP and respond with its own MAC address for the IP address if the IP address belongs to a device on a remote end IPoPSN PW. PE1 shall be configured a priori with the network addresses of remote FR/ATM CEs (1 network address for each FR/ATM CE) at the end of each IPoPSN PW, or alternatively PE3 may use Inverse ARP to discover the IP address of the FR/ATM CE and use another mechanism e.g. PW signaling to relay the IP address to PE1. 9.3.1.3 Encapsulation of traffic from Ethernet site PE1 shall decapsulate the IP traffic from the Ethernet frame and encapsulate the IP traffic in IPoPSN PW, PW Type "IP Layer2 Transport" and send it to PE3. PE3 shall decapsulate the IP packets from the IPoPSN and encapsulate the packets in routed encapsulation to CE3 as defined in [FR-MP] or Expires December 2003 Informational [Page 25] Internet Draft Hybrid VPLS June 2003 [ATM-MP]. 9.3.1.4 Encapsulation of traffic from FR/ATM site PE3 shall decapsulate the IP traffic from FR/ATM CE as defined in [FR-MP] or [ATM-MP]. PE3 shall encapsulate the IP traffic in a IPoPSN PW, PW Type "IP Layer2 Transport" and send it to PE1. PE1 shall decapsulate the IP traffic from the IPoPSN PW and encapsulate it in an Ethernet frame (with appropriate VLAN tag) before sending it on AC1b. 9.3.2 Requirements for Ethernet CE devices for routed encapsulation For 9.2.1 and 9.2.2 above, the requirements on Ethernet CE devices are: i)IP devices (routers and hosts) are able to respond to ARP request and IP routers are Proxy ARP for its routes OR ii) there is a router advertisement protocol (e.g. IS-IS or [IRDP]] 10. Security Considerations Pseudo-Wires as used in this draft do not introduce any new security issues. PWs with IP payload may have security issues, if the Proxy ARP resides on a PE. This issue is being investigated. 11. Acknowledgments This draft benefited from discussions with Arnold Jansen, Jeremy deClercq, Plavin D'Souza, Raymond Chang, Roy Nighswander, Pierre Bolduc, Steve Buchko, Mustapha Aissaoui, David Watkinson, Joel Halpern, Chris Bachalo, Alex Zinin, Zlatko Krstulich, Dimitri Papadimitriou, Hansen Chan and Italo Busi. Thanks to the IS-IS and OSPF mailing lists for clarifying routing issues, IEEE 802.1 and Norm Finn for clarifying bridging issues and the PPVPN mailing lists for comments on the initial version of this draft. Thanks to Radia Perlman for writing Interconnections. 12. Intellectual Property Considerations The IETF is being notified of intellectual property rights claimed in regard to some of the specification contained in this document. For more information consult the online list of claimed rights. Expires December 2003 Informational [Page 26] Internet Draft Hybrid VPLS June 2003 13. References 13.1 Normative References [MARTINI-ENCAP] Martini et al, "Encapsulation Methods for Transport of Ethernet Frames Over IP/MPLS Networks", work in progress, Feb 2003 [ETH-L2TP], Aggarwal et al, "Transport of Ethernet Frames over L2TPv3", work in progess, work in progess, Oct 2002 [802.1D] IEEE, "ISO/IEC 15802-3:1998,(802.1D, 1998 Edition), Information technology --Telecommunications and information exchange between systems --IEEE standard for local and metropolitan area networks --Common specifications-Media access control (MAC) Bridges", June, 1998. [802.1Q] IEEE, "IEEE Standards for Local and Metropolitan Area Networks: Virtual Bridged Local Area Networks", 1998 . [802.3] IEEE, "ISO/IEC 8802-3: 2000 (E), Information technology--Telecommunications and information exchange between systems --Local and metropolitan area networks --Specific requirements --Part 3: Carrier Sense Multiple Access with Collision Detection (CSMA/CD) Access Method and Physical Layer Specifications", 2000. [802.1ad], "Provider Bridges", Tony Jeffree, work in progess, Dec 31 2002 13.2 Informative References [Perlman] R. Perlman, "Interconnections, Bridges and Routers", 1997, Addison-Wesley. [L2VPN-KOMPELLA] K. Kompella et al, "MPLS-based Layer 2 VPNs", work in progress, July 2001. [L2VPN-FW] L. Andersson, E. Rosen, "L2VPN Framework", work in progress, Jan 2003 [VPLS] Lasserre, M, Kompella, V, et al, "Virtual Private LAN Services over MPLS", work in progress, March 2002 [ARP-MEDIATION] H. Shah et al, "ARP Mediation for IP Interworking of Layer 2 VPN", work in progress, October 2002 [IPLS] H Shah et al, IP over LAN Service, work in progress, March 2003 Expires December 2003 Informational [Page 27] Internet Draft Hybrid VPLS June 2003 [MARTINI-CTL], Martini et al, "Transport of Layer 2 Frames over MPLS", work in progress, Feb 2003 [ROSEN-SIG], Rosen, "Provisioning Models and Endpoint Identifiers in L2VPN Signaling", work in progress, May 2003 [VPLS-DNS] J. Heinanen, "DNS/LDP Based VPLS", work in progess, January 2002. [CE-VPL] CY Lee, M Higashiyama, "CE-based VPL", work in progress, Nov 2002 [RFC3439], R. Bush, D. Meyer, "Some Internet Architectural Guidelines and Philosophy", Informational, December 2002 [IRDP], S Deering, Router Discovery Protocol, RFC 1256, 1991 [FR-MP] C. Brown, A. MALIS, "Multiprotocol Interconnect over Frame Relay", Standards Track, Sept 1998 [ATM-MP] D. Grossman, J. Heinanen, "Multiprotocol Encapsulation over ATM Adaptation Layer 5", Standards Track, Sept 1998 Authors' Addresses Cheng-Yin Lee Cheng-Yin.Lee@alcatel.com Sasha Cirkovic Sasha.Cirkovic@alcatel.com Muneyoshi Suzuki suzuki.muneyoshi@lab.ntt.co.jp Atsushi Iwata iwata@ccm.CL.nec.co.jp Expires December 2003 Informational [Page 28]