INTERNET-DRAFT Roger Lapuh Intended Status: Informational Paul Unbehagen Expires: September 30, 2014 Avaya Peter Ashwood-Smith Huawei Phillip Taylor Leeds Metropolitan University March 31, 2014 SPB Deployment Considerations draft-lapuh-spb-deployment-02 Abstract Based on many live deployments, including the Sochi Winter Olympics 2014 network and multiple interoperability events, this document provides advice to network operators about best practices when implementing IEEE 802.1aq Shortest Path Bridging (SPB) networks. It is principally addressed to system integrators and solution providers, including those that do not yet support SPB. Some advice to protocol implementers is also provided. The intention of the advice is to facilitate multi vendor network deployments as well as provide guidance for new installations. Status of this Memo This Internet-Draft is submitted to IETF in full conformance with the provisions of BCP 78 and BCP 79. Internet-Drafts are working documents of the Internet Engineering Task Force (IETF), its areas, and its working groups. Note that other groups may also distribute working documents as Internet-Drafts. Internet-Drafts are draft documents valid for a maximum of six months and may be updated, replaced, or obsoleted by other documents at any time. It is inappropriate to use Internet-Drafts as reference material or to cite them other than as "work in progress." The list of current Internet-Drafts can be accessed at http://www.ietf.org/1id-abstracts.html The list of Internet-Draft Shadow Directories can be accessed at http://www.ietf.org/shadow.html Lapuh, R. Expires September 30, 2014 [Page 1] INTERNET DRAFT SPB Deployment Recommendations March 5th This Internet-Draft will expire on September 30th, 2014. Copyright and License Notice Copyright (c) 2014 IETF Trust and the persons identified as the document authors. All rights reserved. This document is subject to BCP 78 and the IETF Trust's Legal Provisions Relating to IETF Documents (http://trustee.ietf.org/license-info) in effect on the date of publication of this document. Please review these documents carefully, as they describe your rights and restrictions with respect to this document. Code Components extracted from this document must include Simplified BSD License text as described in Section 4.e of the Trust Legal Provisions and are provided without warranty as described in the Simplified BSD License. Table of Contents 1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . 4 1.1 Terminology . . . . . . . . . . . . . . . . . . . . . . . . 4 1.2 Motivation and Background . . . . . . . . . . . . . . . . . 4 2. General Deployment Recommendations . . . . . . . . . . . . . . 4 3. Infrastructure Configuration Recommendations . . . . . . . . . 6 3.1 IS-IS SYSTEM ID AND SPB NICKNAME CONFIGURATION . . . . . . 6 3.2 SPB NETWORK LINKS . . . . . . . . . . . . . . . . . . . . . 6 3.3 SPB AND SPANNING TREE . . . . . . . . . . . . . . . . . . . 7 3.4 SPB FABRIC CONFIGURATION . . . . . . . . . . . . . . . . . 8 3.5 SPB SERVICES MAPPING . . . . . . . . . . . . . . . . . . . 8 3.6 SPB AND IP ROUTING . . . . . . . . . . . . . . . . . . . . 8 4. Standard Improvement Recommendations . . . . . . . . . . . . . 9 5. OA&M . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 9 6. Tenant Separation Considerations and Payment Card Industry Security Requirements . . . . . . . . . . . . . . . . . . . . 9 7 Deployment Experiences . . . . . . . . . . . . . . . . . . . . 10 7.1 DEPLOYMENT SCENARIO A . . . . . . . . . . . . . . . . . . . 10 7.2 DEPLOYMENT SCENARIO B . . . . . . . . . . . . . . . . . . . 10 7.3 DEPLOYMENT SCENARIO C . . . . . . . . . . . . . . . . . . . 11 7.4 DEPLOYMENT SCENARIO D . . . . . . . . . . . . . . . . . . . 11 8 Security Considerations . . . . . . . . . . . . . . . . . . . . 12 9 IANA Considerations . . . . . . . . . . . . . . . . . . . . . . 12 5 References . . . . . . . . . . . . . . . . . . . . . . . . . . 12 5.1 Normative References . . . . . . . . . . . . . . . . . . . 13 5.2 Informative References . . . . . . . . . . . . . . . . . . 13 Lapuh, R. Expires September 30, 2014 [Page 2] INTERNET DRAFT SPB Deployment Recommendations March 5th Authors' Addresses . . . . . . . . . . . . . . . . . . . . . . . . 13 Lapuh, R. Expires September 30, 2014 [Page 3] INTERNET DRAFT SPB Deployment Recommendations March 5th 1 Introduction This document provides a set of recommendations and reference points for the deployment of IEEE 802.1aq/RFC6329 - Shortest Path Bridging (SPB) networks based on MAC in MAC encapsulation. It focuses on the key network design items and does not go into describing the protocol details. The IEEE 802.1aq standard has been ratified in March 2011, thus the recommendations described here are focusing on the standards based implementations. 1.1 Terminology The key words "MUST", "MUST NOT", "REQUIRED", "SHALL", "SHALL NOT", "SHOULD", "SHOULD NOT", "RECOMMENDED", "MAY", and "OPTIONAL" in this document are to be interpreted as described in RFC 2119 [RFC2119]. 1.2 Motivation and Background This document provides a checklist of recommendations which are based on multiple documented multi vendor Interoperability tests [SPBWIKI] and more than three years of standards based production deployment experiences including the following scenarios: Inter data center connectivity for data center migrations and consolidations, hosted data center virtualization, multi-site WAN backbone over carrier Ethernet service infrastructures, campus LAN switching, IP Multicast based video surveillance solutions and event and exposition networks. It summarizes the learning's and experience acquired during those activities. New SPB installations can benefit from following the recommendations und deployment guidelines below. 2. General Deployment Recommendations All the following described deployments have shown sub second convergence times in case of link or nodal failures and recovery within the SPB fabric. Recommendation 1: To achieve quick failure recovery, connection oriented point-to- point interfaces have been used, and shared segments between SPB fabric nodes have been avoided. Ethernet based methods such as Ethernet based remote fault indication (RFI), IEEE 802.ag (CFM) or Lapuh, R. Expires September 30, 2014 [Page 4] INTERNET DRAFT SPB Deployment Recommendations March 5th similar are used to detect link faults quickly to trigger path recalculations in case of a link or nodal failure. Recommendation 2: The end-point-only provisioning for network virtualization with SPB has proven very effective in many of the installations described below. SPB's service ID (I-SID) with its (24 bit) addressing space has helped to keep the VLAN numbering (1 to 4095) local to the respective access network region (e.g. Data Center or tenant), avoiding the complexity of managing a global VLAN space out of a range of only 12 bits. In larger networks, it is recommended to define a global virtualization schema based on I-SIDs, and not tie VLAN ids directly to ISIDs ids in a 1 to 1 relationship throughout the network. Following this practice allows implementing SPB interconnect fabrics spanning multiple data centers with independent VLAN addressing schemas in each data center. This practice is proven especially effective during data center consolidation efforts. Recommendation 3: It has been seen that using SPB to keep Spanning Tree regions local to access networks (therefore reducing the impact of network changes) or bringing SPB all the way out to the user access switches, thus removing the need for Spanning Tree all together, has significantly improved end user experience and at the same time simplified network implementations and deployments. Where possible it is recommended to use SPB as a Spanning Tree protocol replacement. Recommendation 4: Besides the need for L2 traffic virtualization for hypervisor migrations, most deployments were also required to route virtualized traffic between IP subnets/broadcast domains which are provided with an SPB service. Routing between services (ISIDs) can be done with dedicated routers external to the SPB fabric, but there is a performance and deployment advantage if SPB nodes route traffic between services similar to traditional routing switches that are able to perform routing between VLANs/IP subnets in band (on SPB interconnect links). With this approach IEEE 802.1aq based routing deployment models are similar to traditional IEEE 802.1Q based networks, with the advantage that the routing instance can easily be migrated to any fabric node by extending the service accordingly. Lapuh, R. Expires September 30, 2014 [Page 5] INTERNET DRAFT SPB Deployment Recommendations March 5th 3. Infrastructure Configuration Recommendations 3.1 IS-IS SYSTEM ID AND SPB NICKNAME CONFIGURATION As of this writing the IEEE SPB standard defines a single ISIS area for an SPB region, even though large SPB regions can be defined and operated. In the future this will likely be extended to multi-area support. Recommendation 5: It is a good practice to manually configure System IDs and SPB Nicknames with a simple identification scheme, coordinating the system ID numerically with the SPB Nickname for ease of troubleshooting. It is also a good practice to define an area per SPB region. It is recommended to use the upper 24 bits to indicate the area ID. For example, System IDs start with 00.49.bb00.1000 for the first node, 0049.bb00.2000 for the second node and so on. In these System IDs, 0049 indicates a private address, the "00bb" indicates area "00bb", and 1000, 2000, etc., indicate the node number (1 through n). It is also a good practice to keep the first two and last two nibbles at 0 so that the System ID looks like this: 00xx.xxxx.xx00. This is to avoid interfering with the multicast bit as well as other locally assigned MAC addresses, especially if the system ID is manually derived from the chassis MAC address. These System IDs correspond to SPB Nicknames of 1.bb.10, 1.bb,20, 1.bb.30 for nodes 1, 2 and 3 respectively, and so on. Using manually configured BMAC addresses and also coordinating the BMAC with the System ID and SPB Nickname enhances-, ease of identification for management and troubleshooting as well as making possible future PBB- EVPNs. As an additional option, with the goal to reduce configuration tasks, System-IDs could be automatically inherited from the systems chassis MAC addresses. The SPB Nickname could also be derived from the lower bits of the chassis MAC. This approach could be targeted for SPB access switches where a simplified deployment model would be of most interest. 3.2 SPB NETWORK LINKS Details on Recommendation 1: SPB Fabric inter-connections in the preceding SPB deployments are all based on point-to-point Ethernet links, optical CWDM/DWDM connections or some sort of transparent E-LINE service. By avoiding connecting SPB over a shared segment (or E-LAN) failure detection and network Lapuh, R. Expires September 30, 2014 [Page 6] INTERNET DRAFT SPB Deployment Recommendations March 5th convergence times have been kept very low. Failure detection and recovery is thus not dependent on IS-IS hello-multiplier intervals but triggered by lower layer protocols. Such E-LINE services (to interconnect SPB nodes) can be based on any type of transparent Ethernet service (WDM, MPLS or PBB based), as long as they are loop free and the service Maximum Transmission Unit (MTU) size allows for a minimum of [MTU] 1544 bytes for non Jumbo Frames. Tagged IP: = 1522 = 1500(IP MTU)+ 2(Ethertype)+ 12(MAC SA/DA) + 4(TAG) + 4 (CRC) and MacInMac Header = 22 bytes. On dark-fiber based Ethernet connections, link failures can be detected by the Ethernet remote fault detection mechanisms; however, on service provider based links, there can be multiple active components between two SPB nodes, and thus not all failures can be detected by monitoring link status indications. To ensure quick fail- over times across an E-LINE service, an end-to-end connectivity check mechanism such as 802.1ag based Connectivity Check Mechanism (CCM), or similar, is recommended. 3.3 SPB AND SPANNING TREE Details on Recommendation 3: Many networks today are still operating with some sort of Spanning Tree (MSTP/RSTP or proprietary versions). SPB can be leveraged to separate Spanning Tree regions into smaller independent domains. Therefore a Spanning Tree root bridge change impacts smaller regions only and is not spread across the whole network. Keeping root bridge elections and the effect of Topology Change Notifications local has proven a significant improvement of network availability in larger Spanning Tree deployments. Even though SPB allows extending Spanning Tree (MSTP) domains over an SPB region by sharing a MSTP-CST tree, it is only recommended keeping MSTP enabled on SPB NNI links if those links also need to transport non-SBP VLANs in parallel to SPB based L2 services. Result of keeping MSTP enabled on SPB NNI links will be, that all VLANs which share the same MSTP-CST will be affected by a CST topology change. Disabling MSTP on SPB NNI links will avoid any participation of SPB services in the CST state transitions and thus will increase network availability. Lapuh, R. Expires September 30, 2014 [Page 7] INTERNET DRAFT SPB Deployment Recommendations March 5th 3.4 SPB FABRIC CONFIGURATION Recommendation 6: SPB is part of the IEEE standard framework, thus it is fully interoperable and compatible with all other IEEE standards. Therefore SPB can be added to any IEEE standards based network infrastructure without changing existing configurations, as long as there is no VLAN ID overlap. In an SPB network the Backbone VLAN IDs (BVIDs) are used to separate and load-spread SPB traffic across multiple paths. The 802.1aq standard defines up to 16 BVIDs. These BVIDs need to be consistently configured across the SPB region. The BVIDs can be selected out of the available VLAN range [1-4095], however, using a pre-defined set of VLANs is recommended. Usually the lowest 4000 IDs are used by customers for network access VLAN configurations; thus it has been seen as a good practice to use BVLAN numbering that is in the highest upper addressable range, e.g, starting with 4051 for the primary BVLAN and all switches and 4052 to 4066 for the subsequent ones. It is recommended to use at least two BVIDs for load-spreading reasons. 3.5 SPB SERVICES MAPPING Details on Recommendation 2: When network virtualization is extended between regions, for example, to support for Virtual Machine movements, it is very important to use a unique virtualization index to achieve this. SPB, with its 802.1ah based Service-ID (I-SID) provides an inherent virtualization technique which allows local VLAN significance and using the I-SID as a global virtualization index. This is especially true in VMWare deployments where it is advantageous for the Portgroup IDs of VCenter instances to correspond with the VM VLAN memberships. With SPB, it is thus easily possible to run a hosted environment with multiple VCenter instances in parallel on the same infrastructure, without having any VLAN space interference. In the preceding virtual Data Center deployment where multiple domains are interconnected, the VLAN spaces can be kept independent of each other, and the virtualization is achieved by the usage of the 16 million I-SIDs. 3.6 SPB AND IP ROUTING Details on Recommendation 4: In an SPB network the typical size of broadcast domains of user and server subnets are not being changed from what one is used to with traditional technologies. This means that there is always a need for Lapuh, R. Expires September 30, 2014 [Page 8] INTERNET DRAFT SPB Deployment Recommendations March 5th routing functionality. The best case is if a SPB node can directly route individual IP subnets which consist of I-SIDs, similar to how those nodes can route VLAN based IP subnets. Optimally this routing should be available within the SPB Ethernet fabric between I-SID based services. In order to provide IP routing redundancy for access networks, VRRP or similar technologies should be available on fabric nodes at the same time. 4. Standard Improvement Recommendations Recommendation 7: In an SPB network, link-state update propagation is achieved by SPB nodes re-laying topology change notifications through IS-IS on a hop by hop basis in the control plane. With an assumption of 10ms relay latency per node, a ring of 20 SPB nodes could see up to 100ms of propagation latency to reach all nodes in the ring. As an optimization of SPB, the default L2 service instance described in IEEE 802.1aq could be used to flood all propagation changes into this default service, reducing the propagation delay in this example from 100ms to 10ms. The SPB IEEE standard could be enhanced to also include this default-flooding behavior. 5. OA&M Recommendation 8: In all deployment experiences, the use of L2 based OAM capabilities have been invaluable in managing the network. It is recommended that IEEE 802.1ag based connectivity check mechanisms: Layer 2 Ping, Layer 2 Traceroute and Layer 2 Tracetree are being implemented. 6. Tenant Separation Considerations and Payment Card Industry Security Requirements Customer, tenant and application separation are key requirements in provider or cloud hosting solutions, however are also becoming a key requirement in any Enterprise network where credit card transactions are being relayed. SPB separates any type of traffic at the edge of the SPB region into its own service instance (I-SID). Classification into the I-SID can be done based on port, vlan, a combination of port/vlan or any other unique packet identifier. Once a customer- or Lapuh, R. Expires September 30, 2014 [Page 9] INTERNET DRAFT SPB Deployment Recommendations March 5th application traffic is classified into an I-SID, it is kept separate until it exits the SPB region, very similar to MPLS with its tunnel and service labels. In SPB the "tunnel label" is comprised of the BMAC pair and the "service label", which is the I-SID. Thus SPB is as secure as any other packet switching solution. Today there are many service provider based networks in production using the same 802.1ah (PBB) packet encapsulation methods as SPB is using. 7 Deployment Experiences 7.1 DEPLOYMENT SCENARIO A SPB AS INTER-DATACENTER-FABRIC FOR DC REDUNDANCY OR DC MIGRATIONS Typically in a large enterprise core, it is not viewed as good practice to extend L2 broadcast domains across the backbone network. However, with the advent of server virtualization, it has become a common requirement to extend server VLAN segments between geo- redundant Data Centers to dynamically, efficiently and cost effectively leverage the ability to perform Virtual Machine migrations and run load balancing techniques across multiple Data Centers. The deployment of SPB as a data center connect allows the following challenges to be addressed: In many cases SPB can be deployed on the existing network architecture with IS-IS running side by side and independently from other routing protocols such as OSPF. IS-IS or OSPF is being used to populate the IP routing table and provide L3 routed connectivity. IS- IS is being used for SPB, bringing the ability to extend server VLANs across the backbone. Typically the server VLANs to be extended across the network are locally configured within the Data Centers, on the server access (i.g. Top of Rack) switches, the server VLANs are assigned to a service ID (I-SID) and then extended across the SPB network, thus becoming available in any other server access switch in the local or if required remote data center. The distribution nodes can act as a routing gateway for the server VLANs. VRRP is also being used to create a single default gateway IP address for the server VMs. The VRRP instance per Server IP-subnet thus exists on all distribution nodes and provides redundant and distributed default gateway functionality. Core failure recovery times in the SPB region can be kept well below 1 second and L3 recovery times, depends on the configured VRRP timers. 7.2 DEPLOYMENT SCENARIO B Lapuh, R. Expires September 30, 2014 [Page 10] INTERNET DRAFT SPB Deployment Recommendations March 5th SPB TO RE-ARCHITECT SECURITY ZONES There are other valid reasons why it might be necessary to extend L2 segments across the enterprise core. A good example is a major manufacturing plant which has a very rigorous design based on a pure IP routed architecture with a strong focus on firewalling different parts of the network. This is achieved by physically wedging firewalls within the physical topology in such a way as to deny any unwanted interaction between different network zones. The security provided by this model has to be offset by the rigidity it imposes in terms of where devices are allowed to be connected to the network. In this particular example, connecting devices in locations where they were not initially intended to be located was addressed by laying additional cabling, with the costs and delays that this involves. Once deployed, SPB brought to this model the ability to decouple the physical infrastructure from the logical connectivity running above it. This means that it is no longer necessary to wedge firewalls into the physical topology to intercept traffic, but rather let SPB force L2 VLANs to reach the desired firewalls, wherever those firewalls might be located on the network. It is now possible to connect devices anywhere on the physical network infrastructure and simply connect these devices to the VLAN segment to which they need to belong. 7.3 DEPLOYMENT SCENARIO C SPB FOR CAMPUS VIRTUALIZATION Another example of where it is useful to extend L2 segments can be found in the health care vertical. An operational challenge, typical of most hospitals, is to be able to support network connectivity for mobile medical equipment which typically needs to connect to a server application hosted in the Data Center. The real challenge with this equipment is often the fact that it is supplied and maintained by separate, often external, technicians with little or no IP skills. As such this equipment is usually not able, or not configured, to use DHCP and instead uses a single flat IP subnet which encompasses the mobile units as well as the server application in the Data Center. The hospital's network team essentially has limited control over the IP configuration of these devices and hence a desire to segregate such applications within a constrained L2 service. By deploying SPB L2 instances, it is now possible to much more easily manage such applications. 7.4 DEPLOYMENT SCENARIO D Lapuh, R. Expires September 30, 2014 [Page 11] INTERNET DRAFT SPB Deployment Recommendations March 5th SPB AS MULTI-TENANT FABRIC SOLUTION In a multi-tenant deployment SPB was leveraged to provide secured and separated services for several tenants. In this implementation SPB leverages 10 Gigabit Ethernet heavily. In the two geo-disbursed data centers LAN and IP connectivity is utilized in a way that makes both appear as one virtual data center. A common 3-tier design is utilized for the entire network. There may be multiple tenants per edge which are then segregated into their own private broadcast domain. Over 500 L2 services are spread across the network providing IP subnet connectivity to any of the tenants. At the data center those IP subnets are assigned to over a dozen of Virtual Router Forwarding (VRF)instances corresponding to their security requirements. VRRP is used to provide router redundancy. Layer 3 routing between VRF instances, hence between tenants, to external organizations and to the Internet is performed by stateful firewalls. This simplified model utilizing Layer 2 I-SIDs, including routing between service instances, across a common SPB backbone allows this solution provider to quickly and effectively extend either Layer 2 services or Layer 3 services to any location in the network for any application. For Voice over IP a Quality of Service (QoS) framework for traffic prioritization has been employed. IP Differentiated Services (DiffServ) EF DSCP and several specific AF DSCP groups are mapped into the appropriate 802.1p priority classes at the SPB BEB nodes to provide the necessary traffic prioritization within the SPB backbone. 8 Security Considerations Security implications of SPB deployments are to be discussed in separate documents. 9 IANA Considerations This document makes no requests to IANA 5 References Lapuh, R. Expires September 30, 2014 [Page 12] INTERNET DRAFT SPB Deployment Recommendations March 5th 5.1 Normative References 5.2 Informative References [RFC6329] Fedyk, D., "IS-IS Extensions Supporting IEEE 802.1aq Shortest Path Bridging" July 2011 [SPBWIKI] http://en.wikipedia.org/wiki/Shortest_Path_Bridging Authors' Addresses Roger Lapuh (editor) Avaya Wallisellen, 8304 Switzerland EMail: rogerlapuh@avaya.com Paul Unbehagen Avaya 1300 W. 120th Avenue Westminster, CO 80234 USA Email: unbehagen@avaya.com Peter Ashwood-Smith (editor) Huawei Technologies Canada Ltd. 303 Terry Fox Drive, Suite 400 Kanata, Ontario, K2K 3J1 CANADA EMail: Peter.AshwoodSmith@huawei.com Phillip Taylor Leeds Metropolitan University 404 Portland Building Calverley Street Leeds, LS1 3HE, UK Email: p.p.taylor@leedsmet.ac.uk Steven Emert Avaya 225 South Sixth Street, Suite 4350 Lapuh, R. Expires September 30, 2014 [Page 13] INTERNET DRAFT SPB Deployment Recommendations March 5th Minneapolis, Minnesota 55402-4619 US Email: semert@avaya.com Ludovico Stevens (editor) Avaya 25 Allee Pierre Ziller 06560 Valbonne France EMail: ludovicostev@avaya.com Srikanth Keesara Avaya 600 Technology Park Billerica MA 01821 US EMmail: skeesara@avaya.com Lapuh, R. Expires September 30, 2014 [Page 14]