Internet Engineering Task Force Dave Cooper, Global Crossing INTERNET-DRAFT Blaine Christian, UUNET TEWG Working Group Daniel Awduche Expiration: August 2001 Movaz Communications Vijay Gill Metromedia Fiber Alan Hannan, RoutingLoop Jim Boyle, Level 3 February 2001 Applicability Statement for Traffic Engineering with MPLS draft-many-tewg-te-applicability-00.txt Status of this Memo This document is an Internet-Draft and is in full conformance with all provisions of Section 10 of RFC2026. Internet-Drafts are working documents of the Internet Engineering Task Force (IETF), its areas, and its working groups. Note that other groups may also distribute working documents as Internet- Drafts. Internet-Drafts are draft documents valid for a maximum of six months and may be updated, replaced, or obsoleted by other documents at any time. It is inappropriate to use Internet- Drafts as reference material or to cite them other than as "work in progress." The list of current Internet-Drafts can be accessed at http://www.ietf.org/ietf/1id-abstracts.txt The list of Internet-Draft Shadow Directories can be accessed at http://www.ietf.org/shadow.html. Abstract This memo describes the applicability of Multiprotocol Label Switching (MPLS) to traffic engineering in IP networks. Special considerations for deployment of MPLS for traffic engineering in operational contexts are discussed and the limitations of the MPLS approach to traffic engineering are highlighted. This document is intended for the Internet informational track. Cooper et. al. [Page 1] Internet Draft draft-many-tewg-te-applicability-00.txt February 2001 1. Introduction It is generally acknowledged that one of the most significant initial applications of Multiprotocol Label Switching (MPLS) is traffic engineering [1][2] in IP networks. A significant community of IP service providers have found that traffic engineering of their traffic can have tactical and strategic value [2][3][4]. To support traffic engineering application, extensions have been specified for IS-IS and OSPF [5][6], and to signaling protocols RSVP and LDP [7][8]. The extensions for ISIS, OSPF and RSVP have all been developed and deployed in scale, and in a multi-vendor environment. This memo discusses the applicability of Traffic Engineering to Internet Service Provider networks. It augments protocol and operational statements on applicability and concern made in relation to the use of RSVP-TE in particular [9]. Special considerations for deployment of MPLS in operational contexts are discussed and the limitations of this approach to traffic engineering are highlighted. 2. Technical Overview of ISP traffic engineering Traffic Engineering in this context consists of mapping IP traffic flows onto the existing physical topology. Techniques used to accomplish this include but are not limited to: (1) IGP Cost (Metrics) (2) Explicit Routing using constrained virtual circuit switching techniques such as ATM or Frame Relay SPVCs. (3) Explicit Routing using constrained path setup techniques such as MPLS This document primarily concerns itself with MPLS techniques. Specifically, dealing with the ability to move traffic away from the shortest paths selected by the IGP and onto other paths to avoid congestion in the network. This is achieved by the use of explicitly signalled LSP-tunnels. The explicit route to be used may be computed offline and configured down into the router, or characteristics of the LSP may be configured into the router, which will then use SPF techniques subject to the constraints of the characteristics. These characteristics can include: o) Bandwidth required o) Priority o) Nodes to include or exclude in the path o) Affinities to include or exclude in the path Affinities are arbitrary, provider assigned attributes applied to links and carried in the Traffic Engineering extensions for the Cooper et. al. [Page 2] Internet Draft draft-many-tewg-te-applicability-00.txt February 2001 IGPs. They can be used to affect certain policies, or route preferences. They are unique to MPLS. 3. Applicability of Internet Traffic Engineering As mentioned in [2] and [7], traffic engineering with MPLS is appropriate to maintain explicitly routed paths in an IP network for traffic placement. LSP-tunnels can be used to forward subsets of traffic through paths that are independent of routes computed by conventional Interior Gateway Protocol (IGP) Shortest Path First (SPF) algorithms. This allows Network Operators significant flexibility in directing traffic flows and allows policies to be implemented that can result in the performance optimization of networks. Examples of where use of MPLS for traffic engineering in ISP environments is applicable are given below. The uses are certainly not restricted to these. 3.1 Avoidance of congested resources In order to lower the utilization on congested link(s), an operator may have a variety of approaches to routing a subset of traffic on that link onto less congested topology elements. Operators who do not make extensive use of LSP tunnels might create an LSP between two points known to be a source of the congestion and explicitly route this traffic along another path. Operators who do make extensive use of LSP tunnels, either for measurement or automated traffic control, might also choose to explicitly route a subset of the LSPs onto other paths. This can be employed as a quick response when the bandwidth signalled by the LSPs does not accurately match the actual traffic on the LSPs, yet the provider determines that changing the bandwidth parameters is not time effective or of lasting impact. Approaches that entail measurement of traffic loads on LSPs and using these to configure explicit routes for LSPs (also known as offline traffic engineering [10]) or solely the bandwidths and other parameters such as bandwidth, priority and/or affinities (online traffic engineering) can be used strategically to react to periods of congestion in a network. This congestion may occur as a result of equipment or facility failure, slower than expected provisioning cycles for new circuits, unexpected surges in traffic levels and even to effectively utilize a more geographically meshed topology due to size of available routing or switching equipment relative to demand. 3.2 Utilization of parallelized topologies Transoceanic connectivity often comes in numerous smaller circuits, such as NxDS3, NxSTM-1. This can also be the case into bandwidth constrained cities for continental networks. Effectively distributing the load across these circuits can be attempted by employing MPLS (and often ATM) traffic engineering. Cooper et. al. [Page 3] Internet Draft draft-many-tewg-te-applicability-00.txt February 2001 Approaches include solely using bandwidth signalling, explicitly configuring routes for LSP tunnels, and assigning affinities to parallel elements, and tying LSP tunnels to different affinities. A similar problem and solution approach involves networks which have a replicated topology, such as an NxOC3/12/48 topology. 3.3 Affecting policies for use of certain circuits It is sometimes desirable to get certain traffic on or off certain links in the topology for policy or engineering reasons. Suppose a global service provider has a flat IGP. MPLS TE functionality, in the form of affinities, can be used to keep continental traffic from traversing trans-oceanic resources. Another example of using MPLS TE affinities to exclude certain traffic from a subset of circuits might be to keep inter-regional LSPs off of circuits that are reserved for intra-regional traffic. 4. Implementation Considerations 4.1 Architectural and Operational When deploying traffic engineering in a service provider environment, the network impact of administrative policies and the selection of nodes that will serve as endpoints for LSP-tunnels should be carefully thought through and understood. Furthermore, when devising a virtual topology for LSP-tunnels, special consideration should be given to the tradeoff between the operational complexity associated with a large number of LSP-tunnels and the control granularity that large numbers of LSP-tunnels allow. Stated otherwise, a large number of LSP-tunnels allow greater control over the distribution of traffic across the network, but increases network operational complexity. In large networks, it may be advisable to start with a simple LSP-tunnel virtual topology and then introduce additional complexity based on observed or anticipated traffic flow patterns. Administrative policies should guide the amount of bandwidth to be allocated. One may choose to set the bandwidth of a particular LSP to the measured peak, or a particular Percentile or Quantile of observed utilization over a determined period. Sufficient over subscription (of LSPs) or under reporting bandwidth on actual links should be used to account for flows that exceed their normal limits on an event driven basis. Flows should be monitored for trends that indicate when a LSP should be resized for additional bandwidth required. Flows should be monitored dynamically for extreme variance from expected levels. Should an unpoliced flow greatly exceed its subscription factor, efforts should be made to determine the cause and remedy the problem. Policing flows is an effective method for containing congestion to flows that are in need of remediation. When creating LSPs, a hierarchical network approach can be used to Cooper et. al. [Page 4] Internet Draft draft-many-tewg-te-applicability-00.txt February 2001 alleviate problems associated with full meshes. In general, operational experience has shown that large macro flows (city pair) are long-lived and tractable, while smaller flows (edge to edge) are not. Using a TE Core to aggregate the edge of the network maps nicely into this hierarchy. Over aggregation of flows can result in a flow so large that precludes the constraint based routing entity from finding a path through a network. Splitting the flow by using two or more parallel LSPs to map the flow onto can solve this problem at the expense of introducing more state. Failure scenarios should also be addressed when creating a flow over several links. It is of little value to establish a finely balanced set of flows over a set of links only to find that upon link failure the balance reacts poorly, or does not revert upon circuit restoration. 4.1 Network Management Aspects Networks planning to deploy MPLS for Traffic Engineering must consider the performance and fault management aspects of Network Management. With the addition of the MPLS component to any infrastructure, complex operations are added which require constant monitoring to ensure they do not impede the performance of end-to-end traffic delivery. In addition, traffic characteristics, such as latency across an LSP, must be assessed on a regular basis to ensure service-level guarantees are achieved. Attaining information on LSP behavior is critical in determining the volatility of a MPLS network. When LSPs transition or path changes occur, packets may be dropped which results in a significant impact on network performance. It should be the goal of any network deploying MPLS to minimize the volatility of LSPs and reduce the root causes that induce this volatility. Unfortunately, there are very few, if any, NMS systems deployed capable of providing such root cause analysis and must be accomplished using in-house code development. This creates a significant constraint in deploying wide-scale MPLS networks. Performance of an MPLS network is also highly dependent on the configured values of bandwidth for each LSP. Since congestion is a common cause of performance degradation, it is important to proactively avoid these situations. While MPLS was designed to minimize congestion on links by utilizing bandwidth reservations, it is still heavily reliant on user configurable data. If the LSP bandwidth value does not adequately represent the traffic flows of that LSP, over utilization may occur and cause significant congestion within a network. Therefore it is imperative to build and maintain an adequate statistical modeling tool for determining LSP bandwidth size. Failure to do so may result in sub optimal network performance. Cooper et. al. [Page 5] Internet Draft draft-many-tewg-te-applicability-00.txt February 2001 5. Limitations Sufficiently understanding how this technology works and identifying problems with implementations can take significant engineering and testing resources. Initial deployment of code and configurations to support traffic engineering can consume significant engineering, operation and system development resources. Developing automated systems to create configurations for network elements can require significant software development and hardware resources. Getting to a point where configurations for routers are updated in an automated fashion can be a time consuming process. Tracking manual tweaks to router configurations, or problems associated with these can be an endless task. Certain architectural choices can lead to operational, protocol and router implementation scalability problems. This is especially the case as the number of LSP tunnels or configuration data in a network increases, which can be exasperated by designs incorporating meshes, which create N^2 LSPs. In these cases minimizing N through hierarchy, regionalization, or proper selection of tunnel termination points can affect the network's ability to scale. Loss of scale in this sense can be via protocol instability, inability to change network configurations to accommodate growth, inability for router implementations to be updated, hold or properly process configurations or loss of ability to adequately manage the network. Although widely deployed, this is new technology when compared to standard fares such as ISIS, OSPF and BGP. It also has more configuration and protocol options. As such, implementations are less battle-hardened and automated testing of various configurations are not as thorough. Multivendor environments are achievable, though not without additional reconciliation effort. Common approaches of traffic engineering in service provider environments switch the forwarding paradigm from connectionless to connection oriented. Thus operational analysis of the network may be complicated in some regards (and improved in others). Inconsistencies in forwarding state result in dropped packets whereas with connectionless the packet will either loop and drop, or be misdirected onto another branch in the routing tree. Currently deployed MPLS TE approaches can be adversely affected by both internal and external router and link failures. This can create a mismatch between the signalled capacity and the traffic an LSP tunnel carries. Many routers in service provider environments are already under stressful processing and operational loads just running IGP, BGP and IPC. Enabling traffic engineering in an MPLS environment involves adding traffic engineering databases and processes, adding additional information to be carried by the routing Cooper et. al. [Page 6] Internet Draft draft-many-tewg-te-applicability-00.txt February 2001 processes, and adding signalling state and processing to these network elements. In some environments, this additional load may not be feasible. MPLS in general and MPLS-TE in particular is not a panacea for lack of network capacity, or lack of proper capacity planning and provisioning. MPLS-TE generally causes network traffic to traverse more network elements as well as incur greater latency. Generally this added inefficiency is done to prevent shortcomings in capacity planning execution or available resources path to avoid hot spots. The ability of traffic engineering to accommodate more traffic on a given topology can also be characterized as a short term gain during periods of persistent traffic growth. Failure to properly capacity plan and execute will lead to congestion, no matter what technological aides are employed. 6. Conclusion The applicability of traffic engineering in Internet Service Provider environments has been discussed in this document. The focus has been on use of MPLS based approaches to achieve traffic engineering in this context. The applicability of Traffic Engineering and associated management and deployment considerations are detailed, as well as limitations noted. The capability to monitor point to point traffic statistics between two routers and the capability to control the forwarding paths of subsets of traffic through a given network topology together make traffic engineering with MPLS applicable and useful for traffic engineering within service provider networks. 7. Security Considerations This document does not introduce new security issues. When deployed in service provider networks, it is mandatory to ensure that only authorized entities are permitted to initiate establishment of LSP-tunnels. 8. Acknowledgments The effectiveness of the MPLS protocols for traffic engineering in service provider networks is in large part due to the experience gained and foresight given by network engineers and developers familiar with traffic engineering with ATM in these environments. In particular, the authors wish to acknowledge the authors of RFC 2702 for the clear articulation of the requirements, as well as the developers and testers of code in deployment today for keeping their focus. 9. References [1] E. Rosen, A. Viswanathan, R. Callon, "A Proposed Architecture for MPLS", RFC3031 Cooper et. al. [Page 7] Internet Draft draft-many-tewg-te-applicability-00.txt February 2001 [2] D. Awduche, J. Malcolm, J. Agogbua, M. O'Dell, J. McManus, "Requirements for Traffic Engineering Over MPLS," RFC 2702, September 1999 [3] X. Xiao, A. Hannan, B. Bailey, L. Ni, "Traffic Engineering with MPLS in the Internet", IEEE Transactions on Networking, March/April 2000 [4] V. Springer, C. Pierantozzi, J. Boyle, "Level3 MPLS Protocol Architecture", work in progress, August 2000. [5] T. Li, H. Smit "IS-IS extensions for Traffic Engineering", work in progress, September 2000. [6] D. Katz, D. Yeung "Traffic Engineering extensions to OSPF", work in progress, September 2000. [7] D. Awduche et. al. "RSVP-TE: Extensions to RSVP for LSP Tunnels", work in progress, August 2000. [8] B. Jamousi, "Constraint-Based LSP Setup using LDP", work in progress, July 2000. [9] D. Awduche, A. Hannan, X. Xiao "Applicability Statement for Extensions to RSVP for LSP-Tunnels", work in progress, April 2000. [10] D. Awduche, A. Chiu, A. Elwalid, I. Widjaja, X. Xiao, "A Framework for Internet Traffic Engineering", work in progress, July 2000. 10. AUTHORS' ADDRESSES Dave Cooper Blaine Christian Global Crossing MCI/WorldCom/UUNET Campus 960 Hamlin Court 22001 Loudoun County Parkway Sunnyvale, CA 94089 Room D1-2-737 (916) 415-0437 Ashburn, VA 20147 email: dcooper@gblx.net email: blaine@uu.net Vijay Gill Daniel Awduche Metromedia Fiber Network Movaz Networks 8075 Leesburg Pike 7926 Jones Branch Drive, Suite 615 Viena, VA 22182 McLean, VA 22102 (410) 262-0660 (703) 847-7250 email: vgill@mfnx.net email: awduche@movaz.com Alan Hannan Jim Boyle RoutingLoop Intergalactic Level3 Communications 112 Falkirk Court 1025 Eldorado Boulevard Sunnyvale, CA 94087 Broomfield, CO 80021 (408) 666-2326 (720) 888-1192 email: alan@routingloop.net email: jboyle@level3.net [Page 8]