Internet Engineering Task Force Dave Cooper, Global Crossing
INTERNET-DRAFT Blaine Christian, UUNET
TEWG Working Group Daniel Awduche
Expiration: August 2001 Movaz Communications
Vijay Gill
Metromedia Fiber
Alan Hannan, RoutingLoop
Jim Boyle, Level 3
February 2001
Applicability Statement for Traffic Engineering with MPLS
draft-many-tewg-te-applicability-00.txt
Status of this Memo
This document is an Internet-Draft and is in full conformance with
all provisions of Section 10 of RFC2026.
Internet-Drafts are working documents of the Internet Engineering
Task Force (IETF), its areas, and its working groups. Note that
other groups may also distribute working documents as Internet-
Drafts.
Internet-Drafts are draft documents valid for a maximum of six months
and may be updated, replaced, or obsoleted by other documents at any
time. It is inappropriate to use Internet- Drafts as reference
material or to cite them other than as "work in progress."
The list of current Internet-Drafts can be accessed at
http://www.ietf.org/ietf/1id-abstracts.txt
The list of Internet-Draft Shadow Directories can be accessed at
http://www.ietf.org/shadow.html.
Abstract
This memo describes the applicability of Multiprotocol Label
Switching (MPLS) to traffic engineering in IP networks. Special
considerations for deployment of MPLS for traffic engineering in
operational contexts are discussed and the limitations of the
MPLS approach to traffic engineering are highlighted.
This document is intended for the Internet informational track.
Cooper et. al. [Page 1]
Internet Draft draft-many-tewg-te-applicability-00.txt February 2001
1. Introduction
It is generally acknowledged that one of the most significant
initial applications of Multiprotocol Label Switching (MPLS) is
traffic engineering [1][2] in IP networks. A significant
community of IP service providers have found that traffic
engineering of their traffic can have tactical and strategic value
[2][3][4]. To support traffic engineering application, extensions
have been specified for IS-IS and OSPF [5][6], and to signaling
protocols RSVP and LDP [7][8]. The extensions for ISIS, OSPF and
RSVP have all been developed and deployed in scale, and in a
multi-vendor environment.
This memo discusses the applicability of Traffic Engineering to
Internet Service Provider networks. It augments protocol and
operational statements on applicability and concern made in
relation to the use of RSVP-TE in particular [9]. Special
considerations for deployment of MPLS in operational contexts are
discussed and the limitations of this approach to traffic
engineering are highlighted.
2. Technical Overview of ISP traffic engineering
Traffic Engineering in this context consists of mapping IP traffic
flows onto the existing physical topology. Techniques used to
accomplish this include but are not limited to:
(1) IGP Cost (Metrics)
(2) Explicit Routing using constrained virtual circuit switching
techniques such as ATM or Frame Relay SPVCs.
(3) Explicit Routing using constrained path setup techniques such
as MPLS
This document primarily concerns itself with MPLS techniques.
Specifically, dealing with the ability to move traffic away from
the shortest paths selected by the IGP and onto other paths to
avoid congestion in the network. This is achieved by the use of
explicitly signalled LSP-tunnels. The explicit route to be used
may be computed offline and configured down into the router, or
characteristics of the LSP may be configured into the router,
which will then use SPF techniques subject to the constraints of
the characteristics. These characteristics can include:
o) Bandwidth required
o) Priority
o) Nodes to include or exclude in the path
o) Affinities to include or exclude in the path
Affinities are arbitrary, provider assigned attributes applied to
links and carried in the Traffic Engineering extensions for the
Cooper et. al. [Page 2]
Internet Draft draft-many-tewg-te-applicability-00.txt February 2001
IGPs. They can be used to affect certain policies, or route
preferences. They are unique to MPLS.
3. Applicability of Internet Traffic Engineering
As mentioned in [2] and [7], traffic engineering with MPLS is
appropriate to maintain explicitly routed paths in an IP network
for traffic placement. LSP-tunnels can be used to forward subsets
of traffic through paths that are independent of routes computed
by conventional Interior Gateway Protocol (IGP) Shortest Path
First (SPF) algorithms. This allows Network Operators significant
flexibility in directing traffic flows and allows policies to be
implemented that can result in the performance optimization of
networks. Examples of where use of MPLS for traffic engineering
in ISP environments is applicable are given below. The uses are
certainly not restricted to these.
3.1 Avoidance of congested resources
In order to lower the utilization on congested link(s), an
operator may have a variety of approaches to routing a subset of
traffic on that link onto less congested topology elements.
Operators who do not make extensive use of LSP tunnels might
create an LSP between two points known to be a source of the
congestion and explicitly route this traffic along another path.
Operators who do make extensive use of LSP tunnels, either for
measurement or automated traffic control, might also choose to
explicitly route a subset of the LSPs onto other paths. This can
be employed as a quick response when the bandwidth signalled by
the LSPs does not accurately match the actual traffic on the LSPs,
yet the provider determines that changing the bandwidth parameters
is not time effective or of lasting impact.
Approaches that entail measurement of traffic loads on LSPs and
using these to configure explicit routes for LSPs (also known as
offline traffic engineering [10]) or solely the bandwidths and
other parameters such as bandwidth, priority and/or affinities
(online traffic engineering) can be used strategically to react to
periods of congestion in a network. This congestion may occur as
a result of equipment or facility failure, slower than expected
provisioning cycles for new circuits, unexpected surges in traffic
levels and even to effectively utilize a more geographically
meshed topology due to size of available routing or switching
equipment relative to demand.
3.2 Utilization of parallelized topologies
Transoceanic connectivity often comes in numerous smaller
circuits, such as NxDS3, NxSTM-1. This can also be the case into
bandwidth constrained cities for continental networks.
Effectively distributing the load across these circuits can be
attempted by employing MPLS (and often ATM) traffic engineering.
Cooper et. al. [Page 3]
Internet Draft draft-many-tewg-te-applicability-00.txt February 2001
Approaches include solely using bandwidth signalling, explicitly
configuring routes for LSP tunnels, and assigning affinities to
parallel elements, and tying LSP tunnels to different affinities.
A similar problem and solution approach involves networks which
have a replicated topology, such as an NxOC3/12/48 topology.
3.3 Affecting policies for use of certain circuits
It is sometimes desirable to get certain traffic on or off certain
links in the topology for policy or engineering reasons.
Suppose a global service provider has a flat IGP. MPLS TE
functionality, in the form of affinities, can be used to keep
continental traffic from traversing trans-oceanic resources.
Another example of using MPLS TE affinities to exclude certain
traffic from a subset of circuits might be to keep inter-regional
LSPs off of circuits that are reserved for intra-regional traffic.
4. Implementation Considerations
4.1 Architectural and Operational
When deploying traffic engineering in a service provider
environment, the network impact of administrative policies and the
selection of nodes that will serve as endpoints for LSP-tunnels
should be carefully thought through and understood. Furthermore,
when devising a virtual topology for LSP-tunnels, special
consideration should be given to the tradeoff between the
operational complexity associated with a large number of
LSP-tunnels and the control granularity that large numbers of
LSP-tunnels allow. Stated otherwise, a large number of
LSP-tunnels allow greater control over the distribution of traffic
across the network, but increases network operational
complexity. In large networks, it may be advisable to start with a
simple LSP-tunnel virtual topology and then introduce additional
complexity based on observed or anticipated traffic flow patterns.
Administrative policies should guide the amount of bandwidth to be
allocated. One may choose to set the bandwidth of a particular
LSP to the measured peak, or a particular Percentile or Quantile
of observed utilization over a determined period. Sufficient over
subscription (of LSPs) or under reporting bandwidth on actual
links should be used to account for flows that exceed their normal
limits on an event driven basis. Flows should be monitored for
trends that indicate when a LSP should be resized for additional
bandwidth required. Flows should be monitored dynamically for
extreme variance from expected levels. Should an unpoliced flow
greatly exceed its subscription factor, efforts should be made to
determine the cause and remedy the problem. Policing flows is an
effective method for containing congestion to flows that are in
need of remediation.
When creating LSPs, a hierarchical network approach can be used to
Cooper et. al. [Page 4]
Internet Draft draft-many-tewg-te-applicability-00.txt February 2001
alleviate problems associated with full meshes. In general,
operational experience has shown that large macro flows (city
pair) are long-lived and tractable, while smaller flows (edge to
edge) are not. Using a TE Core to aggregate the edge of the
network maps nicely into this hierarchy.
Over aggregation of flows can result in a flow so large that
precludes the constraint based routing entity from finding a path
through a network. Splitting the flow by using two or more
parallel LSPs to map the flow onto can solve this problem at the
expense of introducing more state.
Failure scenarios should also be addressed when creating a flow
over several links. It is of little value to establish a finely
balanced set of flows over a set of links only to find that upon
link failure the balance reacts poorly, or does not revert upon
circuit restoration.
4.1 Network Management Aspects
Networks planning to deploy MPLS for Traffic Engineering must
consider the performance and fault management aspects of Network
Management. With the addition of the MPLS component to any
infrastructure, complex operations are added which require
constant monitoring to ensure they do not impede the performance
of end-to-end traffic delivery. In addition, traffic
characteristics, such as latency across an LSP, must be assessed
on a regular basis to ensure service-level guarantees are
achieved.
Attaining information on LSP behavior is critical in determining
the volatility of a MPLS network. When LSPs transition or path
changes occur, packets may be dropped which results in a
significant impact on network performance. It should be the goal
of any network deploying MPLS to minimize the volatility of LSPs
and reduce the root causes that induce this volatility.
Unfortunately, there are very few, if any, NMS systems deployed
capable of providing such root cause analysis and must be
accomplished using in-house code development. This creates a
significant constraint in deploying wide-scale MPLS networks.
Performance of an MPLS network is also highly dependent on the
configured values of bandwidth for each LSP. Since congestion is
a common cause of performance degradation, it is important to
proactively avoid these situations. While MPLS was designed to
minimize congestion on links by utilizing bandwidth reservations,
it is still heavily reliant on user configurable data. If the LSP
bandwidth value does not adequately represent the traffic flows of
that LSP, over utilization may occur and cause significant
congestion within a network. Therefore it is imperative to build
and maintain an adequate statistical modeling tool for determining
LSP bandwidth size. Failure to do so may result in sub optimal
network performance.
Cooper et. al. [Page 5]
Internet Draft draft-many-tewg-te-applicability-00.txt February 2001
5. Limitations
Sufficiently understanding how this technology works and
identifying problems with implementations can take significant
engineering and testing resources. Initial deployment of code and
configurations to support traffic engineering can consume
significant engineering, operation and system development
resources. Developing automated systems to create configurations
for network elements can require significant software development
and hardware resources. Getting to a point where configurations
for routers are updated in an automated fashion can be a time
consuming process. Tracking manual tweaks to router
configurations, or problems associated with these can be
an endless task.
Certain architectural choices can lead to operational, protocol
and router implementation scalability problems. This is
especially the case as the number of LSP tunnels or configuration
data in a network increases, which can be exasperated by designs
incorporating meshes, which create N^2 LSPs. In these cases
minimizing N through hierarchy, regionalization, or proper
selection of tunnel termination points can affect the network's
ability to scale. Loss of scale in this sense can be via protocol
instability, inability to change network configurations to
accommodate growth, inability for router implementations to be
updated, hold or properly process configurations or loss of
ability to adequately manage the network.
Although widely deployed, this is new technology when compared to
standard fares such as ISIS, OSPF and BGP. It also has more
configuration and protocol options. As such, implementations are
less battle-hardened and automated testing of various
configurations are not as thorough. Multivendor environments are
achievable, though not without additional reconciliation effort.
Common approaches of traffic engineering in service provider
environments switch the forwarding paradigm from connectionless to
connection oriented. Thus operational analysis of the network may
be complicated in some regards (and improved in others).
Inconsistencies in forwarding state result in dropped packets
whereas with connectionless the packet will either loop and drop,
or be misdirected onto another branch in the routing tree.
Currently deployed MPLS TE approaches can be adversely affected by
both internal and external router and link failures. This can
create a mismatch between the signalled capacity and the traffic
an LSP tunnel carries.
Many routers in service provider environments are already under
stressful processing and operational loads just running IGP, BGP
and IPC. Enabling traffic engineering in an MPLS environment
involves adding traffic engineering databases and processes,
adding additional information to be carried by the routing
Cooper et. al. [Page 6]
Internet Draft draft-many-tewg-te-applicability-00.txt February 2001
processes, and adding signalling state and processing to these
network elements. In some environments, this additional load may
not be feasible.
MPLS in general and MPLS-TE in particular is not a panacea for
lack of network capacity, or lack of proper capacity planning and
provisioning. MPLS-TE generally causes network traffic to traverse
more network elements as well as incur greater latency. Generally
this added inefficiency is done to prevent shortcomings in
capacity planning execution or available resources path to avoid
hot spots. The ability of traffic engineering to accommodate more
traffic on a given topology can also be characterized as a short
term gain during periods of persistent traffic growth. Failure to
properly capacity plan and execute will lead to congestion, no
matter what technological aides are employed.
6. Conclusion
The applicability of traffic engineering in Internet Service
Provider environments has been discussed in this document. The
focus has been on use of MPLS based approaches to achieve traffic
engineering in this context. The applicability of Traffic
Engineering and associated management and deployment
considerations are detailed, as well as limitations noted.
The capability to monitor point to point traffic statistics
between two routers and the capability to control the forwarding
paths of subsets of traffic through a given network topology
together make traffic engineering with MPLS applicable and useful
for traffic engineering within service provider networks.
7. Security Considerations
This document does not introduce new security issues. When
deployed in service provider networks, it is mandatory to ensure
that only authorized entities are permitted to initiate
establishment of LSP-tunnels.
8. Acknowledgments
The effectiveness of the MPLS protocols for traffic engineering in
service provider networks is in large part due to the experience
gained and foresight given by network engineers and developers
familiar with traffic engineering with ATM in these environments.
In particular, the authors wish to acknowledge the authors of RFC
2702 for the clear articulation of the requirements, as well as
the developers and testers of code in deployment today for keeping
their focus.
9. References
[1] E. Rosen, A. Viswanathan, R. Callon, "A Proposed
Architecture for MPLS", RFC3031
Cooper et. al. [Page 7]
Internet Draft draft-many-tewg-te-applicability-00.txt February 2001
[2] D. Awduche, J. Malcolm, J. Agogbua, M. O'Dell, J. McManus,
"Requirements for Traffic Engineering Over MPLS," RFC 2702,
September 1999
[3] X. Xiao, A. Hannan, B. Bailey, L. Ni, "Traffic Engineering with
MPLS in the Internet", IEEE Transactions on Networking, March/April
2000
[4] V. Springer, C. Pierantozzi, J. Boyle, "Level3 MPLS Protocol
Architecture", work in progress, August 2000.
[5] T. Li, H. Smit "IS-IS extensions for Traffic Engineering", work
in progress, September 2000.
[6] D. Katz, D. Yeung "Traffic Engineering extensions to OSPF",
work in progress, September 2000.
[7] D. Awduche et. al. "RSVP-TE: Extensions to RSVP for LSP
Tunnels", work in progress, August 2000.
[8] B. Jamousi, "Constraint-Based LSP Setup using LDP", work in
progress, July 2000.
[9] D. Awduche, A. Hannan, X. Xiao "Applicability Statement for
Extensions to RSVP for LSP-Tunnels", work in progress, April 2000.
[10] D. Awduche, A. Chiu, A. Elwalid, I. Widjaja, X. Xiao, "A
Framework for Internet Traffic Engineering", work in progress, July
2000.
10. AUTHORS' ADDRESSES
Dave Cooper Blaine Christian
Global Crossing MCI/WorldCom/UUNET Campus
960 Hamlin Court 22001 Loudoun County Parkway
Sunnyvale, CA 94089 Room D1-2-737
(916) 415-0437 Ashburn, VA 20147
email: dcooper@gblx.net email: blaine@uu.net
Vijay Gill Daniel Awduche
Metromedia Fiber Network Movaz Networks
8075 Leesburg Pike 7926 Jones Branch Drive, Suite 615
Viena, VA 22182 McLean, VA 22102
(410) 262-0660 (703) 847-7250
email: vgill@mfnx.net email: awduche@movaz.com
Alan Hannan Jim Boyle
RoutingLoop Intergalactic Level3 Communications
112 Falkirk Court 1025 Eldorado Boulevard
Sunnyvale, CA 94087 Broomfield, CO 80021
(408) 666-2326 (720) 888-1192
email: alan@routingloop.net email: jboyle@level3.net
[Page 8]