Internet DRAFT - draft-livingood-low-latency-deployment
draft-livingood-low-latency-deployment
Independent Stream J. Livingood
Internet-Draft Comcast
Intended status: Informational 15 December 2022
Expires: 18 June 2023
Comcast ISP Low Latency Deployment Design Recommendations
draft-livingood-low-latency-deployment-01
Abstract
The IETF's Transport Area Working Group (TSVWG) has finalized
experimental RFCs for Low Latency, Low Loss, Scalable Throughput
(L4S) and new Non-Queue-Building (NQB) per hop behavior. These
documents do a good job of describing a new architecture and protocol
for deploying low latency networking. But as is normal for many such
standards, especially those in experimental status, certain design
decisions are ultimately left to implementers. This document
explores the potential implications of key deployment design
decisions and makes recommendations for those decisions that may help
drive adoption.
Status of This Memo
This Internet-Draft is submitted in full conformance with the
provisions of BCP 78 and BCP 79.
Internet-Drafts are working documents of the Internet Engineering
Task Force (IETF). Note that other groups may also distribute
working documents as Internet-Drafts. The list of current Internet-
Drafts is at https://datatracker.ietf.org/drafts/current/.
Internet-Drafts are draft documents valid for a maximum of six months
and may be updated, replaced, or obsoleted by other documents at any
time. It is inappropriate to use Internet-Drafts as reference
material or to cite them other than as "work in progress."
This Internet-Draft will expire on 18 June 2023.
Copyright Notice
Copyright (c) 2022 IETF Trust and the persons identified as the
document authors. All rights reserved.
Livingood Expires 18 June 2023 [Page 1]
Internet-Draft ISP L4S Deployment Design December 2022
This document is subject to BCP 78 and the IETF Trust's Legal
Provisions Relating to IETF Documents (https://trustee.ietf.org/
license-info) in effect on the date of publication of this document.
Please review these documents carefully, as they describe your rights
and restrictions with respect to this document.
Table of Contents
1. Introduction . . . . . . . . . . . . . . . . . . . . . . . . 2
2. A New Understanding of Application Needs . . . . . . . . . . 3
3. New Thinking on Low Latency Packet Processing . . . . . . . . 4
4. Network Neutrality and Low Latency Networking . . . . . . . . 5
4.1. Prioritization: Same for All Traffic . . . . . . . . . . 6
4.2. Thoughput: Shared Across All Traffic . . . . . . . . . . 6
4.3. A New Network Capability - For All Networks . . . . . . . 6
5. Recommended Deployment Design Decisions . . . . . . . . . . . 7
5.1. Only Applications Mark Traffic . . . . . . . . . . . . . 7
5.2. All Application Providers Welcome . . . . . . . . . . . . 8
5.3. End User CPE Choice . . . . . . . . . . . . . . . . . . . 8
5.4. Opt Out Capability . . . . . . . . . . . . . . . . . . . 9
5.5. Consider Traffic Protection . . . . . . . . . . . . . . . 9
5.6. Avoid Remarking of DSCP Values . . . . . . . . . . . . . 9
6. Summary of Recommended Deployment Design Decisions . . . . . 10
7. Acknowledgements . . . . . . . . . . . . . . . . . . . . . . 10
8. IANA Considerations . . . . . . . . . . . . . . . . . . . . . 10
9. Security Considerations . . . . . . . . . . . . . . . . . . . 10
10. Privacy Considerations . . . . . . . . . . . . . . . . . . . 10
11. Revision History . . . . . . . . . . . . . . . . . . . . . . 11
12. Open Issues . . . . . . . . . . . . . . . . . . . . . . . . . 11
13. Informative References . . . . . . . . . . . . . . . . . . . 11
Author's Address . . . . . . . . . . . . . . . . . . . . . . . . 13
1. Introduction
The IETF's Transport Area Working Group (TSVWG) has finalized
experimental RFCs for Low Latency, Low Loss, Scalable Throughput
(L4S) and Non-Queue-Building (NQB) per hop behavior
[I-D.ietf-tsvwg-l4s-arch] [I-D.ietf-tsvwg-aqm-dualq-coupled]
[I-D.ietf-tsvwg-ecn-l4s-id] [I-D.ietf-tsvwg-l4sops]
[I-D.ietf-tsvwg-nqb] [I-D.ietf-tsvwg-dscp-considerations]. These
documents do a good job of describing a new architecture and protocol
for deploying low latency networking. But as is normal for many such
standards, especially those in experimental status, certain design
decisions are ultimately left to implementers.
This document explores the potential implications of key deployment
design decisions and makes recommendations for those decisions that
may help drive adoption. In particular, there are best practices
Livingood Expires 18 June 2023 [Page 2]
Internet-Draft ISP L4S Deployment Design December 2022
based on prior experience as a network operator that should be
considered and there are network neutrality types of considerations
as well. These technologies are benign on their own, but the way
they are operationally implemented can determine whether they are
ultimately perceived positively and adopted by the broader Internet
ecosystem. That is a key issue for low latency networking, because
the more applications developers and edge platforms that adopt new
packet marking for low latency traffic, then the greater the value to
end users, so ensuring it is received well is key to driving strong
initial adoption.
It is worth stating though that these decisions are not embedded in
or inherent to L4S and NQB per se, but are decisions that can change
depending upon differing technical, regulatory, business or other
requirements. Even two network operators with the same type of
access technology in the same market area may choose to implement in
different ways. Nevertheless, this document suggests that certain
specific deployment decisions can help maximize the value of low
latency networking to both users and network operators.
It is also apparent from the IETF's work that it is clear that nearly
all modern application types need low latency to some degree and that
applications are best positioned to express their needs via
application code and packet marking. Furthermore, unlike with
bandwidth priority on a highly/fully utilized link, low latency using
these new approaches is not a zero sum game - everyone can
potentially have lower latency at no one else's expense (with some
caveats - see Section 3).
For additional background on latency and why latency matters so much
to the Internet, please read [BITAG]
2. A New Understanding of Application Needs
In the course of working to improve the responsiveness of network
protocols, the IETF concluded with their L4S and NQB work that there
were fundamentally two types of Internet traffic and that these two
major traffic types could benefit from having separate network
processing queues in order to improve the way the Internet works for
all applications, and especially for interactive applications.
Livingood Expires 18 June 2023 [Page 3]
Internet-Draft ISP L4S Deployment Design December 2022
One of the two major traffic types is Queue Building (QB) - things
like file downloads and backups that are designed utilize as much
network capacity as possible but for which users are not interacting
with in real-time. The other was Non-Queue-Building (NQB) - such as
DNS lookups, voice interaction with artificial intelligence (AI)
assistants, video conferencing, gaming, and so on. NQB flows tend to
be limited in their capacity needs - for example a DNS lookup will
not need to consume the full capacity of an end user's connection -
but the end user is highly sensitive to any delays.
Thus, the IETF created specifications for how two different network
processing queues can be designed and operated. Early results, such
as from the IETF-114 hackathon [IETF-114-Slides], demonstrate that
L4S and NQB (a.k.a. dual queue networking, and simply "low latency
networking" hereafter) can work across a variety of access network
technologies and deliver extraordinary levels of responsiveness for a
variety of applications. It seems likely that this new capability
will enable entirely new classes of applications to become possible,
driving a wave of new Internet innovation, while also improving the
applications people use today.
3. New Thinking on Low Latency Packet Processing
The Introduction says "Furthermore, unlike with bandwidth priority on
a highly/fully utilized link, low latency using these new approaches
is not a zero sum game - everyone can potentially have lower latency
at no one else's expense." But this bears a bit more discussion to
understand more fully.
L4S does *not* provide low latency in the same way as previous
technologies like DiffServ (QoS). That prior QoS approach used
packet prioritization, where it was possible to assign a higher
relative priority to certain application traffic, such as Voice over
IP (VoIP) telephony. This approach could provide consistent and
relatively low latency by assigning high priority to a partition of
the capacity of a link, and then policing the rate of packets using
that partition. For example, on a 10 Mbps link, a high QoS priority
could be assigned to VoIP with a dedicated capacity of 1 Mbps of the
10 Mbps link capacity. The other 9 Mbps would be available to lower
QoS priority, such as best effort general Internet traffic that was
not VoIP.
Livingood Expires 18 June 2023 [Page 4]
Internet-Draft ISP L4S Deployment Design December 2022
But even when QoS was used in this manner, the latency may have been
relatively good but it was not ultra low latency of the sort that low
latency networking (NQB and L4S) can deliver. As well, that QoS
approach is to some extent predicated on an idea that network
capacity is very limited and that links are often highly utilized.
But in today's Internet, it is increasingly the case that there is an
abundance of capacity to end users, which makes QoS approaches
ineffective in delivering ever-lower latency.
The result, as noted in the prior section, has been the role of dual
queue networking. With these approaches, the new low latency packet
processing queue is introduced on one or more links on the end-to-end
path. The internal L4S queuing may still use a sort of internal
prioritization, but this is not QoS in the typical sense because this
is happening on an extremely short timescale - sub-round trip time
(so microseconds or a few milliseconds). A more important and
impactful force at play is the rapid congestion signals that are
exchanged that will cause a sender to dynamically yeild to other
traffic (as if the other traffic had no QoS priority, which it does
not) - which can be thought of as back pressure to signal the sender
to backoff prior to packetloss occuring.
4. Network Neutrality and Low Latency Networking
Network Neutrality (a.k.a. Net Neutrality, and NN hereafter) is a
concept that can mean a variety of things within a country, as well
as between different countries, based on different opinions, market
structures, business practices, laws, and regulations. Generally
speaking, at least in the context of the United States' marketplace,
it has come to mean that Internet Service Providers (ISPs) should not
block, throttle, or deprioritize lawful application traffic, and
should not engage in paid prioritization, among other things. The
meaning of NN can be complex and ever changing, so the specific
details are out of scope for this document. Despite that, NN
concerns certainly bear on the deployment of new technologies by
ISPs, at least in the US, and so should be taken into account in
making deployment design decisions.
It is also possible that there can be confusion for people who are
not deep in this highly technical subject between prioritization,
provisioned end user capacity (throughput), and low latency
networking. As it is envisioned in the design of the protocols, the
addition of a low latency packet processing queue at a network link
is merely a second packet queue and does not mean that this queue is
prioritized or that it has different or greater capacity from the
classic queue. Thus, a low latency queue does not create a so-called
"fast lane" (in the way that this term is used in policy-related
discussions in the U.S. to describe higher than best effort priority
Livingood Expires 18 June 2023 [Page 5]
Internet-Draft ISP L4S Deployment Design December 2022
or greater capacity being assigned to some traffic compared to
default traffic) - but there are certainly other NN considerations in
the operational implementation worth exploring.
4.1. Prioritization: Same for All Traffic
As noted above, a key aspect of NN goals is that traffic to certain
Internet destinations or for certain applications should not be
prioritized over other Internet traffic. This means in practice that
all Internet traffic in an ISP network should be carried at the same
(best effort) priority and that any network management practices
imposed by the network should be protocol (application) agnostic.
Low latency networking is fully consistent with this aspect of NN,
because it is designed so that all traffic is treated on a best
effort (BE) basis in the ISP network (this may not necessarily be the
case for a user's in-home Wi-Fi network due to the particulars of how
the IEEE 802.11 wireless protocol functions at the current time).
In addition, as noted above, unlike with bandwidth priority on a
highly/fully utilized link, low latency is not a zero sum game -
everyone can potentially have lower latency at no one else's expense.
4.2. Thoughput: Shared Across All Traffic
Low latency networking is also consistent with the NN goal of not
creating a fast lane, because the same end user throughput in an ISP
access network is shared between both classic and low latency (L4S/
NQB) queues. Thus, applications do not get access to greater or
different throughput depending on whether or not the leverage low
latency networking.
4.3. A New Network Capability - For All Networks
Ultimately, the emergence of low latency networking represents a
fundamental new network capability that applications can choose to
utilize as their needs dictate. It reflects a new ground truth about
two fundamentally different types of application traffic and
demonstrates that networks continue to evolve in exciting ways.
In addition, this new network capability can be implemented in a
variety of network technologies. For example in access network
technologies this could be implemented in DOCSIS [LLD], 5G
[Ericsson], PON [CTI], and many other types of networks. Anywhere
that a network bottleneck could occur may benefit from this
technology.
Livingood Expires 18 June 2023 [Page 6]
Internet-Draft ISP L4S Deployment Design December 2022
5. Recommended Deployment Design Decisions
Like any network or system, a good deployment design and
configuration matters and can be the difference between a well-
functioning and accepted design and one that experiences problems and
faces opposition. In the context of deploying low latency networking
in an ISP network, this document describes some recommended
deployment design decisions that should help to ensure a deployment
is resilient, well-accepted, and creates the environment for
generating strong network effects. In contrast, creating barriers to
adoption in the early stages through design and policy decisions will
presumably reduce the predicted potential network effect, thus
choking off further investment across the Internet ecosystem, leading
to a vicious circle of decline - and then the potential value is
never realized.
5.1. Only Applications Mark Traffic
Only applications should mark traffic, not the network. This is for
several reasons:
* According to the end-to-end principle, this function is best
delegated to the edge of the network as an architectural best
practice (the edge being the application in this case).
* Application marking maintains the loose coupling between the
application and network layers, eliminating the need for close
coordination between networks and application developers.
* Application developers know best whether their application is
compatible with low latency networking and which aspects of their
traffic flows will or will not benefit.
* Application traffic is almost entirely encrypted, which makes it
very difficult for networks to accurately determine application
protocols and to further infer which flows will benefit from low
latency and which flows may be harmed because they need to build a
queue.
* To correctly utilize L4S, the application needs to use a scalable
congestion control algorithm in order to use the packet marking
for L4S. This is done by using the ECN field of the packet
header, with an ECT(1) marking, according to Section 4.1 of
[I-D.ietf-tsvwg-ecn-l4s-id]. But only the application (not the
network) knows what congestion control it is using. So, with L4S,
the network cannot properly mark on behalf of the application.
Livingood Expires 18 June 2023 [Page 7]
Internet-Draft ISP L4S Deployment Design December 2022
* To correctly utilize NQB for non-L4S traffic, then the DSCP field
of the packet header is used, with a DSCP 45 marking, according to
Section 4.1 od [I-D.ietf-tsvwg-nqb]. But the majority of traffic
is now encrypted, so it seems implausible for a network to try to
infer the type of traffic and whether an application could benefit
from NQB treatment; this is best left to application developers to
determine as they are the experts in the particular needs of their
application.
* Network operators and equipment vendors attempting to infer
application type and application need will always make mistakes,
incorrectly classifying traffic [Lotus], and potentially
negatively affecting certain flows.
* The pace of innovation and iteration is necessarily faster-moving
in the application edge at layer 7, rather than in the network at
layer 3 (and below) - where there is greater standards stability
and a lower rate of major changes. As a result, the application
layer is best suited to rapid experimentation and iteration.
Network operators and equipment vendors trying to infer
application needs will in comparison always be in a reactive mode,
one step behind changes made in applications.
5.2. All Application Providers Welcome
Any application provider should be able to mark their traffic for the
low latency queue, with no restrictions other than standards
compliance or other reasonable and openly documented technical
guidelines. This maintains the loose cross-layer coupling that is a
key tenet of the Internet's architecture by eliminating or greatly
reducing any need for application providers and networks to
coordinate on deployment (though such coordination is normal in the
early experimental phase of any deployment).
As noted above, this is another example that low latency networking
will have strong network effects, any barriers to adoption such as
this should be avoided in order to maximize the value to users and
the network of a new low latency queue.
5.3. End User CPE Choice
Both customer-owned and leased end user Customer Premise Equipment
(CPE) should be supported. This avoids the risk that an ISP can be
perceived as giving preference to their own network demarcation
devices, which may carry some monthly recurring fee or other cost.
This also means that retail CPE manufacturers need to make the
necessary development investment to correctly implement low latency
networking, though this may not interest or may be outside the
Livingood Expires 18 June 2023 [Page 8]
Internet-Draft ISP L4S Deployment Design December 2022
capabilities of some organizations. In any case, the more devices
that implement then adoption is broader, positively driving network
effects.
5.4. Opt Out Capability
In early phases of deployment of low latency networking, ISPs should
consider making available some mechanism for users to opt out of
(deactivate) it. If low latency networking is working correctly, it
seems extremely unlikely that a user should ever want or need to turn
it off. On the other hand, it is also possible that it may be
desirable in some troubleshooting situations to turn it off, such as
in in cases where a particular application has incorrectly
implemented low latency networking and the developer is working on a
bug fix for an extended period of time. As application use of this
technology matures, it seems likely that there will not be a long
term need or practical benefit to having an opt out mechanism (and it
may be counter productive if it insulates developers from having to
fix bugs or misconfigurations in their software), though an opt out
mechanism may still prove useful.
5.5. Consider Traffic Protection
The specifications in [I-D.ietf-tsvwg-nqb] describe a concept of
Traffic Protection, also known as a Queue Protection Function [ref].
The document says that Traffic Protection is optional and may not be
needed in certain networks. In the case of an ISP deploying low
latency networking with two queues, an ISP should consider deploying
such a network function to at least detect mismarking (if not
necessarily to correct mismarking). This may be implemented for
example in end user CPE, last mile network equipment, and/or
elsewhere in the ISP network - or closely monitors network statistics
and user feedback for any indication of widespread NQB packet
mismarking by applications.
5.6. Avoid Remarking of DSCP Values
If possible, based on a network's existing use of DSCP values, a
network should try to maintain the use of DSCP 45 on an end-to-end
basis without remarking. While this may not be possible in all
networks, it can reduce complexity, enable simpler network
operations, and ease troubleshooting of NQB traffic flows. In some
cases a network may need to migrate an existing, private internal use
of DSCP 45 to some other mark to achieve this. In the long term that
may be best, even if it takes a bit more initial effort when
deploying low latency networking. In addition, if a network does
have their own private internal use of DSCP 45, then they alone
should be responsible for any necessary remarking for traffic passing
Livingood Expires 18 June 2023 [Page 9]
Internet-Draft ISP L4S Deployment Design December 2022
through their network (it would be unfair and unreasonable for a
given network's private use of a DSCP mark to pose a burden on other
networks).
6. Summary of Recommended Deployment Design Decisions
1 Only Applications Mark Traffic: Not the network
2 All Application Providers Welcome: Any application provider can
mark with no restrictions other than standards compliance or other
reasonable and openly documented technical guidelines
3 Device Choice: Both customer-owned and leased cable modem devices
supported
4 User Opt Out: Customers can opt out
5 Consider Traffic Protection: Consider potentially deploying a
network function to detect mismarking of NQB traffic
6 Avoid Remarking of DSCP Values: Try to maintain DSCP 45 on an end-
to-end basis with remarking
7. Acknowledgements
Thanks to Bob Briscoe, Mat Ford, Sebastian Moeller, Sebnem Ozer, Dan
Rice, and Greg White for their review and feedback on this document.
8. IANA Considerations
RFC Editor: Please remove this section before publication.
This memo includes no requests to or actions for IANA.
9. Security Considerations
RFC Editor: Please remove this section before publication.
This memo includes no security considerations.
10. Privacy Considerations
RFC Editor: Please remove this section before publication.
This memo includes no security considerations.
Livingood Expires 18 June 2023 [Page 10]
Internet-Draft ISP L4S Deployment Design December 2022
11. Revision History
RFC Editor: Please remove this section before publication.
v00: First draft
v01: Incorporate comments from 1st version after IETF-115
12. Open Issues
RFC Editor: Please remove this section before publication.
- Open issues are being tracked in a GitHub repository for this
document at https://github.com/jlivingood/IETF-L4S-Deployment/issues
13. Informative References
[I-D.ietf-tsvwg-l4s-arch]
Briscoe, B., De Schepper, K., Bagnulo, M., and G. White,
"Low Latency, Low Loss, Scalable Throughput (L4S) Internet
Service: Architecture", Work in Progress, Internet-Draft,
draft-ietf-tsvwg-l4s-arch-20, 29 August 2022,
<https://www.ietf.org/archive/id/draft-ietf-tsvwg-l4s-
arch-20.txt>.
[I-D.ietf-tsvwg-aqm-dualq-coupled]
De Schepper, K., Briscoe, B., and G. White, "DualQ Coupled
AQMs for Low Latency, Low Loss and Scalable Throughput
(L4S)", Work in Progress, Internet-Draft, draft-ietf-
tsvwg-aqm-dualq-coupled-25, 29 August 2022,
<https://www.ietf.org/archive/id/draft-ietf-tsvwg-aqm-
dualq-coupled-25.txt>.
[I-D.ietf-tsvwg-ecn-l4s-id]
De Schepper, K. and B. Briscoe, "Explicit Congestion
Notification (ECN) Protocol for Very Low Queuing Delay
(L4S)", Work in Progress, Internet-Draft, draft-ietf-
tsvwg-ecn-l4s-id-29, 29 August 2022,
<https://www.ietf.org/archive/id/draft-ietf-tsvwg-ecn-l4s-
id-29.txt>.
[I-D.ietf-tsvwg-l4sops]
White, G., "Operational Guidance for Deployment of L4S in
the Internet", Work in Progress, Internet-Draft, draft-
ietf-tsvwg-l4sops-04, 7 November 2022,
<https://www.ietf.org/archive/id/draft-ietf-tsvwg-l4sops-
04.txt>.
Livingood Expires 18 June 2023 [Page 11]
Internet-Draft ISP L4S Deployment Design December 2022
[I-D.ietf-tsvwg-nqb]
White, G. and T. Fossati, "A Non-Queue-Building Per-Hop
Behavior (NQB PHB) for Differentiated Services", Work in
Progress, Internet-Draft, draft-ietf-tsvwg-nqb-14, 24
October 2022, <https://www.ietf.org/archive/id/draft-ietf-
tsvwg-nqb-14.txt>.
[I-D.ietf-tsvwg-dscp-considerations]
Custura, A., Fairhurst, G., and R. Secchi, "Considerations
for Assigning a new Recommended DiffServ Codepoint
(DSCP)", Work in Progress, Internet-Draft, draft-ietf-
tsvwg-dscp-considerations-08, 13 December 2022,
<https://www.ietf.org/archive/id/draft-ietf-tsvwg-dscp-
considerations-08.txt>.
[BITAG] Broadband Internet Technical Advisory Group, "Latency
Explained", 10 January 2022,
<https://bitag.org/documents/BITAG_latency_explained.pdf>.
[Lotus] Eckerseley, P., "Packet Forgery By ISPs: A Report on the
Comcast Affair", 28 November 2007,
<https://www.eff.org/wp/packet-forgery-isps-report-
comcast-affair>.
[IETF-114-Slides]
White, G., "First L4S Interop Event @ IETF Hackathon", 25
July 2022,
<https://datatracker.ietf.org/meeting/114/materials/
slides-114-tsvwg-update-on-l4s-work-in-ietf-114-hackathon-
00.pdf>.
[LLD] White, G., Sundaresan, K., and B. Briscoe, "Low Latency
DOCSIS: Technology Overview", February 2019,
<https://cablela.bs/low-latency-docsis-technology-
overview-february-2019>.
[Ericsson] Willars, P., Wittenmark, E., Ronkainen, H., Johansson, I.,
Strand, J., Ledl, D., and D. Schnieders, "Enabling time-
critical applications over 5G with rate adaptation", May
2021, <https://www.ericsson.com/49bc82/assets/local/
reports-papers/white-papers/26052021-enabling-time-
critical-applications-over-5g-with-rate-adaptation-
whitepaper.pdf>.
[CTI] International Telecommunications Union - Telecommunication
Standadardization Sector (ITU-T), "Optical line
termination capabilities for supporting cooperative
dynamic bandwidth assignment", Series G: Transmission
Livingood Expires 18 June 2023 [Page 12]
Internet-Draft ISP L4S Deployment Design December 2022
Systems and Media, Digital Systems and Networks Supplement
71, April 2021,
<https://www.itu.int/rec/T-REC-G.Sup71-202104-I>.
Author's Address
Jason Livingood
Comcast
Philadelphia, PA
United States of America
Email: jason_livingood@comcast.com
Livingood Expires 18 June 2023 [Page 13]
