DETNET T. Eckert Internet-Draft Futurewei Technologies USA Intended status: Informational S. Bryant Expires: January 13, 2022 Stewart Bryant Ltd July 12, 2021 Problems with existing DetNet bounded latency queuing mechanisms draft-eckert-detnet-bounded-latency-problems-00 Abstract The purpose of this memo is to explain the challenges and limitations of existing (standardized) bounded latency queuing mechanisms for desirable (large scale) MPLS and/or IP based networks to allow them to support DetNet services. These challenges relate to low-cost, high-speed hardware implementations, desirable network design approaches, system complexity, reliability, scalability, cost of signaling, performance and jitter experience for the DetNet applications. Many of these problems are rooted in the use of per- hop, per-flow (DetNet) forwarding and queuing state, but highly accurate network wide time synchronization can be another challenge for some networks. This memo does not intend to propose a specific queuing solution, but in the same way in which it describes the challenges of mechanisms, it reviews how those problem are addressed by currently proposed new queuing mechanisms. Status of This Memo This Internet-Draft is submitted in full conformance with the provisions of BCP 78 and BCP 79. Internet-Drafts are working documents of the Internet Engineering Task Force (IETF). Note that other groups may also distribute working documents as Internet-Drafts. The list of current Internet- Drafts is at https://datatracker.ietf.org/drafts/current/. Internet-Drafts are draft documents valid for a maximum of six months and may be updated, replaced, or obsoleted by other documents at any time. It is inappropriate to use Internet-Drafts as reference material or to cite them other than as "work in progress." This Internet-Draft will expire on January 13, 2022. Eckert & Bryant Expires January 13, 2022 [Page 1] Internet-Draft bounded-latency-problems July 2021 Copyright Notice Copyright (c) 2021 IETF Trust and the persons identified as the document authors. All rights reserved. This document is subject to BCP 78 and the IETF Trust's Legal Provisions Relating to IETF Documents (https://trustee.ietf.org/license-info) in effect on the date of publication of this document. Please review these documents carefully, as they describe your rights and restrictions with respect to this document. Code Components extracted from this document must include Simplified BSD License text as described in Section 4.e of the Trust Legal Provisions and are provided without warranty as described in the Simplified BSD License. Table of Contents 1. Summary . . . . . . . . . . . . . . . . . . . . . . . . . . . 3 1.1. Problem: High speed forwarding, high scale fan-in/fan-out 3 1.2. Solution goal: Lightweight, per-hop, per-flow stateless transit hop forwarding . . . . . . . . . . . . . . . . . 4 1.3. Requirement: Support for existing stateless / steering solutions . . . . . . . . . . . . . . . . . . . . . . . . 4 1.4. Requirement: PCE to ingress/egress LSR only flow signaling . . . . . . . . . . . . . . . . . . . . . . . . 4 1.5. Requirement: Support for DiffServ QoS model on transit hops. . . . . . . . . . . . . . . . . . . . . . . . . . . 4 1.6. Requirement: Low jitter bounded latency solutions. . . . 4 1.7. Requirement: Dynamic, application signalled DetNet flows 5 2. Evolution of IP/MPLS network technologies and designs . . . . 5 2.1. Guaranteed Service with RSVP . . . . . . . . . . . . . . 5 2.2. Hardware forwarding and DiffServ . . . . . . . . . . . . 6 2.3. MPLS and RSVP-TE . . . . . . . . . . . . . . . . . . . . 6 2.4. Path Computation Engines (PCE) . . . . . . . . . . . . . 7 2.5. Segment Routing (SR) . . . . . . . . . . . . . . . . . . 8 2.6. BIER . . . . . . . . . . . . . . . . . . . . . . . . . . 8 2.7. Summary . . . . . . . . . . . . . . . . . . . . . . . . . 8 3. Additional current considerations . . . . . . . . . . . . . . 9 3.1. Impact of application based state in networks . . . . . . 9 3.2. Experience from IP multicast . . . . . . . . . . . . . . 9 3.3. Service Provider and Private MPLS Networks . . . . . . . 10 3.4. Mission-specific vs. shared infrastructures . . . . . . . 11 3.5. PTP and challenges with clock synchronization . . . . . . 12 3.6. Jitter - in-time versus on-time . . . . . . . . . . . . . 13 4. Challenges for high-speed packet forwarding hardware . . . . 15 5. A reference network design . . . . . . . . . . . . . . . . . 16 6. Standardized Bounded Latency algorithms . . . . . . . . . . . 19 6.1. Guaranteed Service (GS) . . . . . . . . . . . . . . . . . 19 Eckert & Bryant Expires January 13, 2022 [Page 2] Internet-Draft bounded-latency-problems July 2021 6.2. TSN Asynchronous Traffic Shaping (TSN-ATS) . . . . . . . 19 6.3. Cyclic Queuing and Forwarding (CQF) . . . . . . . . . . . 20 7. Candidate solution directions . . . . . . . . . . . . . . . . 22 7.1. Packet tagging based CQF . . . . . . . . . . . . . . . . 22 7.2. Packet tagging based CQF with SR . . . . . . . . . . . . 23 7.3. Per-hop latency indications for Segment Routing . . . . . 23 7.4. Latency Based Forwarding . . . . . . . . . . . . . . . . 24 8. Conclusions . . . . . . . . . . . . . . . . . . . . . . . . . 25 9. Security Considerations . . . . . . . . . . . . . . . . . . . 25 10. IANA Considerations . . . . . . . . . . . . . . . . . . . . . 25 11. Acknowledgements . . . . . . . . . . . . . . . . . . . . . . 25 12. Informative References . . . . . . . . . . . . . . . . . . . 26 Authors' Addresses . . . . . . . . . . . . . . . . . . . . . . . 28 1. Summary The architectural evolution of IP/MPLS networks (Section 2) in service provider and other "larger-than-building" (Section 3.3), shared-infrastructure service networks (Section 3.4) has led to a range of requirements against per-hop forwarding mechanisms which are currently not supported by the current DetNet MPLS forwarding plane [RFC8964] and per-hop, per-flow queueing model[RFC8655], Section 3.2, especially with respect to the QoS support of per-hop bounded latency. The authors of this memo think that solutions for these requirements are relatively easily added to the existing DetNet architecture by adding support for already existing and/or proposed, but not standardized per-hop forwarding and queuing options. The following sub-sections summarize the problem, solution goals and requirements as perceived by the authors. The reasoning for these is explained in the following sections. Note that requirements are somewhat overlapping in so far as solving one of them also solves others, but each addresses the problems from a different perspective, and are therefore easier understood for different stakeholders. For example: Operators that do want to see support of DetNet for example for Segment Routing (SR) would not think that this is "naturally" the same as DetNet supporting the DiffServ architecture, even though solutions would have a hard time to support only one of the two. 1.1. Problem: High speed forwarding, high scale fan-in/fan-out Forwarders with bounded latency need to support interface speeds of 100 Gbps up to Tbps, likely over a period of 10 years from initial deployment of possible DetNet solutions. Hundreds of interfaces may need to be supported in a single forwarder (fan-in/fan-out). Eckert & Bryant Expires January 13, 2022 [Page 3] Internet-Draft bounded-latency-problems July 2021 Supporting bounded latency at these speeds and fan-in/fan-out raises cost and feasibility challenges beyond those that had led to past IETF IntServ (GS) standards ([RFC2210], [RFC2212]) or more recent TSN bounded latency solutions. Note that these high speed and scale requirements even cause challenges when DetNet bounded latency traffic is intended to be used for only a small percentage of the interfaces traffic. 1.2. Solution goal: Lightweight, per-hop, per-flow stateless transit hop forwarding Both high-speed hardware and network architecture design (for reasons of simplicity and minimization of shared risk functions) do favor architectures that support a lightweight transit hop forwarding plane design that requires no forwarding plane or control plane operations whose scale support depends on the number of services/service- instances (e.g.: DetNet flows) offered, but at best only on the size of the network (e.g.: no per-flow, per-hop state). 1.3. Requirement: Support for existing stateless / steering solutions There should be DetNet bounded latency options that work in conjunction with per-transit-hop stateless traffic forwarding such as through Shortest Path First (SPF) routing with IP/MPLS), engineered steering (e.g.: SR) and stateless replication, such as Bit Indexed Explicit Replication with/without Tree Engineering (BIER, BIER-TE). 1.4. Requirement: PCE to ingress/egress LSR only flow signaling There should be DetNet bounded latency options that for the purpose of traffic engineering (including assurance of bounded latency across the network) only require per-flow Path Computation Engine (PCE) signaling to network ingress/egress router, but not to transit hop routers. 1.5. Requirement: Support for DiffServ QoS model on transit hops. There should be DetNet bounded latency options that support the DiffServ QoS model instead of only the IntServ model. 1.6. Requirement: Low jitter bounded latency solutions. There should be DetNet bounded latency options that together with the other requirements also provide a better than worst-case jitter for DetNet traffic. Eckert & Bryant Expires January 13, 2022 [Page 4] Internet-Draft bounded-latency-problems July 2021 1.7. Requirement: Dynamic, application signalled DetNet flows The DetNet architecture should support signaling and forwarding that would make support for automatically application instantiated DetNet flows scalable and lightweight to operate. 2. Evolution of IP/MPLS network technologies and designs To help readers understand especially the per-hop stateless requirement from above, the following sections summarizes the historical evolution of technologies and operational principles that the authors think are relevant to understand the requirements outlined above and asks to see supported in DetNet. 2.1. Guaranteed Service with RSVP The original (first and only) IETF standardized packet forwarding layer standardized queuing option for bounded latency in the IETF is "Guaranteed Service", [RFC2212] (GS), see the DetNet bounded latency document, [DNBL] section 6.5. At the time the RFC was published (1997), the standardized signaling was proposed to be RSVP [RFC2205], and the use of RSVP with GS was standardized in [RFC2210]. The function to support GS bounded latency in the forwarding plane is the per-flow reshaping on every forwarder hop along the path where GS packets of one flow may get delayed in the egress interface queue due to packets from other GS flows. In typical networks, this is every hop along the path. Early (1990/2000) forwarders for which RSVP was implemented where using so-called "software" forwarding. This meant that the forwarding plane was implemented through a general purpose CPU without additional hardware support for QoS functions such as shaping or queuing. While these forwarders did support traffic flow shaping, GS was never implemented on them and their RSVP implementations did also not support (but ignored) the RSVP TSPEC/RSPEC signaling parameters used for bounded latency. Instead, RSVP implementations only supported the parameters for bandwidth reservation, which was henceforth called Call Admission Control (CAC). In one instance, a software forwarder implementation with RSVP supported the Controlled Load (CL) service [RFC2211], which does not provide for bounded but instead for controlled latency. This service is achieved by creating a per-flow queue and applying weighted fair queuing (WFQ) with weights according to the reserved bandwidth of the flows (see [RFC2211], section 11). This functionality did not proliferate into later generations of routers because the execution cost of WFQ was too high for a multitude of flows and the scheduling Eckert & Bryant Expires January 13, 2022 [Page 5] Internet-Draft bounded-latency-problems July 2021 accuracy was too inaccurate in interrupt driven CPU software forwarding with higher speed interfaces (100Mbps...1Gbps). 2.2. Hardware forwarding and DiffServ With the rise of forwarding planes with "acceleration" through ASIC based Forwarding Plane Elements (FPE) instead of general purpose CPUs and/or dedicated QoS hardware, the ability of forwarders to support shaping evolved to only be supported, if at all, on DiffServ (DS) boundary nodes, but not on DS interior nodes. This included both shaping as well as complex queuing such as WFQ. The DS architecture, [RFC2475], was specifically targeted to enable the evolving, now common Service Provider network services architecture, in which "high-touch" service functions are only performed on so-called Provider Edge (PE) routers, which as required are DS boundary nodes, whereas the hop-by-hop forwarding through so- called Provider (P) (core) routers is meant to utilize only a reduced set of forwarding functions, specifically excluding per-hop, per-flow QoS forwarding plane functions such as shaping or policing. DiffServ therefore allowed to build higher speed, lower cost forwarding plane P routers. It also enabled to build equally higher speed, lower costs PE routers by supporting boundary node functions only on (lower speed) customer facing interfaces/line cards, but not on core facing interfaces. 2.3. MPLS and RSVP-TE With the advent of MPLS [RFC3031], RSVP was extended to support MPLS through the RSVP-TE [RFC3209] extensions. RSVP-TE manages p2p (later on also p2mp) MPLS Label Switched Paths (LSP), which when signaled through RSVP-TE are also called RSVP-TE tunnels. These can be seen as the equivalent of IP flows that RSVP manages for IP. RSVP-TE tunnels can support a variety of traffic engineering functions, but none of the implementations known to the authors ever implemented GS or CL services, specifically because hardware forwarding for service provider networks was not designed to support these QoS functions for P Label Switched Routers (LSR). Because CL/GS where not targeted with RSVP-TE, the signaling extensions for Interior Gateway Protocols (IGP) required in the classical RSVP-TE reservation model (such as [RFC8570] for IS-IS) have no parameters to signal per-hop GS queuing latency or buffer capacity utilization. In result, the existing IGP signaling for RSVP-TE only supports RSVP-TE to perform bandwidth but not non- queuing path latency resource calculations and therefore no latency based traffic engineering. Eckert & Bryant Expires January 13, 2022 [Page 6] Internet-Draft bounded-latency-problems July 2021 2.4. Path Computation Engines (PCE) Even though RSVP-TE implementations support only DiffServ (but not GS/CL) with respect to per-hop QoS functions, its traffic-steering (path selection) and signaling model introduced per-flow (per-tunnel) control plane and forwarding plane overhead onto every P-hop. Through the 200x's, this RSVP-TE overhead was seen as undesirable complexity and overhead by many service providers using it. There was also a much larger number of service providers that desired some of the benefits provided by RSVP-TE, but who were not willing to commit to the complexity, costs and operational risk introduced into the network by complex per-flow signaling of RSVP-TE. The on-path, per-hop signaling of RSVP-TE for example introduced so much overhead, that reconvergence of RSVP-TE paths after a failure or recovery took as much as 20 minutes in networks with 10,000 or more RSVP-TE tunnels. The design of RSVP-TE's (decentralized) on path signaling model specifically showed problematic under high resource utilization. In the original, decentralized RSVP-TE deployment model, ingress PE LSR would perform so-called Constrained Shored Path Forwarding (CSPF) calculations to determine the shortest path with enough free resources for a new flow. Afterwards the ingress PE would signal the path via RSVP-TE. The IGP would signal to all ingress PE how many (bandwidth) resources where left on every link. Under high load, when multiple ingress PE where performing this process in parallel this would cause high load, churn and reservation collisions. These problems of de-centralized RSVP-TE plus IGP signaling lead to the introduction of a so-called Path Computation Element (PCE) based architecture, in which the (competing and uncoordinated) traffic engineering computations on every de-centralized RSVP-TE ingress LSR where replaced by a centralized PCE function (or at least a coordinated PE function), which would send the calculated results back as a path object to the headend LSR, in result limiting the functions of RSVP-TE to the signaling of a steered traffic path through the network to establish the hop-by-hop LSP. The use of a PCE can likewise eliminate all the reservation state dependent signaling from the RSVP-TE IGP extensions, because all the reservation calculations solely need to happen only on the PCE. Nevertheless, the PCE does not eliminate the per-hop signaling overhead of RSVP-TE to establish LSPs and hence it did not eliminate for example the majority of the platform and convergence cost of RSVP-TE in the network, especially for the control plane of P nodes adn could hence not resolve the concerns of service providers who had chosen not to adopt RSVP-TE. Eckert & Bryant Expires January 13, 2022 [Page 7] Internet-Draft bounded-latency-problems July 2021 2.5. Segment Routing (SR) The introduction of centralized PCE had obsoleted most of the reasons for RSVP: headends did not need to do path calculation, and P router did not need to manage the available and allocated bandwdth for TE tunnels. In most service-provider use-cases this left RSVP-TE only serving as a very complex solution to do traffic steering, and the PCE was doing the rest. This ultimately lead to the design of the Segment Routing [RFC8402] architecture, and its mapping to the MPLS forwarding plane, SR-MPLS [RFC8660]. Later, a mapping to IPv6 was defined with SRv6 [RFC8986]. SR relies on strict or loose hop-by-hop hop source routing information, contained in each packet header, therefore eliminating the need to set up per-path flow state via RSVP-TE, and allowed in conjunction with DiffServ for hop-by-hop QoS a complete per-hop, per-flow stateless forwarding solution, arguably therefore lightweight, easy to implement at high performance and scalable to large number of flows. 2.6. BIER In the same way as SR eliminated the need for hop-by-hop traffic steering forwarding state from RSVP-TE in P-routers for unicast traffic, Bit Indexed Explicit Replication [RFC8279] (BIER) solves this problem for shortest path multicast replication state across P-routers, by replacing it with a BIER packet header [RFC8296] and therefore eliminating any per-application/flow, per-hop forwarding state for multicast in P-routers. BIER also removed the associated overhead of prior ingress replication solutions Service Providers where looking into to avoid the per-hop state. Finally, BIER-TE [I-D.ietf-bier-te-arch] adds traffic steering with replication to the BIER architecture and calls this Tree Engineering. Likewise, this is without the need for per-hop/per-flow steering or replication state. 2.7. Summary Service Provider networks have evolved especially in the past 25 years into an architecture, where high-speed, low-cost and high- reliability are based on designs that eliminate or reduce as much as possible any form of unnecessary control-plane and even more so per- flow, per-application plane complexity from P-routers/transit-nodes. This has led to the development of the DiffServ QoS architecture that eliminated IntServ/per-flow QoS from P-routers, and later on to the evolution from MPLS/RSVP-TE to SR and BIER that eliminated per-flow/ tunnel forwarding/steering and replication state from the same P-nodes. Eckert & Bryant Expires January 13, 2022 [Page 8] Internet-Draft bounded-latency-problems July 2021 Finally, early experience with Traffic Engineering churn under high load and todays requirements for often NP-complete optimization lead to an architectural preference for off-path/centralized model for TE calculations via PCE to also free P-routers from signaling complexity and perform dynamic/service-dependent signaling only to PE-routers. 3. Additional current considerations The following subsections look at further into the background for why per-hop, per-flow state can be problematic and discuss problems beyond this core issue. 3.1. Impact of application based state in networks RSVP-TE was (and is) solely used for services where the operator of a domain explicitly provisions RSVP-TE tunnels across its domain (for example using a PCE) and can therefore fairly easily know the worst- case scaling impact. For example the number of tunnels does is not a chance value arising through dynamic subscriber action, and the number of tunnels in the network is primarily impacted by topological changes and the (over time relatively rare) of occurrences of additional services and/or service instances being provisioned. For RSVP-TE there was never (to the knowledge of the authors) an end-to- end application layer interface such as there was for RSVP over IP, for example as supported by earlier versions of Microsoft Windows QoS enabled IP sockets. When per-flow operations including per-hop signaling or even worse per-hop forwarding plane or QoS state is not a result of well- controlled provisioning or well plannable/predictable failure behavior but instead driven by applications not under the control of network operators, the per-hop state requirements can become much more an operational and cost problem, because of its unpredictability. 3.2. Experience from IP multicast The widest experience with dynamic, application based signaling in Service Provider networks likely exist for IP multicast, where creation of per-hop forwarding/replication state is triggered by applications not under the control of network operations but by customer managed applications/application-instances. Managing the amount of state and the control plane load on P-routers was and is one of the mayor concerns when operationalizing IP Multicast services in SPs. Service Provider L2-VPN and L3-VPN services can offer IP Multicast via architectures such as [RFC6513] that attempt to solve/reduce the Eckert & Bryant Expires January 13, 2022 [Page 9] Internet-Draft bounded-latency-problems July 2021 problem of customer application driven, per-multicast application in a variety of ways, but they all come with their own problems: o In ingress-replication, the ingress-PE sends a separate unicast copy to every egress-PE. This creates significant excess traffic on links close to the ingress-PE and potentially higher-cost ingress-PE attachment speeds. o In L3VPN aggregates-trees, the traffic for multiple trees is sent across a common tree reaching the superset of all egress-PE of all included trees. This reduces the number of trees from one per- customer application to a lower number of aggregates this, but it creates potentially significant excess traffic towards egress-PE that do not need all the aggregated traffic and may even result in a requirement for access core access link speeds for those egress routers. Finally, the per P-router stateless BIER solution solved these issues. It does not require any per P-router, per tree state creation, and achieves a 256x better traffic efficiency than ingress replication (with 256 long BIER bit strings). 3.3. Service Provider and Private MPLS Networks With DetNet services being targeted primarily for so-called private networks such as (but not limited to) those for industrial, theme parks, power supply systems, road, river, airport and train transportation networks, it is important to understand how concerns for SP networks will apply to such private networks: While the aforementioned evolution of MPLS networks focused on large- scale service provider networks, the very same architectural evolution is or will also happen in any private MPLS networks in the same way as the DiffServ architecture equally became the only widely adopted QoS architecture in any larger scale (campus or beyond) private networks. While some of the scaling, cost, performance and reliability issues mentioned above for service providers may not equally apply to smaller scale private networks, past experience has shown that that it is unlikely for a critical mass for different solutions to develop across a large variety of vertical private type of networks. For this reason, in the past any larger scale enterprise networks have preferred to adopt solutions that had proven themselves through SP deployments and that where based on cross-vendor IETF based architecture principles and widely, interoperable vendor implementations. Eckert & Bryant Expires January 13, 2022 [Page 10] Internet-Draft bounded-latency-problems July 2021 Another reason for private network operators looking for service provider calls designs is that it also is simplifies potential service provider based management of the network and/or outsourcing of the network to a service provider. This was seen often when large enterprises that had to support multi-tenants evolved from ad-hoc network virtualization solutions (such as VRF-lite) over to BGP/MPLS- VPN designs and later outsourced those very networks. In that same line of future proofing, networking technologies first developed for enterprises would also be picked up and reused in Service Provider networks as long as they would fit. IP Multicast for example had (since about 1996) ca. 10 years of deployment for business critical enterprise use cases (such as financial market data distribution), before it was adopted widely for IPTV in service providers. 3.4. Mission-specific vs. shared infrastructures Whereas the previous section points to the practice and benefits to share technologies between private and SP network, this section highlights one core additional requirement of SP networks not found in most private networks from which pre-DetNet deterministic service requirements will likely originate. In architectural terms, the desire and need to minimize or avoid per- application/flow forwarding/control-plane state and per-hop control plane interactions (be it through on-path signaling or direct PCE to P-router signaling) is not primarily a matter of SP/private networks or not even of size, but foremost a matter of whether or not the network itself is seen as the (a) communications fabric of a large distributed application or (b) as an independently running shared infrastructure across a potentially wide variety of application/ services with diverging requirements. (a) is the dominant view of the network specifically from many (single) mission specific networks such as many industrial networks and even non-public High Performance Compute (HPC) center architectures. In either of these case, it is a single architectural entity that can control both network infrastructure and application to build a mission optimized compound. For example, switches in HPC Data Centers had traditionally very shallow interface packet buffering for cost reasons, resulting in inferior performance under peak load with predominant older TCP congestion control stacks. Instead of using better, more expensive switches, it was easier to improve application device TCP stacks, leading for example to BBR TCP. While this is very much in line with the desired Internet architecure that is putting a significant Eckert & Bryant Expires January 13, 2022 [Page 11] Internet-Draft bounded-latency-problems July 2021 responsibility onto transport layer protocols in hosts (not limited to TCP) to behave "fair" or "ideal", the reality even in many private missions centric networks such as manufacturing plant is different. Dealing with misbehaving user devics or applications is one of the main challenge. In the example, that is the case when a DC is offering public cloud services, where TCP stacks can not be controlled, and hence deeper buffers and/or better AQM are a core requirement. In general: In networks following the (b) shared infrastructure design principle, any resource that needs to be shared across different services or even service instances becomes a potential three party reliability and costing issue between the provider running the network and the two (or more) parties whose services utilize the common resource. Minimizing the total amount of complex, failure-prone and hard to quantify in a cost-effective manner shared resources is thus at the base of any shared infrastructure network design. This again points to the model, where all network control can happen on the edge, and due to the absence of per-hop, per-flow state there simply is no shared flow state table that needs to be managed across multiple different services/subscribers. 3.5. PTP and challenges with clock synchronization Some bounded latency solution require accurate clock synchronization across network nodes performing the bounded latency algorithm. The most commonly used (family of) protocol(s) for this is the Precision Time Protocol (PTP), standardized in IEEE1588 and various market specific profiles thereof. PTP can achieve long-term Maximum Time Interval Errors (MTIE) of as little as 10th of nsec. MTIE is the maximum time difference between the clocks of two PTP nodes measured over long period of time. Implementing PTP in devices comes at a range of design requirements. At high degree of accuracy, PTP requires accordingly accurate local oscillators that includes hardware such as regulated heating to operate under constant temperature. It includes accurate distribution of clock across all components of the system, which can be especially challenging in modular, large-scale devices, and accurate insertion and retrieval of timestamp field into packet headers. While PTP is becoming more and more widely available, consistent support of high accuracy across all target type of switches and routers in wide area networks cannot be taken for granted to be a Eckert & Bryant Expires January 13, 2022 [Page 12] Internet-Draft bounded-latency-problems July 2021 feasible new requirement raised for DetNet when it did not exist in before. Today, PTP is often found in mobile network fronthauls, but not their backhauls or any other broadband aggregation, distribution or core networks. This is because there is, as of today, no strong business case requirement for PTP at high precision in those networks, whereas technologies such as eCPRI raise such requirements against mobile fronthauls. Instead, those other networks most often resort to at best msec accuracy NTP protocol deployments which is typically sufficient for control-plane and operational event tracing as its main, accuracy defining use-case. The larger the network and more multi-vendor varied the deployed equipment is, the higher will also be the operational cost of maintaining and controlling the accuracy of a PTP service. This primarily has been cited in the past as a reason to not deploy PTP even if hardware was supporting it. This operational challenge will especially apply when PTP support may be required for only a small percentage of traffic in a high speed wide area network. The revenue from the service needs to cover the operational cost incurred by its exclusive components (hardware, software and operations). 3.6. Jitter - in-time versus on-time This section discusses how low-jitter bounded latency applications can be highly beneficial for DetNet applications. Depending on the bounded latency algorithm, the jitter experienced by packets varies based on the amount of competing traffic. In algorithms and their resulting end-to-end service which this memo calls "in-time", such as GS and [TSN-ATS], the experienced latency in the absence of any competing traffic is zero, and in the presence of the maximum amount of permissible competing traffic, latency is the maximum, guaranteed bounded latency. In result, the jitter provided by these algorithms is the highest possible. In algorithms and their resulting end-to-end service which this memo calls "on-time", the experienced latency is completely or most significantly independent of the amount of competing traffic and the jitter therefore null or minimal. In these algorithms, the network buffers packets when they are earlier than guaranteed, whereas in- time algorithms deliver packets (almost) as fast as possible. This memo argues that on-time queuing algorithms provide an additional value-add over in-time algorithms, especially for use in metropolitan or wide-area networks. Whatever algorithm is used, the receiving application only has a guarantee for the maximum bounded latency, and the real (shorter) latency of any received packet is no indication for the latency of the next packet. Instead, the receiver Eckert & Bryant Expires January 13, 2022 [Page 13] Internet-Draft bounded-latency-problems July 2021 application has to be prepared for each and any future packet to arrive with the worst possible, e.g.: the bounded latency. The majority of applications require some higher layer function synchronously to the sender application: Rendering of audio/video and other media information needs to happen at the same frequency or event intervals at which the media was encoded. When these applications receive packets earlier than the time at which they can be processed (which is equal or close to the bounded latency), these applications buffer media in a so-called playout buffer and release them only at that target time. Likewise, remote control loops including industrial Programmable Logic Controller (PLC) loops or remote controlling of robots or cars is typically based on synchronous operations. In these applications, early packets are also delayed to then be processed "synchronously" later. In all cases, where applications need to buffer (or otherwise remember) received data when it is too early, in-time queueing latency raises the challenge to application developers to be able to predict the networks worst possible jitter, and this can be particularly challenging for embedded, if not constrained receiver devices with minimum memory to buffer/remember. When these devices are designed against one particular type of network with well-known low jitter, then they will not necessarily operate correctly in networks with larger jitter. And in metropolitan and WAN networks, jitter with in-time services can be highly variable based on its design and the relative location of the communicating nodes in the topology (see Section 5 for an example network design). One example of such issues was encountered when digital TV receivers (Set Top Boxes, STB) designed for (mostly synchronous) digital cable transmission where evolved to become IPTV STB, but the playout buffer of < 50 msec was not sufficient to compensate for a > 50 msec jitter experienced in IP metropolitan networks. Note that this section does not claim that all applications will benefit from on-time service, nor that no application would benefit more from in-time service than from on-time service. Nevertheless, the authors are not aware of instances of [RFC8578] application for whom in-time service would be more beneficial than on-time service. Of course, this comparison is only about the benefit to the application and other factors such as the cost/scale of the service for the network itself have also to be taken into account. Eckert & Bryant Expires January 13, 2022 [Page 14] Internet-Draft bounded-latency-problems July 2021 4. Challenges for high-speed packet forwarding hardware The problems of cost and operational feasibility in shared- infrastructure networks specifically applies to scaling of hardware resources such as per-application-flow forwarding or QoS state in high-speed network routers: Even if the business case makes it clear that only e.g. 1 Gbps worth of traffic may require this advanced state (such as multicast replication or per-flow shaping for bounded latency), it will be more expensive to build this functionality into a 100 Gbps transit switch/router than into a 1 Gbps switch/router. This too is based on experience from migrating services of low-speed mission specific networks, such as IP multicast onto high speed, shared-infrastructure service provider networks. The reason for this higher cost at higher speed is that the 1 Gbps worth of "advanced" traffic still has to be built into 100 times faster hardware and each of the "advanced" packets forwarded would needs to be replicated/shaped 100 times faster. This packet processing issue may look like it applies equally to both per-hop, per-flow stateful based forwarding as well as solely in- packet based mechanisms, in practice, per-flow state may requires a lot more high-speed memory access because of the need to access an entry from a state table. In most cases, this table space can only be made to work at line rate packet processing when it is on-chip, hence it is not only most expensive, it is also crucial to scale right. And as the 1 vs. 100 Gbps example above showed, it is very hard to come by an appropriate scale smaller than "would work for 100% of traffic" - because network operator providing shared infrastructure networks really do not want to be responsible for predicting how individual services may grow in adoption by making a specific hardware selection that constrains any such grows. Last, but not least, on-chip high-speed state tables become even more expensive when they do not only have to be read only, but also when they have to be written at line rate and even worse, when they have to operate for line-rate speed read/write/read control loops: The main issue with scaling state in hardware routers is that designs will be hesitant to work against unclear growth predictions. Even if at some point in time only 1 Gbps of DetNet traffic was expected to be required on a 100 Gbps platform, hardware designers will be much more likely want to scale against the worst (best) case service growth expectation so that customers will not feel that they would buy into a product that becomes obsolete under success. Whereas steering state, such as MPLS label entries can easily scale to hundreds of thousands, the same is not clear about shapers or Eckert & Bryant Expires January 13, 2022 [Page 15] Internet-Draft bounded-latency-problems July 2021 interleaved regulators. They are more challenging because they require fast (on-chip) read-write memory for the state variables, especially when forwarding is parallelized across multiple execution unit. This does incur additional complexity to split up the state and its packets across multiple execution units and/or to provide consistent cross-execution units shared read/writeable memory. Even only writeable (but not cross-execution units then also readable) memory has traditionally been a sparse resource the faster the forwarding engines are. This can be seen from (often very limited) scale of packet monitoring state such as for IPfix. But the main issue of per-hop, per-flow forwarding state that could be quite dynamic because it might be triggered by applications is the control plane to forwarding-plane-state interactions. Updating hardware forwarding engine state tables is often one of the key performance limits of routers. Adding significant additional state with likely ongoing changes is easily seen as a big contributor to churn in the control plane and likely reason for stability and reduced peak performance under key events such as reconvergence of all or large parts of IGP or BGP routing tables. 5. A reference network design The following picture shows an example, worst-case network topology of interest (in the opinion of the authors) for bounded latency considerations. This section does not claim that greenfield rollouts may or want to use all aspects of this topology. What his memo does claim is that many existing brownfield networks, especially large metropolitan areas show all or many of these aspects, and that it would be prudent for bounded latency network technologies to support networks like these so as to not create new constraints against network designers by only supporting physical network topologies optimized for a particular type of service (bounded latency). Eckert & Bryant Expires January 13, 2022 [Page 16] Internet-Draft bounded-latency-problems July 2021 Subscribers, Towers, IoT devices ............. ...Access.... National-Core PE100 ... PE199 Exchanges/ | | Peerings | \----\ / \ | \ / \ --- P11 --- P12 --- P13 --- P14 -- / \ Edge -----P21 P15 DC PE | | ------P21 P17 \ / --- P20 --- ......... --- P18 -- / \ Edge --- P30 P40 DC PE \ / ----- P31 -- .... P38 --- P39 \ / \ / PE200...PE299 ...Access.... ............. Subscribers, Towers, IoT devices Figure 1: Reference Network Topology An example metropolitan scale network as shown in Figure 1 may consist of one or more rings of forwarders. A ring provides the minimum cost n+1 redundancy between the ring nodes, especially when, as is common in metropolitan networks, new fibre cannot cost- effectively be put into new optimum trenches, but existing fibre and/ or trenches have to be used. This is specifically true when the area includes not dense populated suburban areas (higher cost per subscriber and mile for rollouts). Multiple, so-called subtended rings typically occur when existing networks are expanded into new areas: A new ring is simply connected at two most economic points into the existing infrastructure. Likewise, such a topology may become more complicated over time by addition of capacity, which resulting from TE planning calculations may not follow any of the pre-existing ring paths. Edge Data-Center (DC), connections to Exchanges/Peerings or national cores of the provider itself, as well as all subscribers including Mobile Network Towers, and IoT devices connect to these ring directly via PE edge-forwarders and (more often) via additional CE type devices. P nodes may also double as PE nodes. Eckert & Bryant Expires January 13, 2022 [Page 17] Internet-Draft bounded-latency-problems July 2021 In densely populated regions, P, or PE nodes may have a high number of attached devices, shown in the picture with the example of 100 PE forwarder connecting to a single P forwarder (or rather two P for redundancy and therefore support of PREOF). In summary, the following aspects of these networks are relevant for bounded latency: o Link speeds today are at least 100 Gbps and will be Tbps in the near future. Even if only a small percentage of that traffic has to support bounded latency, the queuing mechanism need to support these high-speed interfaces. o Fan-in/out at PE or P nodes may be (worst case) in the order of hundred(s) of incoming interfaces. Bounded latency mechanisms whose number of queues depend on the number (#I) of interfaces in a more than linear fashion, such as (#I^2) in the case of [TSN-ATS], may introduce significant challenges for cost-effective hardware. o Through the advent of decentralized edge Data Center and peerings between different operators and content providers, traffic flows of interest will not solely be between one central site from/to subscribers hub&spoke. Instead arbitrary, traffic engineered paths across the topology between any two edges need to be supportable in scale with the bounded latency queuing mechanism. o The total number of edge (#E) nodes (PE or CE) for a bounded latency service can easily be in the thousands. Aggregation of bounded latency flows on the order of (#E^2), which is the best option in per-hop, per-flow solutions such as [TSN-ATS], is likely insufficient to significantly reduce the number of flows that need to be managed across P nodes in such bounded latency queuing mechanisms. o The total number of P nodes may be in the hundreds and bounded latency flows in the tenths of thousands. It should also be expected that such flows are not necessarily long-term static but may need to be provisionable in the time-scale order of for example telephone calls (such as flows supporting remote control of devices or operations). Bounded latency solutions that require per-flow, per-node state maintenance on the P nodes themselves may therefore be undesirable from a network operational/complexity/reliability perspective, but also from a hardware engineering cost perspective, especially with respect to the control plane cost of dynamically setting up per-flow bounded latency for flow whenever there is a new flow or all of them Eckert & Bryant Expires January 13, 2022 [Page 18] Internet-Draft bounded-latency-problems July 2021 whenever there are topology or load changes that make rerouting desirable. Beyond queuing concerns, path selection too specifically for deterministic services is a challenge in these networks: o Path lengths may be significantly longer than e.g. 3 hops. In large metropolitan networks, they can reach 20 or more hops. Speed of light end-to-end in these networks will be in the order of low number of msec. End-to-end queuing latency can be in the same range, if not higher. o To avoid undesirable re-routing under failure when PREOF and engineered disjoint paths are used, traffic steering needs to support efficiently supportable hop-by-hop traffic steering. In networks designed for source-routing (e..: SR routing), efficiently encoded strict-hop-by-hop steering for as much as those (e.g.: 20) hops may be desirable to support. 6. Standardized Bounded Latency algorithms [DNBL] gives an overview of the math for the most well-known existing deterministic bounded latency algorithms/solutions. This section reviews the relevant currently standardized algorithms from the perspective of the above listed problems for high-speed, high-scale, shared services infrastructures and to provide additional background about them. 6.1. Guaranteed Service (GS) GS is described in section 6.5 of [DNBL]. Section 2.1 describes its historical evolution and challenges. We skip further detailing of its issues here to concentrate on IEEE Time Synchronuous Networking - Asynchronous Traffic Shaping [TSN-ATS], which in general is seen as superior to GS for high speed hardware implementation. All the concerns described in the TSN-ATS section apply equally or even more to GS. 6.2. TSN Asynchronous Traffic Shaping (TSN-ATS) Section 6.4 of [DNBL] describes the bounded latency used for TSN Asynchronous Traffic Shaping [TSN-ATS]. Like GS, this bounded latency solution also relies on per-flow shaper state, except that it uses optimized shapers called "Interleaved Regulator" as explained in section 4.2.1 of [DNBL]. The concept and simplification in interleaved regulators over traditional shapers and the concept of interleaved regulators is a Eckert & Bryant Expires January 13, 2022 [Page 19] Internet-Draft bounded-latency-problems July 2021 resulting from mathematical work done in the last 10 years starting with [UBS]. In a system with e.g. N=10,000 flows each with a shaper, the forwarder needs to have 10,000 shapers each of which would need to calculate the earliest feasible send-time of the first queued packet of the flow and all these send-times would need to be compared by a scheduler picking the absolute first packet to send. Of course it is unlikely that the router would have to queue at least one packet for all queues at any point in time, but the complexity to implement the scheduler scales with N. With interleaved regulators, there is still the per-flow state required to hold each flows traffic parameters and its next-packet earliest departure time, but instead of requiring a scheduler to compare N entries, packets are queued into one out of (#IIF,#PRIO) FIFO queues, one queue for all the packets arriving from the same Incoming InterFace (IIF) and targeted the same worst-case queuing latency/PRIOrity (PRIO) on this hop. The shaper now only needs to calculate the earliest departure time of the head of each of these M= #IIF * #PRIO queues and the complexity of a scheduler to select the first packet across those interleave regulators is therefore reduced by a factor of O(N/M). Unfortunately, while industrial ethernet switches today often have no more than 24 IIF, aggregation routers in metropolitan networks may have thousands of IIF, so the benefit of interleaved regulators over per-flow shaper will likely be much higher in classical TSN environments than it would be for example likely DetNet target routers in metropolitan networks. In addition, the aforementioned core problems for shapers (Section 4), namely control plane, read/write/read cycle access and scale equally apply to interleaved regulators, so the main optimization benefits of interleaved regulators is for the original targets of [UBS] / [TSN-ATS]: low-speed (1..10Gbps switches) with limited number of interfaces - but to a much lower degree for likely important type of DetNet deployments. 6.3. Cyclic Queuing and Forwarding (CQF) TSN Cyclic Queuing and Forwarding as described in [DNBL], section 6.6, is a per-flow, per-transit-hop stateless forwarding mechanism, which solves the concerns with per-hop, per-flow state issues described earlier in this memo. It also provides an on-time service in which the per-hop and end-to-end jitter is very small, namely in the order of a cycle time. Eckert & Bryant Expires January 13, 2022 [Page 20] Internet-Draft bounded-latency-problems July 2021 [CQF] operates by forwarders sending packets in periodic cycles. These cycles are derived from clock synchronization: The start of each cycle (and by implication the end of the prior cycle) are simply periodically increasing clock timestamps that have to be synchronized across adjacent forwarders, usually via PTP. This method to operate cycles allows [CQF] to operate without additional [CQF] data packet headers, but it is also the reason for the two issues of [CQF], and both relate to the so-called dead time (DT). For the receiving node to correctly associate a [CQF] packet to the same cycle as the sending node, the last bit of the last packet in the cycle on the sending node needs to be received by the receiving node before the cycle ends. [DNBL] explains that DT is the sum of latencies 1,2,3,4 as of [DNBL] Figure 1, but that is missing the MTIE between the forwarders: If a cycle is for example 10 usec, and the PTP MTIE is 1 usec, then only 9 usec of the cycle could be used (without even yet considering the other factors contributing to MTIE). If MTIE is not taken into account, a packet might arrive in time from the perspective of the sending forwarder, but not in the perspective of the 1 usec earlier receiving node. In practice, MTIE should be equal or lower than 1% of the cycle time. When forwarders and links increase in speed, cycle times could become proportionally smaller to reduce per-hop cycle time latency. When this is done, MTIE needs to equally become smaller, raising the costs of the solution. Therefore, [CQF] has a challenge with higher speed networks. The second and even more important problem is that DT includes the link latency (2 in [DNBL], Figure 1). With a speed of light in fibre of 200,000 Km, link latency is 10 usec for 2 Km. This makes [CQF] very problematic and limited in metropolitan and wide-area networks. If the longest link of a network was 10 Km, this would cause a DT on that link of 50 usec and with a cycle time of 100 usec, only 50% bandwidth could be used for cycle-time (bounded latency) traffic (excluding all other DT factors). When links are subject to thermal expansion also known as sag on hanging wires, such as broadband copper wires (Cable Networks), their length can also change by as much as 20% between noon and night temperatures, which without changes in the design has to be taken into account as part of DT. In conclusion, [CQF] solves many of the problems discussed in this memo, but it's reliance on timestamp synchronized cycles may pose undesirable challenges with the required accuracy of PTP in high Eckert & Bryant Expires January 13, 2022 [Page 21] Internet-Draft bounded-latency-problems July 2021 speed network and especially limits [CQF] ability to support wider- scale networks due to DT. 7. Candidate solution directions As this memo outlines, per-hop, per-flow stateless forwarding is the one core requirement for to support Gbps speed metropolitan or wide- area networks. This section gives an overview and evaluation from the perspective of the authors of this memo of currently known non-standardized proposals for per-hop-stateless forwarding with the explicit goal and/or possibility of bounded latency forwarding and in relationship. to the concerns and desires described in the previous sections. 7.1. Packet tagging based CQF To overcome the challenges outlined in Section 6.3, [I-D.qiang-DetNet-large-scale-DetNet] and [I-D.dang-queuing-with-multiple-cyclic-buffers] (tagged-CQF) propose a modified [CQF] mechanism in which timestamp based cycle indication of [CQF] is replaced by indicating the senders cycle in an appropriate packet header field, so that the receiver can accordingly map the received packet to the right local cycle. This approach completely eliminates the link-latency as a factor impacting the effectiveness of the mechanism, because in this approach, the link latency does not impact the DT. Instead the link latency is used to calculate which cycle from the sender needs to be mapped to which cycle on the receiver, and this is programmd during setup of links into the receiving routers cycle mapping table. Depending on the number of cycles configured, it is also possible to compensate for variability in the link-latency and higher MTIE (picture TBD). If one more cycle is used for example, this would allow for MTIE to be the order of one cycle time as opposed to a likely target of 1% of cycle time as in [CQF], reducing the required PTP clock accuracy by a factor of 100. This possible reduction in required accuracy of operations by appropriate configuration does not only cover PTP but also extends into any forwarding operation within the nodes, e.g.: it could also reduce the cost of implementation of forwarding hardware at higher speeds accordingly. In MPLS networks, packet tagged CQF with a small number of cycle tags (such as 3 or 4) could easily be realized and standardized by relying on E-LSP where 3 or 4 EXP code points would be used to indicate the cycle value. Given how such deterministic bounded latency traffic is not subject to congestion control, it also does not require Eckert & Bryant Expires January 13, 2022 [Page 22] Internet-Draft bounded-latency-problems July 2021 additional ECN EXP code points, so those would be available for e.g.: best-effort traffic that should use the same E-LSP. 7.2. Packet tagging based CQF with SR [I-D.chen-DetNet-sr-based-bounded-latency] applies the taged-CQF mechanisms to Segment Routing (SR) by proposing SR style header elements to indicate the per-segment/hop cycle. This eliminates the need to set up on every hop a cycle mapping table. It is unclear to the authors of this memo how big a saving this is given how the PCE would need to update all the ingress router per- flow configurations where header imposition happens when links change, whereas the mapping table approach would require only localized changes on the affected routers. 7.3. Per-hop latency indications for Segment Routing [I-D.stein-srtsn] describes a mechanism in which a source-routed header in the spirit of a Segment Routing (SR) header can be used to enable a per-transit-hop per-flow stateless latency control. For every hop, a maximum latency is specified. The draft outlines a control plane which similarly to packet tagging based CQF or [TSN-ATS] would put the work of admitting flows, determining their paths and admitting their resources along those paths to some form of PCE/SDN-Controller. The basic principle of forwarding in this proposal is to put received packets into a priority heap and schedule them in order of their urgency (shortest latency) for this hop. The draft explicitly does not prescribes specific algorithms on the forwarders to take the indicated latency for the hop into account in a way that the controller can calculate the resource availability, such as specific queuing or scheduling algorithms. It is not entirely clear to the authors of this memo, if the sole indication of such deadline latencies is sufficient to completely eliminate per-transit-hop, per-flow state and still achieve deterministic latency because of the [UBS] work. Consider that the packets latency for a hop could be used to derive a priority queue on the hop relative to other packets with higher or lower latency for this hop, As was shown in the research work leading up to [TSN-ATS], the priority queuing on each hop alone is not sufficient to achieve a simple, solely per-hop calculated latency bound under high load because of the problem of multi-hop burst aggregation and the Eckert & Bryant Expires January 13, 2022 [Page 23] Internet-Draft bounded-latency-problems July 2021 resulting hard to calculate incurred upper latency bound. To overcome that calculation issue, shapers or as in [TSN-ATS] their optimization, interleaved regulators, are used in [TSN-ATS] and GS. Shapers/interleaved requires to maintain across packets from the same flow per-flow state. Nevertheless, appropriate mathematical models for SDN controllers may be possible to develop deterministic per-hop forwarding models relying not only on the per-hop indicated latency but also on additional constraints such as limited number of hops or sufficiently low degrees of maximum admitted amount of traffic. Or else this may be used for to be developed latency models that are not 100% deterministic, but close enough in probability such that the amount of late packets would be in the same order as otherwise unavoidable problems such as BER based packet loss. To that end, the author of [I-D.stein-srtsn] has conducted simulations of the proposed mechanism, contrasting it with other mechanisms. These results, which will be published elsewhere, show hat this mechanism excels in cases with high load and a small number of flows with tight budgets. However, some small percentage of packets will miss their end-to-end latency bounds, and must be treated as lost packets. Depending on the algorithms chosen, solutions may or may not rely on strong, weak, or no clock synchronization across nodes. 7.4. Latency Based Forwarding "High-Precision Latency Forwarding over Packet-Programmable Networks", NOMS 2020 conference [LBF] describes a framework for per- transit-hop, per-flow stateless forwarding based on three packet parameters: The minimum and maximum desired end-to-end latency, set by the sender and not changed by the network, and the experienced latency updated by every hop. Routers supporting this LBF mechanism do also extend their routing (e.g.: IGP) to be able to calculate the non-queueing latency towards the destination. Based on the in-packet parameters and the future latency prediction are used to prioritize packets in queuing including giving them higher priority when they are late due to prior hop incurred latency, or delaying them when they are too early. LBF was started as a more fundamental research into how application experience could be improved when they are allowed to indicate such differential min/max latency Service Level Objectives (SLO). Benefits include the ability to compensate for prior hop incurred queuing latency, but also to automatically prioritize packets on a Eckert & Bryant Expires January 13, 2022 [Page 24] Internet-Draft bounded-latency-problems July 2021 single hop based on their future path length, all without the need for any explicit admission control. The LBF algorithm is completely without need for clock synchronization across nodes. Instead, it assumes mechanisms to know or learn link latency and the remaining latencies (as defined in the DetNet architecture) can be calculated locally (e.g.: latency through a forwarder). The authors have not yet tried to define a mathematical model that would allow to derive completely deterministic behavior for this original LBF algorithm in conjunction with a PCE/SDN controller. Due to the absence of per-flow (shaper/interleaved-regulator), the authors believe that deterministic solutions would as outlined above for SRTSN (Section 7.3) likely only be possible under additional assumed constraints. 8. Conclusions Bounded Latency for DetNet have been designed by trying to adopt solutions developed either several decades ago (GS) or recently for limited scope and scale L2 networks [TSN-ATS]. To allow DetNet solutions to explore opportunities in larger speed & scale shared network infrastructures, both private and service provider networks, it is highly desirable for DetNet WG (and/or other IETF WGs claiming responsibility in conjunction with DetNet as the driver) to explore the opportunities to standardize additional, and in the opinion of the authors better per-hop forwarding models in support of (near) deterministic bounded latency by mean of standardizing per-flow stateless/"DiffServ" style per-hop forwarding behavior (PHB) with appropriate network packet header parameters. 9. Security Considerations This document has no security considerations (yet?). 10. IANA Considerations This document has no IANA considerations. 11. Acknowledgements Thanks for Yaakov Stein for reviewing and proposing text for Section 7.3. Eckert & Bryant Expires January 13, 2022 [Page 25] Internet-Draft bounded-latency-problems July 2021 12. Informative References [CQF] IEEE Time-Sensitive Networking (TSN) Task Group., "IEEE Std 802.1Qch-2017: IEEE Standard for Local and Metropolitan Area Networks -- Bridges and Bridged Networks -- Amendment 29: Cyclic Queuing and Forwarding", 2017. [DNBL] Finn, N., Boudec, J. L., Mohammadpour, E., Zhang, J., Varga, B., and J. Farkas, "DetNet Bounded Latency", draft- ietf-detnet-bounded-latency-06 (work in progress), May 2021. [I-D.chen-DetNet-sr-based-bounded-latency] Chen, M., Geng, X., and Z. Li, "Segment Routing (SR) Based Bounded Latency", draft-chen-DetNet-sr-based-bounded- latency-01 (work in progress), May 2019. [I-D.dang-queuing-with-multiple-cyclic-buffers] Liu, B. and J. Dang, "A Queuing Mechanism with Multiple Cyclic Buffers", draft-dang-queuing-with-multiple-cyclic- buffers-00 (work in progress), February 2021. [I-D.ietf-bier-te-arch] Eckert, T., Cauchie, G., and M. Menth, "Tree Engineering for Bit Index Explicit Replication (BIER-TE)", draft-ietf- bier-te-arch-10 (work in progress), July 2021. [I-D.qiang-DetNet-large-scale-DetNet] Qiang, L., Geng, X., Liu, B., Eckert, T., Geng, L., and G. Li, "Large-Scale Deterministic IP Network", draft-qiang- DetNet-large-scale-DetNet-05 (work in progress), September 2019. [I-D.stein-srtsn] Stein, Y. (., "Segment Routed Time Sensitive Networking", draft-stein-srtsn-00 (work in progress), February 2021. [LBF] Clemm, A. and T. Eckert, "High-Precision Latency Forwarding over Packet-Programmable Networks", IEEE 2020 IEEE/IFIP Network Operations and Management Symposium (NOMS 2020), doi 10.1109/NOMS47738.2020.9110431, April 2020. [RFC2205] Braden, R., Ed., Zhang, L., Berson, S., Herzog, S., and S. Jamin, "Resource ReSerVation Protocol (RSVP) -- Version 1 Functional Specification", RFC 2205, DOI 10.17487/RFC2205, September 1997, . Eckert & Bryant Expires January 13, 2022 [Page 26] Internet-Draft bounded-latency-problems July 2021 [RFC2210] Wroclawski, J., "The Use of RSVP with IETF Integrated Services", RFC 2210, DOI 10.17487/RFC2210, September 1997, . [RFC2211] Wroclawski, J., "Specification of the Controlled-Load Network Element Service", RFC 2211, DOI 10.17487/RFC2211, September 1997, . [RFC2212] Shenker, S., Partridge, C., and R. Guerin, "Specification of Guaranteed Quality of Service", RFC 2212, DOI 10.17487/RFC2212, September 1997, . [RFC2475] Blake, S., Black, D., Carlson, M., Davies, E., Wang, Z., and W. Weiss, "An Architecture for Differentiated Services", RFC 2475, DOI 10.17487/RFC2475, December 1998, . [RFC3031] Rosen, E., Viswanathan, A., and R. Callon, "Multiprotocol Label Switching Architecture", RFC 3031, DOI 10.17487/RFC3031, January 2001, . [RFC3209] Awduche, D., Berger, L., Gan, D., Li, T., Srinivasan, V., and G. Swallow, "RSVP-TE: Extensions to RSVP for LSP Tunnels", RFC 3209, DOI 10.17487/RFC3209, December 2001, . [RFC6513] Rosen, E., Ed. and R. Aggarwal, Ed., "Multicast in MPLS/ BGP IP VPNs", RFC 6513, DOI 10.17487/RFC6513, February 2012, . [RFC8279] Wijnands, IJ., Ed., Rosen, E., Ed., Dolganow, A., Przygienda, T., and S. Aldrin, "Multicast Using Bit Index Explicit Replication (BIER)", RFC 8279, DOI 10.17487/RFC8279, November 2017, . [RFC8296] Wijnands, IJ., Ed., Rosen, E., Ed., Dolganow, A., Tantsura, J., Aldrin, S., and I. Meilik, "Encapsulation for Bit Index Explicit Replication (BIER) in MPLS and Non- MPLS Networks", RFC 8296, DOI 10.17487/RFC8296, January 2018, . [RFC8402] Filsfils, C., Ed., Previdi, S., Ed., Ginsberg, L., Decraene, B., Litkowski, S., and R. Shakir, "Segment Routing Architecture", RFC 8402, DOI 10.17487/RFC8402, July 2018, . Eckert & Bryant Expires January 13, 2022 [Page 27] Internet-Draft bounded-latency-problems July 2021 [RFC8570] Ginsberg, L., Ed., Previdi, S., Ed., Giacalone, S., Ward, D., Drake, J., and Q. Wu, "IS-IS Traffic Engineering (TE) Metric Extensions", RFC 8570, DOI 10.17487/RFC8570, March 2019, . [RFC8578] Grossman, E., Ed., "Deterministic Networking Use Cases", RFC 8578, DOI 10.17487/RFC8578, May 2019, . [RFC8655] Finn, N., Thubert, P., Varga, B., and J. Farkas, "Deterministic Networking Architecture", RFC 8655, DOI 10.17487/RFC8655, October 2019, . [RFC8660] Bashandy, A., Ed., Filsfils, C., Ed., Previdi, S., Decraene, B., Litkowski, S., and R. Shakir, "Segment Routing with the MPLS Data Plane", RFC 8660, DOI 10.17487/RFC8660, December 2019, . [RFC8964] Varga, B., Ed., Farkas, J., Berger, L., Malis, A., Bryant, S., and J. Korhonen, "Deterministic Networking (DetNet) Data Plane: MPLS", RFC 8964, DOI 10.17487/RFC8964, January 2021, . [RFC8986] Filsfils, C., Ed., Camarillo, P., Ed., Leddy, J., Voyer, D., Matsushima, S., and Z. Li, "Segment Routing over IPv6 (SRv6) Network Programming", RFC 8986, DOI 10.17487/RFC8986, February 2021, . [TSN-ATS] Specht, J., "P802.1Qcr - Bridges and Bridged Networks Amendment: Asynchronous Traffic Shaping", IEEE , July 2020, . [UBS] Specht, J. and S. Samii, "Urgency-Based Scheduler for Time-Sensitive Switched Ethernet Networks", IEEE 28th Euromicro Conference on Real-Time Systems (ECRTS), 2016. Authors' Addresses Toerless Eckert Futurewei Technologies USA 2220 Central Expressway Santa Clara CA 95050 USA Email: tte@cs.fau.de Eckert & Bryant Expires January 13, 2022 [Page 28] Internet-Draft bounded-latency-problems July 2021 Stewart Bryant Stewart Bryant Ltd Email: sb@stewartbryant.com Eckert & Bryant Expires January 13, 2022 [Page 29]