DetNet D. Guo Internet Draft S. Xu Intended status: Informational New H3C Technologies Co., Ltd Expires: April 23, 2024 October 23, 2023 Jitter Reduction Mechanism for DetNet draft-guo-detnet-jitter-reduction-mechanism-01 Abstract In large-scale deterministic networks (LDN), App-flows need to span multiple deterministic network domains, and the latency in multiple domains is added together. The jitter will be increased. In order to realize the protection service function, App-flows should be transmitted on multiple paths. The delay difference in data transmission on different paths is no different from jitter in end- to-end services. Jitter generated by various factors needs to be controlled to meet business requirements. This document describes the end-to-end jitter reduction mechanism in an LDN. This mechanism can effectively control the end-to-end jitter to meet specific business needs and make the planning of multiple paths for service protection more flexible. Status of this Memo This Internet-Draft is submitted in full conformance with the provisions of BCP 78 and BCP 79. This Internet-Draft is submitted in full conformance with the provisions of BCP 78 and BCP 79. This document may not be modified, and derivative works of it may not be created, and it may not be published except as an Internet-Draft. This Internet-Draft is submitted in full conformance with the provisions of BCP 78 and BCP 79. This document may not be modified, and derivative works of it may not be created, except to publish it as an RFC and to translate it into languages other than English. This document may contain material from IETF Documents or IETF Contributions published or made publicly available before November 10, 2008. The person(s) controlling the copyright in some of this material may not have granted the IETF Trust the right to allow modifications of such material outside the IETF Standards Process. Without obtaining an adequate license from the person(s) controlling the copyright in such materials, this document may not be modified outside the IETF Standards Process, and derivative works of it may not be created outside the IETF Standards Process, except to format Guo, et al. Expires April 23, 2024 [Page 1] Internet-Draft Jitter Reduction Mechanism for DetNet October 2023 it for publication as an RFC or to translate it into languages other than English. Internet-Drafts are working documents of the Internet Engineering Task Force (IETF), its areas, and its working groups. Note that other groups may also distribute working documents as Internet- Drafts. Internet-Drafts are draft documents valid for a maximum of six months and may be updated, replaced, or obsoleted by other documents at any time. It is inappropriate to use Internet-Drafts as reference material or to cite them other than as "work in progress." The list of current Internet-Drafts can be accessed at http://www.ietf.org/ietf/1id-abstracts.txt The list of Internet-Draft Shadow Directories can be accessed at http://www.ietf.org/shadow.html This Internet-Draft will expire on April 23, 2024. Copyright Notice Copyright (c) 2023 IETF Trust and the persons identified as the document authors. All rights reserved. This document is subject to BCP 78 and the IETF Trust's Legal Provisions Relating to IETF Documents (http://trustee.ietf.org/license-info) in effect on the date of publication of this document. Please review these documents carefully, as they describe your rights and restrictions with respect to this document. Code Components extracted from this document must include Simplified BSD License text as described in Section 4.e of the Trust Legal Provisions and are provided without warranty as described in the Simplified BSD License. Table of Contents 1. Introduction...................................................3 2. Terminology....................................................4 3. Architecture...................................................6 4. Proposal Description...........................................7 4.1. Data-Plane Overview......................................10 4.1.1. Residency delay processing method 1.................11 4.1.2. Residency delay processing method 2.................13 4.1.3. residence time adjustment...........................13 Guo, et al. Expires April 23, 2024 [Page 2] Internet-Draft Jitter Reduction Mechanism for DetNet October 2023 4.2. Control-Plane Overview...................................13 5. Robustness considerations.....................................14 6. Security Considerations.......................................14 7. IANA Considerations...........................................15 8. References....................................................15 8.1. Normative References.....................................15 9. Acknowledgments...............................................16 10. Authors' Addresses...........................................17 1. Introduction /----\ / \ | DetNet3 | \ / / \--------/ \ /-------\/ \/-------\ / \ / \ | DetNet2| | DetNet4 | \ / \ / \-----/ \-----/ | | ---|--- --|--- / \ / \ | DetNet1 | | DetNet5 | \ / \ / \--|--/ \--|--/ | | Talker Listener Figure 1 Multiple domains In deterministic networks, as stated in [I-D.ietf-detnet-scaling- requirements], end-to-end service may across multiple network domains and adopt a variety of different queuing mechanisms within each domain. For end-to-end services spanning multiple domains, jitter exists in various factors: o Scheduling and traffic admission control at domain boundaries may cause jitter; o The jitter generated by queuing and forwarding mechanisms in the DetNet domain, such as the [IEEE 802.1 QCR] asynchronous shaping method; Guo, et al. Expires April 23, 2024 [Page 3] Internet-Draft Jitter Reduction Mechanism for DetNet October 2023 o Flow aggregation generates jitter. In [RFC 8938], the DetNet data plane also allows for the aggregation of DetNet flows, which can improve scalability by reducing the per-hop state. When the aggregated flows are scheduled, the jitter of the flows cannot be precisely controlled. At the same time, according to [I-D.ietf-detnet-scaling- requirements], large-scale deterministic networks should support cross-domain asynchronous clocks; in addition, multiple domains may be heterogeneous networks (such as TSN, DetNet IP and 5GS). The factors all determine that it is difficult to reduce jitter between domains through a mechanism similar to CSQF[I-D.chen-detnet-sr- based-bounded-latency] or TCQF[I-D.draft-eckert-detnet-tcqf]. While the jitter generated by various factors in the end-to-end transmission path is accumulated, it may not meet the applications' requirements on jitter. This document describes a mechanism to reduce the jitter introduced by a variety of factors in scaling DetNet. 2. Terminology The following terminology is introduced in this document: Actual Delay (ActD): The actual transmission delay of a deterministic data packet passing through a specified network domain is called the Actual Time. Reference Delay (RefD): The maximum delay for packets of DetNet flows to pass through the DetNet domain. Fixed delay (FixD): The approximately constant part of the end-to- end transmission delay of the data packets of the App-flows through the DetNet. Path Reference Delay (PthRefD): The maximum end-to-end transmission delay of data packets of App-flows through the DetNet. Path Actual Delay (PthActD): The end-to-end actual transmission delay of a certain packet of App-flows through the DetNet. Compensation Delay (CompD): The delay required to compensate the actual delay according to the reference delay. Guo, et al. Expires April 23, 2024 [Page 4] Internet-Draft Jitter Reduction Mechanism for DetNet October 2023 Clock Source: Used as a clock source for subnet time synchronization. I-Gw: The ingress gateway of the deterministic subnet. E-Gw: The egress gateway of the deterministic subnet. HEAD NODE: The I-Gw of the first DetNet domain accessed by the App- flows in the DetNet forwarding path is HEAD NODE. COMPENSATION NODE: This node performs calculation and compensation for time delay. INGRESS-RELAY NODE: The I-Gw gateway of each domain except the HEAD NODE. EGRESS-RELAY NODE: The E-Gw gateway of each domain except the COMPENSATION NODE. Virtual Clock Reference Plane (VCRP): Provides a frequency synchronization reference for the clock used for DetNet data plane [DDP] timing. Guo, et al. Expires April 23, 2024 [Page 5] Internet-Draft Jitter Reduction Mechanism for DetNet October 2023 3. Architecture *------------------------------------------------------------* / / / / / Virtual Clock Reference Plane / / (VCRP) / / / *------+----------+----------------------+------------+------* | | | | | +----+----+ +----+----+ | | | | | | | | | Clock | | Clock | | | | Source2 | | Source3 | | | | | | | | | +---------+ +---------+ | | / \ / \ | | / \ / \ | | +-------+ +-------+ +-------+ +-------+ | | | I-Gw2 +--+ E-Gw2 +---+ I-Gw3 +--+ E-Gw3 | | | +---+---+ +-------+ +-------+ +---+---+ | | | | | | | | | | +-------+ +------+ | | | | | +---+-----+ | | +----+----+ | | | | | | | Clock | | | | Clock | | Source1 | | | | Source4 | | | | | | | +---------+ | | +---------+ / \ | | / \ / \ | | / \ +-------+ +-------+ | | +-------+ +-------+ | I-Gw1 +--+ E-Gw1 +--+ +----+ I-Gw4 +--+ E-Gw4 | +---+---+ +-------+ +-------+ +---+---+ | | | | Talker Listener Figure 2 Architecture Guo, et al. Expires April 23, 2024 [Page 6] Internet-Draft Jitter Reduction Mechanism for DetNet October 2023 In Figure 2, the Clock Source is a part of VCRP. I-Gw1 is HEAD NODE. E-Gw1, E-Gw2, E-Gw3 are EGRESS-RELAY NODE. I-Gw2, I-Gw3 are INGRESS- RELAY NODE. E-Gw4 is COMPENSATION NODE. The I-Gw gateway of each domain except the HEAD NODE should extract the packet receiving time, and calculate the actual delay of the previous domain, and carry it into the packet. The E-Gw gateway of each domain except the COMPENSATION NODE should save the sending time information of the domain. It is difficult to improve the accuracy of time synchronization, because VCRP may have a large geographical span. Clock source of VCRP provides time synchronization for each DetNet domain. Time synchronization is required within each DetNet domain, not between DetNet domains. Each domain is frequency-synchronized with the clock source provided by VCRP to avoid excessive deviation caused by each domain due to the influence of the environment. Because the clock sources that provide synchronization references for each DetNet domain in VCRP may not physically connected, it is difficult to achieve time synchronization in VCRP. On the premise that the transmission delay in each domain is not large, the frequency accuracy of the clock used for timing is relatively low. It is relatively easy for VCRP to provide a stable clock source with a certain accuracy for each DetNet domain for time synchronization. Each compensation needs a reference value, and compensation redundancy must be considered. When a data packet undergoes delay compensation multiple times, the end-to-end delay will be increased. No matter where delay compensation is performed, there is still the possibility that multiple transmitted data after compensation need to be queued, resulting in queuing delay, which reduces the compensation effect. Compensating at the very edge of the network aims to provide a standard way to reduce end-to-end jitter, applicable to various mechanisms, to achieve the best possible results with the smallest implementation cost. 4. Proposal Description Basic idea of the scheme: when establishing a deterministic stream session, obtain the reference delay of the path, obtain the actual delay of the data packet during transmission, calculate the compensation delay according to the reference delay and the actual transmission delay, and compensate the transmission delay at the COMPENSATION NODE connected to the Listener. The implementation principle is described in detail below. Guo, et al. Expires April 23, 2024 [Page 7] Internet-Draft Jitter Reduction Mechanism for DetNet October 2023 | | | | +----------T2------------+ +----------T3------------+ | | | | +----------T'2-----------+ +----------T'3-----------+ | | | | | | | | +--------+--------+------+ +--------+--------+------+ | I-Gw2 | DetNet2| E-Gw2|----+ I-Gw2 | DetNet2| E-Gw2+-+ +----+---+--------+------+ +--------+--------+------+ | | | | | | | +--------------------+ +---------------------------+ | | | | | || || | +----------T1------------+| |+----------T4------------+ | || || | +----------T'2-----------+| |+----------T'4-----------+ | || || | | || || | +--------+--------+------+| |+--------+--------+------+ | I-Gw2 | DetNet2| E-Gw2++ ++ I-Gw4 | DetNet4| E-Gw4| +----+---+--------+------+ +--------+--------+---+--+ | | | | | | Talker Listener Figure 3 The timing model of transmission delay Figure 3 describes the abstract model of multi-domain transmission delay segmentation timing, where Tn (n=1~4 in Figure 3) is the reference delay RefDn of each domain, and this value is obtained according to the selection of queuing mechanism combined with engineering applications. For a specific deterministic path, there is a master clock in the same domain, and each node in the path will perform time synchronization with this clock, and finally obtain intra-domain time synchronization. The reference delay of the end-to-end path is: PthRefD = FixD + T1 + T2 + T3 + T4. Guo, et al. Expires April 23, 2024 [Page 8] Internet-Draft Jitter Reduction Mechanism for DetNet October 2023 If service protection is realized by sending copies of the same data packet through multiple paths, the reference delay of path i is: PthRefD_i = FixD_i + T1_i + T2_i + T3_i + T4_i. Select the reference delay of the path with the largest one as the common reference delay of all the paths, and the final reference delay is: PthRefD = ceil(PthRefD1, PthRefD2, ..., PthRefDn). Let Tn (n=1~4 in Figure 3) be the actual transmission delay ActD of a certain data packet in each domain, and it depends on the time entering the domain and the time of exiting the domain when the data packet is actually delivered. The residence time in domain can also be obtained after the data packet is transmitted. Therefore, the actual transmission delay of this packet is: PthActD = FixD + T1 + T2 + T3 + T4. The delay of the packet needs to be compensated at CompD: CompD = PthRefD - PthActD = (PthRefD - FixD) - (T1 + T2 + T3+ T4). If service protection is achieved by sending copies of the same data packet through multiple paths, the actual delay of path i is PthActD_i = FixD_i + T1_i + T2_i + T3_i + T4_i. The delay of the packet needs to be compensated after transmission in path i is CompD: CompD = (PthRefD - FixD_i) - (T1_i + T2_i + T3_i + T4_i) The formula can be simplified as: CompD = Cap - (T1 + T2 + T3 + T4) (Formula 1), or CompD = Cap_i - (T1_i + T2_i + T3_i + T4_i) (Formula 2), where (PthRefD - FixD) is Cap, and (PthRefD - FixD_i) is Cap_i. Cap or Cap_i can be obtained directly or indirectly before deploying a Guo, et al. Expires April 23, 2024 [Page 9] Internet-Draft Jitter Reduction Mechanism for DetNet October 2023 deterministic streaming session. The specific obtaining method will be added in subsequent updates. Compensation is performed on E-Gw4 according to Formula 1 or Formula 2 when the DetNet data packet is transmitted. 4.1. Data-Plane Overview In the data plane, it is necessary to obtain the reference delay from the HEAD NODE to the COMPENSATION NODE. During actual transmission, collect the actual transmission delay in each domain, and compensate the delay at the COMPENSATION NODE based on the reference delay. For delay collection, two methods provide the relevant information required by each gateway: Method 1: Specify the operation of the subsequent gateway in the HEAD NODE. After the HEAD NODE identifies the flow, it encapsulates the relevant information into a data packet, and the subsequent gateway node performs corresponding operations according to the information in the data packet. The advantage of this method is that it simplifies the subsequent gateway configuration, but the disadvantage is that the overhead is large. Method 2: Configure the relevant information in the edge gateway along the path, and the edge gateway node will identify the DetNet flow and then obtain the information from the configuration for corresponding operations. The advantage of this method is that the overhead is small, and the disadvantage is that subsequent gateways need flow-by-flow configuration. The specific operation will be supplemented in subsequent updates. Guo, et al. Expires April 23, 2024 [Page 10] Internet-Draft Jitter Reduction Mechanism for DetNet October 2023 +-------------+ +-------------+ +-------------+ | HEAD +----+EGRESS RELAY +----+INGRESS RELAY+-+ +-------------+ +-------------+ +-------------+ | | +-------------------------------------------------------+ | | +-------------+ +-------------+ +-------------+ +-+EGRESS RELAY +----|INGRESS RELAY+----+COMPENSATION | +-------------+ +-------------+ +-------------+ Figure 4 Data Flow 4.1.1. Residency delay processing method 1 In order to obtain more accurate residence time, the timestamp is obtained at a lower layer (for example, the PMA sublayer of Ethernet). Only the sending timestamp is filled in EGRESS-RELAY, and the residence time in the domain is not calculated. Calculation is performed at the INGRESS-RELAY of the next network domain. On the one hand, it can reduce the transmission of metadata within the device, and on the other hand, it can reduce the calculation implementation of the hardware layer. The process required for the different roles of the gateway that DetNet flows pass through is described as follows. When receiving a packet, the HEAD NODE performs the following process: 1. Identify the DetNet flow and obtain the cross-domain information provided by the control plane. This cross-domain information includes the actions of the subsequent gateways, which can be encapsulated in the packet or configured by the control plane to the subsequent gateways. 2. Obtain the time when this node receives the packet. 3. Fill the cross-domain information and the time that this node receives the packet into the packet. When receiving a packet, the INGRESS-RELAY NODE performs the following process: Guo, et al. Expires April 23, 2024 [Page 11] Internet-Draft Jitter Reduction Mechanism for DetNet October 2023 1. Identify the DetNet flow and obtain the cross-domain information provided by the control plane from the packet or from the configuration. The gateway takes actions according to the information. The main procedure is: a) Obtain the time when this node receives the packet. b) Calculate the residence delay of the previous domain. 2. Fill the residence delay of the previous domain and the time when this node receives the packet into the packet. When receiving a packet, the EGRESS-RELAY NODE performs the following process: 1. Identify the DetNet flow and obtain the cross-domain information provided by the control plane from the packet or from the configuration. The gateway takes actions according to the information. The main procedure is: a) Fill the sending time of the packet into the specified position of the packet when a packet is sent. When receiving a packet, the COMPENSATION NODE performs the following process: 1. Identify the DetNet flow and obtain the cross-domain information provided by the control plane from the packet or from the configuration. The gateway takes actions according to the information. The main procedure is: a) Obtain the time when this node receives the packet and calculate the residence delay of the packet in this domain. b) Accumulate the residence delays of the packet in each domain. c) Calculate the compensation delay. 2. Send out the packet after completing the compensation. Guo, et al. Expires April 23, 2024 [Page 12] Internet-Draft Jitter Reduction Mechanism for DetNet October 2023 4.1.2. Residency delay processing method 2 The basic processing is similar to Section 4.2.1. The sending timestamp is obtained in EGRESS-RELAY, and combined with the receiving timestamp passed by the metadata, the residence time in the domain is calculated. More metadata is required to be transferred within the device. At the same time, the device needs additional hardware layer calculation implementation, which needs to be carefully implemented to calculate the processing delay after obtaining the timestamp into the residence delay. 4.1.3. residence time adjustment In scenarios with high accuracy requirements, it is necessary to consider the timing frequency offset of the clock source of each domain relative to the final compensation node. For example, if 10ms in a certain domain is equivalent to 10.1ms in the compensation node, then the dwell delay needs to be calculated in the domain, and an adjustment of 1% is added to the calculated value. This adjustment ratio needs to be obtained before deploying deterministic flows, and should be considered to be stored in the message or configured in EGRESS-RELAY or INGRESS-RELAY. When topology changes affect this adjustment ratio, corresponding updates need to be made. 4.2. Control-Plane Overview The control plane needs to cooperate with the data plane to complete the following transactions: 1. Define gateway operations along the path and configure data plane gateways. 2. Cooperate with the data plane to complete the measurement and calculate the reference delay. 3. Configure the reference delay information to the HEAD NODE or COMPENSATION NODE. 4. Maintenance during runtime: periodically collect the reference delay of each domain and calculate the reference delay of the whole path, and refresh the reference delay to the HEAD NODE or COMPENSATION NODE. For delay collection, the control plane has two methods to cooperate with the data plane to supply the relevant information required by each gateway: Guo, et al. Expires April 23, 2024 [Page 13] Internet-Draft Jitter Reduction Mechanism for DetNet October 2023 Method 1: The control plane globally distributes the gateway ID, and configures the ID to each edge gateway. The control plane configures the collected delay information to the HEAD NODE. Method 2: The control plane configures flow-by-flow operations to the domain edge gateways along the path. 5. Robustness considerations In a large-scale network, not only the topology changes, but also the transmission delay, which is regarded as a constant part, changes due to environmental factors. In response to topology changes, a certain amount of redundancy is reserved in the reference value of the compensation delay. For example, in the multi-path service protection that implements PREOF, if one of the paths fails, the transmission delay of the alternative path increases by 1ms (about 200 axioms). However, if the upper limit of the reference delay used for compensation reserves a redundancy of 1ms , after compensation, the business cannot perceive this change. However, if the delay considered as fixed transmission changes due to environmental factors, the upper limit of the reference delay for compensation needs to be adjusted. Considering that the change in transmission delay caused by environmental changes is a gradual change, it is feasible to regularly maintain the reference value of the compensation delay. The method of collecting reference delays will be described in detail in subsequent documents. However, if the transmission delay considered to be a constant part changes due to environmental factors, the upper limit of the reference delay for compensation needs to be adjusted. Considering that the change in transmission delay caused by environmental changes is a gradual change, it is feasible to regularly maintain the reference value of the compensation delay. The method of collecting reference delays will be described in detail in subsequent documents. 6. Security Considerations TBD Guo, et al. Expires April 23, 2024 [Page 14] Internet-Draft Jitter Reduction Mechanism for DetNet October 2023 7. IANA Considerations TBD 8. References 8.1. Normative References [RFC8655] Finn, N., Thubert, P., Varga, B., and J. Farkas, "Deterministic Networking Architecture", RFC 8655, DOI 10.17487/RFC8655, October 2019, . [I-D.ietf-detnet-scaling-requirements] Liu, P., Li, Y., Eckert, T., Xiong, Q., and J. Ryoo, "Requirements for Large-Scale Deterministic Networks", draft-liu-detnet-large-scale- requirements-05 (work in progress), October 2022. [IEEE802.1Qcr] IEEE, "IEEE Standard for Local and Metropolitan Area Networks -- Bridges and Bridged Networks - Amendment 34: Asynchronous Traffic Shaping", IEEE 802.1Qcr-2020, DOI 10.1109/IEEESTD.2020.9253013, 6 November 2020, . [I-D.chen-detnet-sr-based-bounded-latency] Chen, M., Geng, X., and Z. Li, "Segment Routing (SR) Based Bounded Latency", Work in Progress, Internet-Draft, draft-chen-detnet-sr-based- bounded-latency, 3 March 2023, [I-D.eckert-detnet-tcqf] Eckert, T. , Bryant, S., and A. G. Malis, "Deterministic Networking (DetNet) Data Plane - Tagged Cyclic Queuing and Forwarding (TCQF) for bounded latency with low jitter in large scale DetNets", [IEEE802.1AS] IEEE Time-Sensitive Networking (TSN) Task Group., "IEEE Std 802.1AS-2020: IEEE Standard for Local and Metropolitan Area Networks - Timing and Synchronization for Time Sensitive Applications ", 2020. Guo, et al. Expires April 23, 2024 [Page 15] Internet-Draft Jitter Reduction Mechanism for DetNet October 2023 [I-D.ietf-detnet-mpls-over-ip-preof] Varga, B., Farkas, J., Malis, A., "Deterministic Networking(DetNet): DetNet PREOF via MPLS over UDP/IP", Work in Progress, Internet-Draft, draft-ietf-detnet-mpls-over-ip-preof-02, 6 November 2022, < https://www.ietf.org/archive/id/draft-ietf-detnet-mpls- over-ip-preof-02.txt>. [IEEE802.1Qci] IEEE Time-Sensitive Networking (TSN) Task Group., "IEEE Std 802.1Qci-2017: IEEE Standard for Local and Metropolitan Area Networks - Bridges and Bridged Networks- Amendment 28: Per-Stream Filtering and Policing", 2017. [I-D.ietf-detnet-controller-plane-framework] Malis, A., Geng, X., Chen, M., Qin, F., arga, B., "Deterministic Networking (DetNet) Controller Plane Framework" , Work in Progress, Internet-Draft, draft-ietf-detnet-controller-plane- framework-02, 29 June 2022, . 9. Acknowledgments The editor wishes to thank and acknowledge the following contributors for contributing text to this document. Guo, et al. Expires April 23, 2024 [Page 16] Internet-Draft Jitter Reduction Mechanism for DetNet October 2023 Rubing Liu New H3C Technologies Co., Ltd 100094 Email: liurubing@h3c.com Ning Pan New H3C Technologies Co., Ltd 100094 Email: panning@h3c.com Xusheng Chen New H3C Technologies Co., Ltd 100094 Email: cxs@h3c.com Wei Wang New H3C Technologies Co., Ltd 100094 Email: david_wang@h3c.com YOU XUEJUN New H3C Technologies Co., Ltd 100094 Email: yoe@h3c.com 10. Authors' Addresses Daorong Guo New H3C Technologies Co., Ltd Beijing 100094 China Email: guodaorong@h3c.com Shenchao Xu New H3C Technologies Co., Ltd Hangzhou 310052 China Email: xushenchao@h3c.com Guo, et al. Expires April 23, 2024 [Page 17]