Detnet Working Group S. Bryant Internet-Draft M. Chen Intended status: Standards Track Huawei Technologies Expires: September 6, 2018 March 05, 2018 Operation of Deterministic Networks over MPLS draft-bryant-detnet-mpls-dp-00 Abstract This document specifies Deterministic Networking data plane operation over an MPLS Packet Switched Networks. This document is a derivative work from draft-ietf-detnet-dp-sol-01. Whether this is published as a stand-alone text, or serves as a focal point to refine the MPLS design and the key point are subsequently re-merged with draft-ietf-detnet-dp-sol-01 is a matter for the DETNET WG, as is the editorship of any WG text. Status of This Memo This Internet-Draft is submitted in full conformance with the provisions of BCP 78 and BCP 79. Internet-Drafts are working documents of the Internet Engineering Task Force (IETF). Note that other groups may also distribute working documents as Internet-Drafts. The list of current Internet- Drafts is at http://datatracker.ietf.org/drafts/current/. Internet-Drafts are draft documents valid for a maximum of six months and may be updated, replaced, or obsoleted by other documents at any time. It is inappropriate to use Internet-Drafts as reference material or to cite them other than as "work in progress." This Internet-Draft will expire on September 6, 2018. Copyright Notice Copyright (c) 2018 IETF Trust and the persons identified as the document authors. All rights reserved. This document is subject to BCP 78 and the IETF Trust's Legal Provisions Relating to IETF Documents (http://trustee.ietf.org/license-info) in effect on the date of publication of this document. Please review these documents carefully, as they describe your rights and restrictions with respect Bryant & Chen Expires September 6, 2018 [Page 1] Internet-Draft Deterministic MPLS March 2018 to this document. Code Components extracted from this document must include Simplified BSD License text as described in Section 4.e of the Trust Legal Provisions and are provided without warranty as described in the Simplified BSD License. Table of Contents 1. Introduction . . . . . . . . . . . . . . . . . . . . . . . . 3 2. Terminology . . . . . . . . . . . . . . . . . . . . . . . . . 3 2.1. Terms used in this document . . . . . . . . . . . . . . . 3 2.2. Abbreviations . . . . . . . . . . . . . . . . . . . . . . 4 3. Requirements language . . . . . . . . . . . . . . . . . . . . 5 4. DetNet Over an MPLS Underlay . . . . . . . . . . . . . . . . 5 5. DetNet over MPLS Encapsulation Components . . . . . . . . . . 8 5.1. Basic Data Plane Encapsulation . . . . . . . . . . . . . 8 5.2. DetNet Control Word . . . . . . . . . . . . . . . . . . . 9 5.3. Flow identification . . . . . . . . . . . . . . . . . . . 10 5.4. OAM Indication . . . . . . . . . . . . . . . . . . . . . 11 5.5. Flow Aggregation . . . . . . . . . . . . . . . . . . . . 11 5.6. Aggregation at the LSP . . . . . . . . . . . . . . . . . 12 5.7. Aggregating DetNet flows as a new DetNet flow . . . . . . 12 6. Simple Aggregation at the DetNet layer . . . . . . . . . . . 13 7. Indication of the DetNet Payload Type. . . . . . . . . . . . 13 8. Operation of the PREF Functions . . . . . . . . . . . . . . . 14 8.1. Operation of PR . . . . . . . . . . . . . . . . . . . . . 14 8.2. Operation of EF . . . . . . . . . . . . . . . . . . . . . 15 8.3. Packet reordering considerations . . . . . . . . . . . . 15 8.4. Indication of PREF processing . . . . . . . . . . . . . . 16 8.5. Placement of PR and EF in a node . . . . . . . . . . . . 16 9. Operation at DetNet Node types . . . . . . . . . . . . . . . 17 9.1. Operation at an Ingress Edge (PE) Node . . . . . . . . . 17 9.2. Operation at a Relay node (S-PE) . . . . . . . . . . . . 17 9.3. Operation at an Egress Edge (PE) Node . . . . . . . . . . 18 9.4. Operation at a Transit(P) Node . . . . . . . . . . . . . 18 10. Other DetNet data plane considerations . . . . . . . . . . . 19 10.1. Class of Service . . . . . . . . . . . . . . . . . . . . 19 10.2. Quality of Service . . . . . . . . . . . . . . . . . . . 19 10.3. Bidirectional traffic . . . . . . . . . . . . . . . . . 20 10.4. Layer 2 addressing and QoS Considerations . . . . . . . 21 11. Management and control considerations . . . . . . . . . . . . 22 11.1. S-Label assignment and distribution . . . . . . . . . . 22 11.2. Explicit routes . . . . . . . . . . . . . . . . . . . . 22 12. Security considerations . . . . . . . . . . . . . . . . . . . 22 13. IANA considerations . . . . . . . . . . . . . . . . . . . . . 23 13.1. Acknowledgments . . . . . . . . . . . . . . . . . . . . 23 14. References . . . . . . . . . . . . . . . . . . . . . . . . . 23 14.1. Normative References . . . . . . . . . . . . . . . . . . 23 14.2. Informative References . . . . . . . . . . . . . . . . . 23 Bryant & Chen Expires September 6, 2018 [Page 2] Internet-Draft Deterministic MPLS March 2018 Authors' Addresses . . . . . . . . . . . . . . . . . . . . . . . 26 1. Introduction This document is a derivative work from [draft-ietf-detnet-dp-sol- 01]. Editor's Note: We need to point to the exact version that this was derived from, not a generic version of [I-D.ietf-detnet-dp-sol]. Whether this is published as a stand-alone text, or serves as a focal point to refine the MPLS design and the key point are subsequently re-merged with draft-ietf-detnet-dp-sol-01 is a matter for the DETNET WG. Deterministic Networking (DetNet) is a service that can be offered by a network to DetNet flows. DetNet provides these flows extremely low packet loss rates and assured maximum end-to-end delivery latency. General background and concepts of DetNet can be found in [I-D.ietf-detnet-architecture]. This document specifies the encapsulation and operation of deterministic networking over an MPLS data-plane. The approach is modeled on the operation of Pseudowires (PW) over an MPLS Packet Switched Network (PSN) [RFC3985][RFC4385]. The DetNet transport layer functionality that provides congestion protection for DetNet flows is assumed to be in place in a DetNet node. This document does not define the associated control plane functions, or Operations, Administration, and Maintenance (OAM). It also does not specify traffic handling capabilities required to deliver congestion protection and latency control for DetNet flows at the DetNet transport layer. 2. Terminology 2.1. Terms used in this document Editor's note: This section needs to be reviewed when the body of the text is closer to completion. This document uses the terminology established in the DetNet architecture [I-D.ietf-detnet-architecture] and the DetNet Data Plane Solution Alternatives [I-D.ietf-detnet-dp-alt]. Bryant & Chen Expires September 6, 2018 [Page 3] Internet-Draft Deterministic MPLS March 2018 T-Label A label used to identify the LSP used to transport a DetNet flow across an MPLS PSN, e.g., a hop-by-hop label used between label switching routers (LSR). S-Label A DetNet "service" label that is used between DetNet nodes that implement also the DetNet service layer functions. An S-Label is also used to identify a DetNet flow at DetNet service layer. Local-ID A DetNet Edge and Relay node internal construct that uniquely identifies a DetNet flow within a node and never appear on- wire. It may be used to select proper forwarding and/or DetNet specific service function. PREF A Packet Replication and Elimination Function (PREF) does the replication and elimination processing of DetNet flow packets in edge or relay nodes. The replication function is essentially the existing 1+1 protection mechanism. The elimination function reuses and extends the existing duplicate detection mechanism to operate over multiple (separate) DetNet member flows of a DetNet compound flow. DetNet Control Word A control word used for sequencing and identifying duplicate packets at the DetNet service layer. 2.2. Abbreviations Editor's note: This section needs to be reviewed when the body of the text is closer to completion. The following abbreviations used in this document: AC Attachment Circuit. CE Customer Edge equipment. CoS Class of Service. CW Control Word. d-CW DetNet Control Word. DetNet Deterministic Networking. DF DetNet Flow. L2VPN Layer 2 Virtual Private Network. LSR Label Switching Router. Bryant & Chen Expires September 6, 2018 [Page 4] Internet-Draft Deterministic MPLS March 2018 MPLS Multiprotocol Label Switching. MPLS-TP Multiprotocol Label Switching - Transport Profile. MS-PW Multi-Segment Pseudowire (MS-PW). NSP Native Service Processing. OAM Operations, Administration, and Maintenance. PE Provider Edge. PREF Packet Replication and Elimination Function. PSN Packet Switched Network. PW Pseudowire. QoS Quality of Service. TSN Time-Sensitive Network. 3. Requirements language The key words "MUST", "MUST NOT", "REQUIRED", "SHALL", "SHALL NOT", "SHOULD", "SHOULD NOT", "RECOMMENDED", "MAY", and "OPTIONAL" in this document are to be interpreted as described in [RFC2119]. 4. DetNet Over an MPLS Underlay Figure 1 Shows the basic components of a DetNet enabled MPLS network used to transport TSN traffic using an MPLS transport. Bryant & Chen Expires September 6, 2018 [Page 5] Internet-Draft Deterministic MPLS March 2018 DetNet Edge Transit Relay Edge DetNet End Sys Node Node Node Node End Sys +-----+ End to End Service +-----+ |Appln|<..............................................>|Appln| +-----+ +---------+ DN Flow +---------+ +---------+ +-----+ | TSN | | Service |<------->| Service |<>| Service | | TSN | +-----+ +---+ +---+ +-----+ +---+ +---+ +---+ +---+ +-----+ |DNXpt| |Xpt| |LSP| | LSP | |LSP| |LSP| |LSP| |Xpt| |DNXpt| +--.--+ +-.-+ +-.-+ +-.-.-+ +-.-+ +-.-+ +-.-+ +-.-+ +--.--+ : : : : :Link : :Link : : : +-------+ /-------\+-----+ +------+ /-------\ TSN |Sub N/W| |TSN N/W| Link \-------/ \-------/ LSP = MPLS Transport DNXpt & Xpt = DetNet Transport Figure 1: A Simple DetNet Enabled MPLS Network TSN End Systems send and receive packets over the DetNet. The supported packet types are IP and Ethernet. Edge Nodes are responsible for the imposition and disposition of the required DetNet encapsulation. These are functionally similar to pseudowire (PW) Terminating Provide Edge (T-PE) equipment. Relay nodes are strategically placed and used enhance the reliability of delivery by enabling the replication of packets so that multiple copies, possibly over multiple paths. They also reduce the impact of replication by the eliminating surplus copies of DetNet packets. Replication and elimination may also be performed at ingress and egress edge nodes respectively. Edge nodes, and relay nodes are aware of the needs of particular DetNet flows and take care to process them in accordance with the required performance needs. Transit nodes are normal MPLS Label Switching Routers (LSRs). They are unaware of the special requirements of DetNet flows, although they may be configured to provide traffic engineering services to them to enhance the prospect of them meeting the required service level agreement (SLA). The MPLS LSP may be provided by any MPLS method (LSP, RSVP-TE, MPLS- TP, or MPLS Segment Routing (SR)). Bryant & Chen Expires September 6, 2018 [Page 6] Internet-Draft Deterministic MPLS March 2018 Figure 2 illustrates how end to end MPLS-based DetNet service is provided in a more detail. In this case, the end systems are able to send and receive DetNet flows. For example, an end system sends data encapsulated in MPLS. The 'X' in the end systems, edge and relay nodes represents potential DetNet flow packet replication and elimination points. Here the relay nodes may change the underlying transport, for example tunneling MPLS over IP [draft-xu-mpls-sr-over-ip], or simply interconnect network segments. DetNet DetNet Service Transit Transit Service DetNet | |<-Tnl->| |<-Tnl->| | DetNet End | V 1 V V 2 V | End System | +--------+ +--------+ +--------+ | System +---+ | | R1 |=======| R2 |=======| R3 | | +---+ | X...DFa...|._X_....|..DF1..|.__ ___.|..DF3..|...._X_.|.DFa..|.X | |CE1|========| \ | | X | | / |======|CE2| | | | | \_.|..DF2..|._/ \__.|..DF4..|._/ | | | | +---+ | |=======| |=======| | +---+ ^ +--------+ +--------+ +--------+ ^ | Relay Node Relay Node Relay Node | | | |<--------------- End to End DetNet Service -------------->| Figure 2: Flows in a DetNet Enabled MPLS Network An example MPLS DetNet network fragment and packet flow is illustrated in Figure 3. 1111 11111111 111111 222222 2222222 333 CE1----EN1--------R1-------R2-------R3--------EN2----CE2 \2 22222/ 3 / \2222222 /----+ 3 / +------R4------------------------+ 333333333333333333333333 Figure 3: Example Packet flow in DetNet Enabled MPLS Network Customer Equipment CE1 send a packet into the DetNet enabled MPLS network. Edge Node EN1 makes a copy of the packet and encapsulates each copy as a DetNet packet (packet 1 and packet 2). It sends one copy (1) to Relay Node R1 and the other copy (2) to Relay Node R4. R1 sends packet copy 1 to R2. R4 sends one copy to R2, and a further copy (3) to EN2. R2 receives copy (2) before copy 1, and so eliminated copy (1) sending only (2) to EN2. EN2 receives copy (3) first sending it to CE2 and eliminating copy (2). Note the number Bryant & Chen Expires September 6, 2018 [Page 7] Internet-Draft Deterministic MPLS March 2018 illustrates the replication instance and should not be confused with the sequence number which remains unchanged in all packet copies. The above is of course illustrative of many network scenarios that can be configured. Between a pair of relay nodes there may be one or more transport nodes that simply forward the DetNet traffic, but these are omitted for clarity. 5. DetNet over MPLS Encapsulation Components To carry DetNet over MPLS the following is required: 1. A method of identifying the MPLS payload type. 2. A method of identifying the DetNet flow group to the processing element. 3. A method of distinguishing DetNet OAM packets from DetNet data packets. 4. A method of carrying the DetNet sequence number. 5. A suitable LSP to deliver the packet to the egress PE. 6. A method of carrying queuing and forwarding indication. In this design an MPLS service label (the S-Label), similar to a pseudowire (PW) label [RFC3985], is used to identify both the DetNet flow identity and the payload MPLS payload type satisfying (1) and (2) in the list above. OAM traffic discrimination happens through the use of the Associated Channel method described in [RFC4385]. The sequence number is carried in the DetNet Control word which carries the Data/OAM discriminator. The LSP used to transport the DetNet packet may be of any type (MPLS-LDP, MPLS-TE, MPLS-TP[RFC5921], or MPLS-SR[I-D.ietf-spring-segment-routing-mpls]). The LSP (T-Label) label and/or the S-Label may be used to indicate the queue processing as well as the forwarding parameters. Note that when the network consists only of DetNet enabled nodes with no aggregation, Penultimate Hop Popping (PHP) means that the only label in the label stack is the S-label. 5.1. Basic Data Plane Encapsulation Figure 4 shows a DetNet data plane MPLS encapsulation. This is modeled on the encapsulation of pseudowires over MPLS [RFC3985]. The encapsulation consists of: Bryant & Chen Expires September 6, 2018 [Page 8] Internet-Draft Deterministic MPLS March 2018 o DetNet control word (d-CW) containing sequencing information for packet replication and duplicate elimination purposes, and the OAM indicator. There MUST a separate sequence number space for each DetNet flow. o DetNet Label (S-label) that identifies a DetNet flow to the peer node that is to process it. The S-Label is allocated from the platform label space. o Zero or more MPLS transport LSP label(s) (T-label) used to direct the packet along the label switched path (LSP) to the next peer node along the path. When Penultimate Hop Popping is in use there will be no label T-label in the protocol stack on the final hop. o The necessary data-link encapsulation is then applied prior to transmission over the physical media. RFC Editor - if you ever get this text please remove this para (text to make compiler work). +---------------------------------+ | | | DetNet Flow | | Payload Packet | | | +---------------------------------+ <--\ | DetNet Control Word | | +=================================+ +--> DetNet data plane | S-Label | | MPLS encapsulation +---------------------------------+ <--/ | T-Label(s) | +---------------------------------+ | Data-Link | +---------------------------------+ | Physical | +---------------------------------+ Figure 4: MPLS Encapsulation of DetNet Flow aggregation may be necessary to achieve the required scaling. The extensions to basic encapsulation needed to support flow aggregation are described in Section 5.5. 5.2. DetNet Control Word A DetNet Control Word (d-CW) conforms to the Generic PW MPLS Control Word (PWMCW) defined in [RFC4385] and is shown in Figure 5. In a DetNet data packet the upper nibble of the d-CW MUST be set to zero Bryant & Chen Expires September 6, 2018 [Page 9] Internet-Draft Deterministic MPLS March 2018 (0). The effective sequence number bit length is between 0 and 28 bits, and configured either by a control plane or manually for each DetNet flow. The sequence number is aligned to the right (least significant bits) and unused bits MUST be set to zero (0). Each DetNet flow MUST have its own sequence number counter. The sequence number is incremented by one for each new packet. Editor's note: We need to consider whether it is better to allow a multiplicity of sequence number lengths with a length configured for each flow, or a uniform sequence number of 28. On reflection it seems better to go for the simplicity of a standard length. If for any reason a different length becomes desirable, then it is relatively simple to define another type of DetNet d-CW with a different standard sequence number length and there is no ambiguity of operation since the sequence number length is a parameter of the S-Label. The d-CW MUST always be present in a packet. In cases where the sequence number is not used (e.g., for DetNet-t-flows) the control plane or the manual configuration has to define zero (0) bit length sequence number and the value of the sequence number MUST be set to zero (0). Editors Note: Do we set length zero or say it is not used? Also need to add text to stop relay nodes and egress edge nodes from processing the s/n otherwise only one packet ever gets through! 0 1 2 3 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ |0 0 0 0| Sequence Number | +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ Figure 5: DetNet Control Word 5.3. Flow identification DetNet flow identification at a DetNet service layer is realized by an S-label. It maps a DetNet flow to a specific d-CW in a DetNet node. For a data flow the S-label used for flow identification MUST be bottom label of the label stack for a DetNet-s- or DetNet-st-flow and MUST precede the d-CW. Editor's note: We have specified the above for _data_ to leave the door open for the GAL based OAM method as an alternative to the ACH mechanism currently specified. When using GAL the GAL label would be Bryant & Chen Expires September 6, 2018 [Page 10] Internet-Draft Deterministic MPLS March 2018 after the S-Label. Do we leave this option in or shut the door on it? An S-label for a single DetNet flow MUST be unique at the peer at the node that is to process it. The S-label is stored in the platform label table to allow for DetNet packet processing independent of the interface on which the packet is received on. 5.4. OAM Indication OAM follows the procedures set out in [RFC5085] with the restriction that only Virtual Circuit Connectivity Verification (VCCV) type 1 is supported. Editor's Note - is this restriction acceptable? Do we need to support GAL based OAM indication? As shown in Figure 3 of [RFC5085] when the first nibble of the d-CW is 0x0 the payload following the d-CW is normal user data. However, when the first nibble of the d-CW is 0X1, the payload that follows the d-DW is an OAM payload with the OAM type indicated by the value in the d-CW Channel Type field. The reader is referred to [RFC5085] for a more detailed description of the Associated Channel mechanism, and to the DetNet work on OAM for more information DetNet OAM. 5.5. Flow Aggregation The ability to aggregate individual flows, and their associated resource control, into a larger aggregate is an important technique for improving scaling of control in the data, management and control planes. The DetNet data plane allows for the aggregation of DetNet flows, to improved scaling. There are three methods of introducing flow aggregation: 1. Aggregate at the LSP (Transport) 2. Aggregating DetNet flows as a new DetNet flow 3. Simple Aggregation at the DetNet layer A further method of using SR to perform aggregation is for further study. The resource control and management aspects of aggregation (including the queuing/shaping/ policing implications) will be covered in other documents. Bryant & Chen Expires September 6, 2018 [Page 11] Internet-Draft Deterministic MPLS March 2018 5.6. Aggregation at the LSP DetNet flows transported via MPLS can leverage MPLS-TE's existing support for hierarchical LSPs (H-LSPs), see [RFC4206]. H-LSPs are typically used to aggregate control and resources, they may also be used to provide OAM or protection for the aggregated LSPs. Arbitrary levels of aggregation naturally falls out of the definition for hierarchy and the MPLS label stack [RFC3032]. DetNet nodes which support aggregation (LSP hierarchy) map one or more LSPs (labels) into and from an H-LSP. Both carried LSPs and H-LSPs may or may not use the TC field, i.e., L-LSPs or E-LSPs. Such nodes will need to ensure that traffic from aggregated LSPs are placed (shaped/policed/ enqueued) onto the H-LSPs in a fashion that ensures the required DetNet service is preserved. Additional details of the traffic control capabilities needed at a DetNet-aware node may be covered in the new service descriptions mentioned above or in separate future documents. Management and control plane mechanisms will also need to ensure that the service required on the aggregate flow (H-LSP or DSCP) are provided, which may include the discarding or remarking mentioned in the previous sections. 5.7. Aggregating DetNet flows as a new DetNet flow An aggregate can be built by layering DetNet flows as shown below: +---------------------------------+ | | | DetNet Flow | | Payload Packet | | | +---------------------------------+ <--\ | DetNet Control Word | | +=================================+ | | S-Label | | +---------------------------------+ +----DetNet data plane | DetNet Control Word | | MPLS encapsulation +=================================+ | | A-Label | | +---------------------------------+ <--/ | T-Label(s) | +---------------------------------+ | Data-Link | +---------------------------------+ | Physical | +---------------------------------+ Bryant & Chen Expires September 6, 2018 [Page 12] Internet-Draft Deterministic MPLS March 2018 Both the Aggregation (A) label and the S-label have their MPLS S bit set indicating bottom of stack, and the d-CW allows the PREF functions to work. It is a property of the A-label that what follows is d-CW followed by an S-label. A relay node processing the A-label would not know the underlying payload type. This would only be known to a node that was a peer of the node imposing the S-label. However there is no real need for it to know the payload type during aggregation processing. 6. Simple Aggregation at the DetNet layer Another approach would be not to include a d-CW for the aggregated flow. This would be functionally similar to aggregation at the transport layer using H-LSPs, but would confine knowledge of the aggregation to the DetNet layer. Such an approach shares the disadvantage that PREF operations would not be possible. OAM operation in this mode is for further study. +---------------------------------+ | | | DetNet Flow | | Payload Packet | | | +---------------------------------+ <--\ | DetNet Control Word | | +=================================+ | | S-Label | +----DetNet data plane +---------------------------------+ | MPLS encapsulation | A-Label | | +---------------------------------+ <--/ | T-Label(s) | +---------------------------------+ | Data-Link | +---------------------------------+ | Physical | +---------------------------------+ 7. Indication of the DetNet Payload Type. The only node types that needs to know the payload type is the ingress node which has to know how to process the packet it receives on the ingress AC, and egress edge node which has to know how to prepare the packet for transmission on the egress AC. On ingress a DetNet edge node has to classify the packets into those that are for transmission as Detnet packets and those that are for transmission as "normal" packets at one of more lower priorities. Bryant & Chen Expires September 6, 2018 [Page 13] Internet-Draft Deterministic MPLS March 2018 The packet type is indicated to the egress edge node through the value of the S-label. Thus when the egress edge node looks up the S-label one of the parameters returned is the packet type which in turn tells the egress edge node how to prepare the packet for transmission over the egress AC. Editor's note: Do we only find one type on the ingress and egress or do we need to run missed mode, i.e. will we have to send some packets across the network as Ethernet packets and some as IP packets? 8. Operation of the PREF Functions The Packet Replication and Elimination Functions (PREF) are designed to enhance the reliability of delivery by network layer packet replication (PR), whilst at the same time minimizing network congestion and the duplicate delivery of packets over an egress AC by eliminating duplicate packets (EF). PR and EF are independent functions that operate on a DetNet flow at strategic places in the network. The placement of a function is a matter for the network operator. 8.1. Operation of PR A PR function creates two or more copies of a packet, and forwards a copy to each of the designated peers of the replicating node. A PR function may be placed in an ingress edge node, a relay node or an egress edge node. We consider first a DetNet relay node. The packet is received from the upstream DetNet peer, and if present the T-label is popped. The S-label is looked up in the platform label table and if the forwarding instructions indicate that replication is required, the following happens for each next hop in the DetNet layer: o A copy of the payload is made. o An identical copy of the d-CW from the original packet is pushed. o The S-Label required by the next hop in the DetNet layer is pushed. o The DetNet packet copy is forwarded on the LSP to the next hop in the DetNet layer using normal MPLS forwarding. An ingress node operates in the same way as a relay node, except that it is responsible for initial encapsulation of the packet. The packet is received from the AC, classified and prepared for Bryant & Chen Expires September 6, 2018 [Page 14] Internet-Draft Deterministic MPLS March 2018 forwarding over the DetNet network as described in Section 9.1, except that for each next-hop in the DetNet Layer (i.e. each node that is to receive a copy of the DetNet packet) : o A copy of the payload is made. o The d-CW is constructed using the next sequence number in the sequence associated with this service. o An identical copy of the d-CW is pushed. o The S-Label required by the next hop in the DetNet layer to recognize this service is pushed. o The DetNet packet copy is forwarded on the LSP to the next hop in the DetNet layer using normal MPLS forwarding. An egress node operates in the same way as as described in Section 9.1, except that where the S-Label indicates that the the packet is to be forwarded to a multiply attached system in the payload layer a similar copy (modified as needed to conform to any MAC requirements) is forwarded out of each egress AC. Editor's Note: For normal PW, there will be one to one AC to PW binding relationship, for DetNet, no matter the ingress or egress side, there may be multiple AC corresponding to a single DetNet PW (S-label), should we state explicitly? 8.2. Operation of EF The EF function eliminates duplicate copies of a packet. The node identifies the service from the packet S-Label. If the S-Label indicates that the packet elimination function is required to operate on this service at this node, it uses the sequence number in the d-CW to determine whether or not the packet is a duplicate that must be eliminated, and precedes accordingly. The EF may be placed in a Relay node, or an Edge Egress node. 8.3. Packet reordering considerations The DetNet service layer introduces packet reordering functionality for use in DetNet edge and relay node and end system packet processing. The reordering functionality MAY be enabled in a DetNet node. The reordering functionality relies on a presence of sequence numbers the d-CW. Bryant & Chen Expires September 6, 2018 [Page 15] Internet-Draft Deterministic MPLS March 2018 8.4. Indication of PREF processing The indication that PR or EF processing is needed at a DetNet Relay node or a DetNet egress edge node is carried as an implicit characteristic of the S-Label. Thus, when a service is established the ingress edge node is configured to use an active rather than a static zero sequence number, and DetNet relays and egress edge nodes are configured to run PR and/or EF functionality on on services identified by specific S-labels. 8.5. Placement of PR and EF in a node This section is not normative. The placement of the PR and EF functions within the node is a matter for the node designers, and this specification makes no determination on this matter. However the placement may well have implications for the management and control of a node, and thus the following is worth noting. In a bladed system a common processing model is to analyze the packet for forwarding close to the ingress interface and to either to fully prepare it for forwarding as part of ingress processing, or to package it with internal metadata for final preparation close to the egress interface. In such systems the natural place to perform replication is as part of the ingress processing since egress processors often do not have the capability (for example processing power) to further process the packet, often do not normally have the required data paths to facilitate replication for transmission to other line cards. Furthermore, in bladed systems it is to be expected that a packet from a peer in the MPLS layer may arrive on any of the blades. Whilst in principle some constraints could be applied on which interfaces the packets will arrive on, experience shows that such constraints are operationally impractical. To perform elimination state must be shared amongst the forwarders performing elimination. Sharing the state required for elimination between forwarders on different blades without sacrificing performance is technically difficult. Thus, the natural place for elimination in a distributed design is close to the egress interface. On mono-processor systems these constraints do not apply in quite the same way, although even in this case it is necessary for the equipment designer to consider the implications of particular forwarder design, for example the allocation of Network Processing Unit (NPU) cores to particular interfaces. Bryant & Chen Expires September 6, 2018 [Page 16] Internet-Draft Deterministic MPLS March 2018 A bladed system may place the PR and EF functions on a processing function other than the ingress or egress forwarding processor, whereupon the mono-processor considerations apply. As noted above the placement of the PR and EF functions is a matter for the system designer. However it is important that nothing in the control, configuration, or OAM system results in undue difficulties for any of the forwarding models. 9. Operation at DetNet Node types This section considers the operations at the various DetNet node types in more detail. 9.1. Operation at an Ingress Edge (PE) Node An ingress Edge Node first classifies received traffic into DetNet flows and non-Detnet traffic. The DetNet flows are sent over the DetNet network using the procedures defined in the document. The classification process and the handling of non-DetNet traffic is out of scope for this document. The processing of non-DetNet flows is currently outside the scope of this document The packet is encapsulated as described in Section 5.1. First, if the flow is to be carried as IP the MAC header and checksum are removed. Otherwise the checksum is removed. The d-CW is constructed using the next sequence number associated with the service, and is pushed. The S-label corresponding to this DetNet flow is then pushed. The packet is then forwarded to the required egress edge- node by pushing the required T-Labels and the required data-link headers. Editor's Note: we need text on how we indicate IPv4 vs IPv6. In MPLS they are carried on different FECs. Presumable we need to have a different S-Label for each IP type. This implies a different s/n for each IP type, but since it seems unlikely that we need to maintain them in a common sequence that looks to be OK. 9.2. Operation at a Relay node (S-PE) At a relay node packet forwarding follows the same process used in a PW S-PE [RFC5659], save for the additional processing required for any required PREF processing. The packet is received from the incoming LSP and any T-labels which are still present are popped. The S-label is looked up in the Bryant & Chen Expires September 6, 2018 [Page 17] Internet-Draft Deterministic MPLS March 2018 platform label table to determine the forwarding parameters. If PREF processing is required the process described in Section 8 is followed, and if required the locally held sequence number information associated with the S-label is updated to avoid future duplicate forwarding. For each copy of the packet to be forwarded the S-label is swapped for the S-label required by either the next relay node, or by the egress edge node. For each copy of the packet, the T-label(s) needed to reach the next relay node or the egress edge node is pushed and the packet forwarded towards its next hop in the MPLS layer. 9.3. Operation at an Egress Edge (PE) Node At an egress edge node, the S-Label is looked up in the platform label table and is used to determine the egress packet processing parameters. The sequence number in d-CW and the recorded sequence number history is compared to perform any required ER processing Section 8.2. If the packet is not eliminated, the d-CW is stripped. If the packet is to be treated as an IP packet, this is processed in the normal way for an IP packet egressing an MPLS tunnel (for example the data-link header is constructed) and the packet forwarded towards its destination. Editor's note: Do we need to look the IP address up in a forwarding table, if so, which one, or is the next hop a forwarding parameter. Is the MAC address a forwarding parameter, or do we need to run ARP as normal? If the packet is to be treated as an Ethernet packet it is forwarded unmodified, save for the addition of the CRC as described in [RFC4448]. 9.4. Operation at a Transit(P) Node Operation at a transit (P) node is normal MPLS forwarding. The outer label is either swapped or popped as required, and the packet is forwarded along the LSP. If an entropy label is present in the label stack, this may be used by the Equal Cost Multi-Path (ECMP) selection process. No other label is inspected as part of forwarding. The Traffic Class field and/or incoming LSP transport label may be used to indicate how the packet is to be processed and queued including which forwarding resources are to be applied to the packet ((RFC3270}}. Any of the methods of constructing a physical LSP (for RSVP-TE signaling or MPLS-TP style controller based configuration) or a virtual LSP (Segment Routing Bryant & Chen Expires September 6, 2018 [Page 18] Internet-Draft Deterministic MPLS March 2018 [I-D.ietf-spring-segment-routing-mpls]) may be used to indicate not only the next hop, but the priority of processing and any physical resources dedicated to the service [I-D.bryant-rtgwg-enhanced-vpn]. 10. Other DetNet data plane considerations 10.1. Class of Service Class and quality of service, i.e., CoS and QoS, are terms that are often used interchangeably and confused. In the context of DetNet, CoS is used to refer to mechanisms that provide traffic forwarding treatment based on aggregate group basis and QoS is used to refer to mechanisms that provide traffic forwarding treatment based on a specific DetNet flow basis. Examples of existing network level CoS mechanisms include DiffServ which is enabled by IP header differentiated services code point (DSCP) field [RFC2474] and MPLS label traffic class field [RFC5462], and at Layer-2, by IEEE 802.1p priority code point (PCP). CoS for DetNet flows carried in PWs and MPLS is provided using the existing MPLS Differentiated Services (DiffServ) architecture [RFC3270]. Both E-LSP and L-LSP MPLS DiffServ modes MAY be used to support DetNet flows. The Traffic Class field (formerly the EXP field) of an MPLS label follows the definition of [RFC5462] and [RFC3270]. The Uniform, Pipe, and Short Pipe DiffServ tunneling and TTL processing models are described in [RFC3270] and [RFC3443] and MAY be used for MPLS LSPs supporting DetNet flows. MPLS ECN MAY also be used as defined in ECN [RFC5129] and updated by [RFC5462]. One additional consideration for DetNet nodes which support CoS services is that they MUST ensure that the CoS service classes do not impact the congestion protection and latency control mechanisms used to provide DetNet QoS. This requirement is similar to requirement for MPLS LSRs to that CoS LSPs do not impact the resources allocated to TE LSPs via [RFC3473]. 10.2. Quality of Service Quality of Service (QoS) mechanisms for flow specific traffic treatment typically includes a guarantee/agreement for the service, and allocation of resources to support the service. Example QoS mechanisms include discrete resource allocation, admission control, flow identification and isolation, and sometimes path control, traffic protection, shaping, policing and remarking. Example protocols that support QoS control include Resource ReSerVation Protocol (RSVP) [RFC2205] and RSVP-TE [RFC3209] and [RFC3473]. The existing MPLS mechanisms defined to support CoS [RFC3270] can also be used to reserve resources for specific traffic classes. Bryant & Chen Expires September 6, 2018 [Page 19] Internet-Draft Deterministic MPLS March 2018 In addition to explicit routes Section 11.2, and packet replication and elimination, described in Section 8 above, DetNet provides zero congestion loss and bounded latency and jitter. As described in [I-D.ietf-detnet-architecture], there are different mechanisms that maybe used separately or in combination to deliver a zero congestion loss service. These mechanisms are provided by the either the MPLS or IP layers, and may be combined with the mechanisms defined by the underlying network layer such as 802.1TSN. A baseline set of QoS capabilities for DetNet flows carried over MPLS can be provided with Traffic Engineering (MPLS-TE) [RFC3209] and [RFC3473]. TE LSPs can also support explicit routes (path pinning). Current service definitions for packet TE LSPs can be found in "Specification of the Controlled Load Quality of Service", [RFC2211], "Specification of Guaranteed Quality of Service", [RFC2212], and "Ethernet Traffic Parameters", [RFC6003]. Additional service definitions are expected in future documents to support the full range of DetNet services. In all cases, the existing label-based marking mechanisms defined for TE-LSPs and even E-LSPs are use to support the identification of flows requiring DetNet QoS. Packets that are marked with a DetNet Class of Service value, but that have not been the subject of a completed reservation, can disrupt the QoS offered to properly reserved DetNet flows by using resources allocated to the reserved flows. Therefore, the network nodes of a DetNet network MUST: o Defend the DetNet QoS by discarding or remarking (to a non-DetNet CoS) packets received that are not the subject of a completed reservation. o Not use a DetNet reserved resource, e.g. a queue or shaper reserved for DetNet flows, for any packet that does not carry a DetNet Class of Service marker. 10.3. Bidirectional traffic Some DetNet applications generate bidirectional traffic. Using MPLS definitions [RFC5654] there are associated bidirectional flows, and co-routed bidirectional flows. MPLS defines a point-to-point associated bidirectional LSP as consisting of two unidirectional point-to-point LSPs, one from A to B and the other from B to A, which are regarded as providing a single logical bidirectional transport path. This would be analogous of standard IP routing, or PWs running over two reciprocal unidirectional LSPs. MPLS defines a point-to- point co-routed bidirectional LSP as an associated bidirectional LSP which satisfies the additional constraint that its two unidirectional component LSPs follow the same path (in terms of both nodes and Bryant & Chen Expires September 6, 2018 [Page 20] Internet-Draft Deterministic MPLS March 2018 links) in both directions. An important property of co-routed bidirectional LSPs is that their unidirectional component LSPs share fate. In both types of bidirectional LSPs, resource allocations may differ in each direction. The concepts of associated bidirectional flows and co-routed bidirectional flows can be applied to DetNet flows as well. PWs [RFC3985] are always created as bidirectional constructs. While the MPLS data planes must support bidirectional DetNet flows, there are no special bidirectional features with respect to the data plane other than need for the two directions take the same paths. Fate sharing and associated vs co-routed bidirectional flows can be managed at the control level. Note, that there is no stated requirement for bidirectional DetNet flows to be supported using the same MPLS Labels in each direction, and indeed to do so would introduce significant implementation issues. Control mechanisms will be need to support such bidirectional flows DetNet over MPLS, but such mechanisms are out of scope of this document. An example control plane solution for MPLS can be found in [RFC7551]. 10.4. Layer 2 addressing and QoS Considerations Editor's note: Add references needed by this section The Time-Sensitive Networking (TSN) Task Group of the IEEE 802.1 Working Group have defined (and are defining) a number of amendments to IEEE 802.1Q [IEEE8021Q] that provide zero congestion loss and bounded latency in bridged networks. IEEE 802.1CB [IEEE8021CB] defines packet replication and elimination functions that should prove both compatible with and useful to, DetNet networks. As is the case for DetNet, a Layer 2 network node such as a bridge may need to identify the specific DetNet flow to which a packet belongs in order to provide the TSN/DetNet QoS for that packet. It also will likely need a CoS marking, such as the priority field of an IEEE Std 802.1Q VLAN tag, to give the packet proper service. Although the flow identification methods described in IEEE 802.1CB [IEEE8021CB] are flexible, and in fact, include IP 5-tuple identification methods, the baseline TSN standards assume that every Ethernet frame belonging to a TSN stream (i.e. DetNet flow) carries a multicast destination MAC address that is unique to that flow within the bridged network over which it is carried. Furthermore, IEEE 802.1CB [IEEE8021CB] describes three methods by which a packet sequence number can be encoded in an Ethernet frame. Ensuring that the proper Ethernet VLAN tag priority and destination MAC address are used on a DetNet/TSN packet may require further Bryant & Chen Expires September 6, 2018 [Page 21] Internet-Draft Deterministic MPLS March 2018 clarification of the customary L2/L3 transformations carried out by routers and edge label switches. Edge nodes may also have to move sequence number fields among Layer 2, PW, and IPv6 encapsulations. 11. Management and control considerations While management plane and control planes are traditionally considered separately, from the Data Plane perspective there is no practical difference based on the origin of flow provisioning information. This document therefore does not distinguish between information provided by a control plane protocol, e.g., RSVP-TE [RFC3209] and [RFC3473], or by a network management mechanisms, e.g., RestConf [RFC8040] and YANG [RFC7950]. Editor's note: Not sure if RSVP-TE can be used, indeed unsure what routing protocols can be use other than to create point to point MPLS transport paths. Normally we construct a a single path through the network with RSVP-TE, but here we need to construct an explicit mesh at the DetNet layer. The classical routing protocols are not really capable of constructing graphs of the sort needed here either. 11.1. S-Label assignment and distribution A mechanism based on the existing PW label distribution protocol [RFC8077] can be used to exchange labels between DetNet nodes. The protocol may however need extending depending on the preferred format of the DetNet flow identifiers. A mechanism to create the flow graph through the relay nodes will need to be specified. Initially this is likely to be based on a network controller of some sort rather than a distributed routing protocol. The details of the control plane protocol solution required for the label distribution and the management of the label number space are out of scope of this document. 11.2. Explicit routes Editor's note describe the applicability of explicit routes as a method of constructing paths 12. Security considerations The security considerations of DetNet in general are discussed in [I-D.ietf-detnet-architecture] and [I-D.sdt-detnet-security]. Other security considerations will be added in a future version of this draft. Bryant & Chen Expires September 6, 2018 [Page 22] Internet-Draft Deterministic MPLS March 2018 13. IANA considerations This document makes no IANA requests. 13.1. Acknowledgments We acknowledge the authors of draft-ietf-detnet-dp-sol-01. 14. References 14.1. Normative References [RFC2119] Bradner, S., "Key words for use in RFCs to Indicate Requirement Levels", BCP 14, RFC 2119, DOI 10.17487/RFC2119, March 1997, . [RFC4448] Martini, L., Ed., Rosen, E., El-Aawar, N., and G. Heron, "Encapsulation Methods for Transport of Ethernet over MPLS Networks", RFC 4448, DOI 10.17487/RFC4448, April 2006, . [RFC5085] Nadeau, T., Ed. and C. Pignataro, Ed., "Pseudowire Virtual Circuit Connectivity Verification (VCCV): A Control Channel for Pseudowires", RFC 5085, DOI 10.17487/RFC5085, December 2007, . 14.2. Informative References [I-D.bryant-rtgwg-enhanced-vpn] Bryant, S. and J. Dong, "Enhanced Virtual Private Networks (VPN+)", draft-bryant-rtgwg-enhanced-vpn-01 (work in progress), October 2017. [I-D.ietf-detnet-architecture] Finn, N., Thubert, P., Varga, B., and J. Farkas, "Deterministic Networking Architecture", draft-ietf- detnet-architecture-04 (work in progress), October 2017. [I-D.ietf-detnet-dp-alt] Korhonen, J., Farkas, J., Mirsky, G., Thubert, P., Zhuangyan, Z., and L. Berger, "DetNet Data Plane Protocol and Solution Alternatives", draft-ietf-detnet-dp-alt-00 (work in progress), October 2016. Bryant & Chen Expires September 6, 2018 [Page 23] Internet-Draft Deterministic MPLS March 2018 [I-D.ietf-detnet-dp-sol] Korhonen, J., Andersson, L., Jiang, Y., Finn, N., Varga, B., Farkas, J., Bernardos, C., Mizrahi, T., and L. Berger, "DetNet Data Plane Encapsulation", draft-ietf-detnet-dp- sol-01 (work in progress), January 2018. [I-D.ietf-spring-segment-routing-mpls] Bashandy, A., Filsfils, C., Previdi, S., Decraene, B., Litkowski, S., and R. Shakir, "Segment Routing with MPLS data plane", draft-ietf-spring-segment-routing-mpls-12 (work in progress), February 2018. [I-D.sdt-detnet-security] Mizrahi, T., Grossman, E., Hacker, A., Das, S., Dowdell, J., Austad, H., Stanton, K., and N. Finn, "Deterministic Networking (DetNet) Security Considerations", draft-sdt- detnet-security-01 (work in progress), July 2017. [RFC2205] Braden, R., Ed., Zhang, L., Berson, S., Herzog, S., and S. Jamin, "Resource ReSerVation Protocol (RSVP) -- Version 1 Functional Specification", RFC 2205, DOI 10.17487/RFC2205, September 1997, . [RFC2211] Wroclawski, J., "Specification of the Controlled-Load Network Element Service", RFC 2211, DOI 10.17487/RFC2211, September 1997, . [RFC2212] Shenker, S., Partridge, C., and R. Guerin, "Specification of Guaranteed Quality of Service", RFC 2212, DOI 10.17487/RFC2212, September 1997, . [RFC2474] Nichols, K., Blake, S., Baker, F., and D. Black, "Definition of the Differentiated Services Field (DS Field) in the IPv4 and IPv6 Headers", RFC 2474, DOI 10.17487/RFC2474, December 1998, . [RFC3032] Rosen, E., Tappan, D., Fedorkow, G., Rekhter, Y., Farinacci, D., Li, T., and A. Conta, "MPLS Label Stack Encoding", RFC 3032, DOI 10.17487/RFC3032, January 2001, . [RFC3209] Awduche, D., Berger, L., Gan, D., Li, T., Srinivasan, V., and G. Swallow, "RSVP-TE: Extensions to RSVP for LSP Tunnels", RFC 3209, DOI 10.17487/RFC3209, December 2001, . Bryant & Chen Expires September 6, 2018 [Page 24] Internet-Draft Deterministic MPLS March 2018 [RFC3270] Le Faucheur, F., Wu, L., Davie, B., Davari, S., Vaananen, P., Krishnan, R., Cheval, P., and J. Heinanen, "Multi- Protocol Label Switching (MPLS) Support of Differentiated Services", RFC 3270, DOI 10.17487/RFC3270, May 2002, . [RFC3443] Agarwal, P. and B. Akyol, "Time To Live (TTL) Processing in Multi-Protocol Label Switching (MPLS) Networks", RFC 3443, DOI 10.17487/RFC3443, January 2003, . [RFC3473] Berger, L., Ed., "Generalized Multi-Protocol Label Switching (GMPLS) Signaling Resource ReserVation Protocol- Traffic Engineering (RSVP-TE) Extensions", RFC 3473, DOI 10.17487/RFC3473, January 2003, . [RFC3985] Bryant, S., Ed. and P. Pate, Ed., "Pseudo Wire Emulation Edge-to-Edge (PWE3) Architecture", RFC 3985, DOI 10.17487/RFC3985, March 2005, . [RFC4206] Kompella, K. and Y. Rekhter, "Label Switched Paths (LSP) Hierarchy with Generalized Multi-Protocol Label Switching (GMPLS) Traffic Engineering (TE)", RFC 4206, DOI 10.17487/RFC4206, October 2005, . [RFC4385] Bryant, S., Swallow, G., Martini, L., and D. McPherson, "Pseudowire Emulation Edge-to-Edge (PWE3) Control Word for Use over an MPLS PSN", RFC 4385, DOI 10.17487/RFC4385, February 2006, . [RFC5129] Davie, B., Briscoe, B., and J. Tay, "Explicit Congestion Marking in MPLS", RFC 5129, DOI 10.17487/RFC5129, January 2008, . [RFC5462] Andersson, L. and R. Asati, "Multiprotocol Label Switching (MPLS) Label Stack Entry: "EXP" Field Renamed to "Traffic Class" Field", RFC 5462, DOI 10.17487/RFC5462, February 2009, . [RFC5654] Niven-Jenkins, B., Ed., Brungard, D., Ed., Betts, M., Ed., Sprecher, N., and S. Ueno, "Requirements of an MPLS Transport Profile", RFC 5654, DOI 10.17487/RFC5654, September 2009, . Bryant & Chen Expires September 6, 2018 [Page 25] Internet-Draft Deterministic MPLS March 2018 [RFC5659] Bocci, M. and S. Bryant, "An Architecture for Multi- Segment Pseudowire Emulation Edge-to-Edge", RFC 5659, DOI 10.17487/RFC5659, October 2009, . [RFC5921] Bocci, M., Ed., Bryant, S., Ed., Frost, D., Ed., Levrau, L., and L. Berger, "A Framework for MPLS in Transport Networks", RFC 5921, DOI 10.17487/RFC5921, July 2010, . [RFC6003] Papadimitriou, D., "Ethernet Traffic Parameters", RFC 6003, DOI 10.17487/RFC6003, October 2010, . [RFC7551] Zhang, F., Ed., Jing, R., and R. Gandhi, Ed., "RSVP-TE Extensions for Associated Bidirectional Label Switched Paths (LSPs)", RFC 7551, DOI 10.17487/RFC7551, May 2015, . [RFC7950] Bjorklund, M., Ed., "The YANG 1.1 Data Modeling Language", RFC 7950, DOI 10.17487/RFC7950, August 2016, . [RFC8040] Bierman, A., Bjorklund, M., and K. Watsen, "RESTCONF Protocol", RFC 8040, DOI 10.17487/RFC8040, January 2017, . [RFC8077] Martini, L., Ed. and G. Heron, Ed., "Pseudowire Setup and Maintenance Using the Label Distribution Protocol (LDP)", STD 84, RFC 8077, DOI 10.17487/RFC8077, February 2017, . Authors' Addresses Stewart Bryant Huawei Technologies Email: stewart.bryant@gmail.com Mach Chen Huawei Technologies Email: mach.chen@huawei.com Bryant & Chen Expires September 6, 2018 [Page 26]