Internet Engineering Task Force A. Gray, Ed. Internet-Draft Charter Communications Intended status: Informational 5 August 2019 Expires: 6 February 2020 Sampled Traffic Streaming draft-gray-sampled-streaming-00 Abstract This document standardizes the means of requesting a sampled capture stream from a router, receiving details about the resulting data flow, and the structure of the data flow itself. This is specifically tailored to having various hardware ASICs be able to perform this operation as quickly as possible, by allowing communication of the specific bit formats of headers applied to the packet flow, in a way that enhances interoperability between sources and sinks. Historically, NetFlow and its ilk have been used for these use cases, however the growth in hardware forward speeds is far outpacing the growth in CPU speeds, and the CPU-heavy parts of NetFlow is resulting in a reduction of sampling rates that include all of the fields provided by NetFlow that require CPU lookups. Status of This Memo This Internet-Draft is submitted in full conformance with the provisions of BCP 78 and BCP 79. Internet-Drafts are working documents of the Internet Engineering Task Force (IETF). Note that other groups may also distribute working documents as Internet-Drafts. The list of current Internet- Drafts is at https://datatracker.ietf.org/drafts/current/. Internet-Drafts are draft documents valid for a maximum of six months and may be updated, replaced, or obsoleted by other documents at any time. It is inappropriate to use Internet-Drafts as reference material or to cite them other than as "work in progress." This Internet-Draft will expire on 6 February 2020. Copyright Notice Copyright (c) 2019 IETF Trust and the persons identified as the document authors. All rights reserved. This document is subject to BCP 78 and the IETF Trust's Legal Provisions Relating to IETF Documents (https://trustee.ietf.org/ Gray Expires 6 February 2020 [Page 1] Internet-Draft Sampled Traffic Streaming August 2019 license-info) in effect on the date of publication of this document. Please review these documents carefully, as they describe your rights and restrictions with respect to this document. Code Components extracted from this document must include Simplified BSD License text as described in Section 4.e of the Trust Legal Provisions and are provided without warranty as described in the Simplified BSD License. Table of Contents 1. Introduction . . . . . . . . . . . . . . . . . . . . . . . . 2 1.1. Requirements Language . . . . . . . . . . . . . . . . . . 2 1.2. Terminology . . . . . . . . . . . . . . . . . . . . . . . 2 2. Problem Statement / Operator Use Cases . . . . . . . . . . . 3 2.1. Use Case 1 : Traffic Analytics . . . . . . . . . . . . . 3 2.2. Use Case 2 : Network Behavior Verification . . . . . . . 3 2.3. Use Case 3 : Standardization . . . . . . . . . . . . . . 4 3. Stream Setup . . . . . . . . . . . . . . . . . . . . . . . . 4 4. Data Stream Format . . . . . . . . . . . . . . . . . . . . . 5 5. IANA Considerations . . . . . . . . . . . . . . . . . . . . . 7 6. Security Considerations . . . . . . . . . . . . . . . . . . . 8 7. References . . . . . . . . . . . . . . . . . . . . . . . . . 8 7.1. Normative References . . . . . . . . . . . . . . . . . . 8 7.2. Informative References . . . . . . . . . . . . . . . . . 8 Appendix A. Yang Model . . . . . . . . . . . . . . . . . . . . . 9 Author's Address . . . . . . . . . . . . . . . . . . . . . . . . 17 1. Introduction 1.1. Requirements Language The key words "MUST", "MUST NOT", "REQUIRED", "SHALL", "SHALL NOT", "SHOULD", "SHOULD NOT", "RECOMMENDED", "MAY", and "OPTIONAL" in this document are to be interpreted as described in RFC 2119 [RFC2119]. 1.2. Terminology The following terms are used within this document: Client: The device configuring the Replicator. Receiver: The device that will be receiving the packet stream. Replicator: The device doing the actual packet replication, as requested by a Client, and sending it to a Receiver. Point: The location inside the Replicator (generally a forwarding ASIC) that performs the actual packet replication. There may be multiple physical interfaces serviced by one Point, or one Gray Expires 6 February 2020 [Page 2] Internet-Draft Sampled Traffic Streaming August 2019 interface may be serviced by multiple Points, that may have different capabilities. 2. Problem Statement / Operator Use Cases This document is designed around the following use cases that operators have today, or can foresee coming on the near horizion. 2.1. Use Case 1 : Traffic Analytics At present, operators may use a mix of NetFlow, IPFIX, and inline traffic samplers spread throughout the network to gather data for analytics. With the next generation of hardware on the horizon, we have 400Gb/s interfaces starting to become available, with faster speeds already being talked about. This will require at least an augmentation of any inline traffic samplers, which are quite expensive. Additionally, the pace of growth in the data plane is outgrowing the pace of growth in the control plane. This is especially seen with relatively control plane or CPU heavy protocols like NetFlow, where present sampling rates are simply not going to be sustainable long-term, due to on-box control plane hardware limitations. Being able to capture a filtered, sampled collection of actual packets throughout the network is very valuable for understanding how the network is being used, to provide hard data to justify network topology augments or changes. This proposal addresses this use case by making the data replication as simple as possible for hardware fowarding ASICs to implement and send off the device. This allows offloading the heavy calculations to distributed infrastructure than can be scaled as needed using commodity hardware. 2.2. Use Case 2 : Network Behavior Verification This use case focuses on the potential abilioty to have the ASICs also include packets that would normally be dropped, along with a reason why. With bits denoting dropped due to QoS policies, buffer contention, ACLs, etc., this traffic could be captured (potentially at a sampling rate of 1:1, i.e. every packet) and sent off for off- box analysis to determine if this was expected, or to provide alerts that QoS policies may be having adverse effects on the network. Here, including the packet payload provides a lot of additional data for these platforms to effectively "second guess" these policies. This proposal addresses this use case by explicitly including a field for marking what an ASIC is going to do with a packet, and including the payload of the resulting sample. Gray Expires 6 February 2020 [Page 3] Internet-Draft Sampled Traffic Streaming August 2019 2.3. Use Case 3 : Standardization These problems are foreseen by various vendors, who are talking about different variants of these sorts of traffic flows. Standardizing the way these streams are formed and communicated between the Replicators, Clients, and Receivers in a fashion that allows vendors flexibility in what the ASIC has to do (by allowing communicating of an extremely dynamic header in a manner than control planes can manage) allows systems to be used between all compliant platforms. The alternative is having to build independent systems for each form of this sort of packet replication that may end up being developed, resulting in much higher costs. This proposal addresses this use case by allowing the packet header to contain any combination of information, in any position and of any length, to be dynamically determined and information about how to handle this data to be exchanged between the Replicator, Client, and Receiver. This allows ASICs to put on what information they can, in the most efficient format for them. 3. Stream Setup To begin, a Client utilizes NETCONF and the model listed in Appedix A. A Client must first request from the Replicator the available configurations via the 'points' branch, which provides the following information: * A name of the Point. * What interfaces this Point is servicing. * If filtering is available, and if so, what filters can be applied (against certain IP fields, against parts of the frame, etc.) * Minimum and maximum sampling rates. * Any current samplers already active off this Point. * Optionally, the maximum frame length the Point can replicate into the sample. * Optionally, the maximum offset into a frame the Point can inspect. * Optionally, the maximum number of samplers that this Point can accomodate. A Client MUST still check for success, as highly complex filters may reduce the amount of replication the Point can do from this maximum. Gray Expires 6 February 2020 [Page 4] Internet-Draft Sampled Traffic Streaming August 2019 The Client then can request one or more streams to be set up on the Replicator, using that information. These individual streams may be configured with filters and a sampling rate, with the replicated packets sent to the Receiver, wrapped in an outer UDP header. The available filters are provided by the Replicator in the response above - a common option will be for what interface specifically to filter traffic against. The source port of this UDP stream MUST be set to TBD1 by default - Replicators MAY offer an ability to change this default. This is to allow security filters to have a fixed value to key off of as an additional defense to prevent these streams from being inadvertantly sent to undesirable locations. The Client and Receiver MAY be separate devices. The mechanism of exchanging information between Client and Receiver about the setup process is outside the scope of this document. 4. Data Stream Format After the stream setup, the Client MUST re-query the stream configuration to get the stream-format data that the remote device has calculated. The Client MUST NOT assume that the stream-format data is consistent between one instance and any other performed (there may be different versions of ASICs, different capabilities, different versions of operating systems, or different filters may yield a different format), or that the Payload is always at the end (as it is the only variable-width field, it could appear at the beginning or in the middle, and sufficient data is provided by the other fields to extract the data correctly). This provides the Client with what information is provided at what location in the resulting packet. The Replicator MUST follow the expectation that is provided in these fields. As an example (but not a nominative listing of what this may be - the actual format can be any combination of fields, of any size, in any order), the data inside the resulting data stream after the UDP tunnel header may look like the following: Example layout: Gray Expires 6 February 2020 [Page 5] Internet-Draft Sampled Traffic Streaming August 2019 0 1 2 3 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ | Incoming Port | Timestamp | +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ |Act| Frame Length | Internal Data 1 | +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ | Payload | | ... | +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ Figure 1 This non-normative example would be listed in the modeled data for packet-format as per the following table: +-----------+------+----------------+-----------------------------+ | Name | Size | Type | Type-Data | +===========+======+================+=============================+ | Incoming | 8 | ingress-port | A listing of values that | | port | | | may be seen in this field, | | | | | mapped to interface-refs | | | | | from [RFC8343]. | +-----------+------+----------------+-----------------------------+ | Timestamp | 24 | timestamp-nsec | Two 32 bit numbers giving | | | | | when the "0" of this field | | | | | is based off of. | +-----------+------+----------------+-----------------------------+ | Action | 2 | action | A listing of values that | | | | | may be seen in this field, | | | | | mapped to action types | | | | | (accepted, dropped, etc.) | +-----------+------+----------------+-----------------------------+ | Frame | 17 | frame-length | Note that this denotes the | | Length | | | original frame length - the | | | | | payload field MAY NOT | | | | | include the entire payload. | +-----------+------+----------------+-----------------------------+ | Internal | 13 | padding | | | Data 1 | | | | +-----------+------+----------------+-----------------------------+ | Payload | 0 | frame-payload | | +-----------+------+----------------+-----------------------------+ Table 1: Example packet-format response To restate the prior note, the above is purely an example of what the format could be - the actual format used is negotiated between the Gray Expires 6 February 2020 [Page 6] Internet-Draft Sampled Traffic Streaming August 2019 Client and Replicator, and can have practically any layout, with any additional fields. To adequately addresses the use cases stated above, a Replicator SHOULD support as a minimum set of capabilities: * An action field that denotes a pass or drop (ideally with drop reason) * The payload of at least 128 octets * The original frame length * Sampling rates down to 1:1 (i.e. every packet is replicated) * Having different sampling sessions having different sampling rates (to allow a "general" session to be watching a broad selection of traffic, and more specific sessions targeting exact flows or situations) * At least two sessions per physical interface * Filtering on ingress port * Filtering on action A Client SHOULD take efforts to be notified when a change has occurred on the Replicator, and re-verify its replication configurations when such a change is detected. 5. IANA Considerations This document defines a new UDP port number, entitled "Sampled Streaming", and assigns a value of TBD1 from the Service Name and Transport Protocol Port Number Registry https://www.iana.org/assignments/service-names-port-numbers/service- names-port-numbers.xhtml: +------+-------------------+ | Tag | Description | +======+===================+ | TBD1 | Sampled Streaming | +------+-------------------+ Table 2 This document requests registration of a URI in the "IETF XML Gray Expires 6 February 2020 [Page 7] Internet-Draft Sampled Traffic Streaming August 2019 Registry" RFC 3688 [RFC3688]. Following the format in RFC 3688, the following registration is suggested: URI: urn:ietf:params:xml:ns:yang:ietf-sampled-streaming Registrant Contact: The IESG. XML: N/A, the requested URI is an XML namespace. This document registers a YANG module in the "YANG Module Names" registry RFC 6020 [RFC6020]: name: ietf-sampled-streaming namespace: urn:ietf:params:xml:ns:yang:ietf-sampled-streaming prefix: ss reference: This document 6. Security Considerations Vendors and deployments must take into consideration that this functionality allows a mirroring of traffic, with configurable destinations and filters. Similar functionality already exists in various remote packet mirroring systems, and similiar considerations should be taken. Filters utilizing the source port of TBD1 SHOULD be applied at the edges of a provider's network to provide an additional layer of security. 7. References 7.1. Normative References [RFC2119] Bradner, S., "Key words for use in RFCs to Indicate Requirement Levels", BCP 14, RFC 2119, DOI 10.17487/RFC2119, March 1997, . [RFC3688] Mealling, M., "The IETF XML Registry", BCP 81, RFC 3688, DOI 10.17487/RFC3688, January 2004, . [RFC6020] Bjorklund, M., Ed., "YANG - A Data Modeling Language for the Network Configuration Protocol (NETCONF)", RFC 6020, DOI 10.17487/RFC6020, October 2010, . [RFC8343] Bjorklund, M., "A YANG Data Model for Interface Management", RFC 8343, DOI 10.17487/RFC8343, March 2018, . 7.2. Informative References Gray Expires 6 February 2020 [Page 8] Internet-Draft Sampled Traffic Streaming August 2019 Appendix A. Yang Model module ietf-sampled-streaming { namespace "urn:ietf:params:xml:ns:yang:ietf-sampled-streaming"; prefix "ss"; import ietf-yang-types { prefix yang; } import ietf-interfaces { prefix if; } import ietf-inet-types { prefix inet; } organization "IETF Working Group"; contact "Editor: Andrew Gray "; description "This module contains a collection of YANG definitions for managing sampled streaming subscriptions. Copyright (c) 2019 IETF Trust and the persons identified as authors of the code. All rights reserved. Redistribution and use in source and binary forms, with or without modification, is permitted pursuant to, and subject to the license terms contained in, the Simplified BSD License set forth in Section 4.c of the IETF Trust's Legal Provisions Relating to IETF Documents (https://trustee.ietf.org/license-info). This version of this YANG module is part of RFC XXXX (https://www.rfc-editor.org/info/rfcXXXX); see the RFC itself for full legal notices. The key words 'MUST', 'MUST NOT', 'REQUIRED', 'SHALL', 'SHALL NOT', 'SHOULD', 'SHOULD NOT', 'RECOMMENDED', 'NOT RECOMMENDED', 'MAY', and 'OPTIONAL' in this document are to be interpreted as described in BCP 14 (RFC 2119) (RFC 8174) when, and only when, they appear in all capitals, as shown here."; revision 2019-06-25 { Gray Expires 6 February 2020 [Page 9] Internet-Draft Sampled Traffic Streaming August 2019 description "Initial version."; reference "draft-gray-sampled-streaming-00"; } typedef filter-type { type enumeration { enum interfaces { description "List of interfaces to filter against."; } enum action { description "Filter against a list of actions that the Point took (i.e., only consider packets that were actually forwarded)."; } enum ip-version { description "The version number in the IP header."; } enum ip-v4-srcip { description "The IPv4 header's source IPv4 address."; } enum ip-v4-dstip { description "The IPv4 header's destination IPv4 address."; } enum ip-v4-ttl { description "The IPv4 header's Time to Live."; } enum ip-v4-prot { description "The IPv4 header's protocol number."; } enum ip-v6-srcip { description "The IPv6 header's source IPv4 address."; } enum ip-v6-dstip { description "The IPv6 header's destination IPv4 address."; } enum frame-size { description "The total size of the the frame."; } enum frame-payload { description "Specific payload octets."; } } description "The filtering abilities available."; } typedef field-type { type enumeration { Gray Expires 6 February 2020 [Page 10] Internet-Draft Sampled Traffic Streaming August 2019 enum padding { description "Padding bits that MUST be ignored."; } enum ingress-port { description "An indication of the incoming port the frame was received on."; } enum egress-port { description "An indication of the incoming port the frame was transmitted on."; } enum timestamp-msec { description "The timestamp the packet was received at, in integer milliseconds."; } enum timestamp-usec { description "The timestamp the packet was received at, in integer microseconds."; } enum timestamp-nsec { description "The timestamp the packet was received at, in integer nanoseconds."; } enum frame-length { description "The received frame length. Note that due to chipset capabilities, this MAY NOT be the same as the captured packet length."; } enum frame-payload { description "The payload of the frame."; } enum action { description "The action that was taken on this frame. Values are mapped as according to action-type."; } } description "Types of data included in the data stream provided back to the receiver. Note that all fields will not be provided."; } typedef action-type { type enumeration { enum forwarded { description "Forwarded normally through the system."; } enum dropped { description "Generically dropped. A more specific action type code should be used, if possible."; } enum dropped-rate-limit { description "Dropped due to a rate limiter applied."; } enum dropped-buffer { description "Dropped due to no buffer space."; } enum dropped-security { description "Dropped due to a security policy."; Gray Expires 6 February 2020 [Page 11] Internet-Draft Sampled Traffic Streaming August 2019 } enum dropped-error { description "Dropped due to the frame being in error."; } enum remarked { description "Frame was remarked and forwarded."; } } description "Possible actions taken on a packet."; } typedef destination-type { type enumeration { enum udp { description "Sent with a UDP header."; } } description "Different possible destination types."; } list points { description "A listing of the capture points available on this device, what ports they provide for, and what filtering is available at those points."; key name; leaf name { type string; description "The name of this capture point."; } list interface { description "List of interfaces that are available at this point."; config false; leaf if { mandatory true; type if:interface-ref; description "An interface tied to this capture point."; } } list filters { description "List of filtering options available at this point."; config false; leaf filter { description "One specific filter available at this point."; mandatory true; type filter-type; } } leaf min-ratio { Gray Expires 6 February 2020 [Page 12] Internet-Draft Sampled Traffic Streaming August 2019 type uint32 { range "1..max"; } description "The minimum sampling ratio (1:N, with N being this value) this point can provide."; } leaf max-ratio { type uint32 { range "1..max"; } description "The maximum sampling ratio (1:N, with N being this value) this point can provide."; } leaf max-samplers { type uint32; description "The maximum number of samplers that can be installed at this point."; } leaf max-filters { type uint32; description "The maximum number of filtering rules permitted at this location. Note this is an absolute maximum, and fewer rules that are complex may still be rejected by the device."; } leaf max-frame-length-copy { type uint16; description "The maximum size that the point can replicate and copy into the header."; } leaf max-frame-depth-inspect { type uint16; description "The offset of the last octet in a frame the ASIC can perform filtering against."; } list samplers { description "A list of all the samplers attached to this point."; config true; key "name"; leaf name { mandatory true; type string; description "A unique name given to this sampler."; } container destination { description "The destination of where to send the UDP stream to."; leaf type { mandatory true; type destination-type; description "The type of encoding for the destination."; } container udp-parameters { Gray Expires 6 February 2020 [Page 13] Internet-Draft Sampled Traffic Streaming August 2019 description "Parameters for destination-type=udp. Source port is always the port number assigned by IANA."; when "../type='udp'"; leaf destination-ip { mandatory true; type inet:ip-address-no-zone; description "The destination IP to send the stream to."; } leaf destination-port { mandatory true; type inet:port-number; description "The destination UDP port number to send the stream to."; } } } leaf ratio { mandatory true; type uint32 { range "1..max"; } description "The requested sampling ratio (1:N, with N being this value)."; } list filters { description "A list of filters applied to this sampling. Multiple filters are logically ANDed together."; key "name"; leaf name { description "A name for this filter."; type string; } list interfaces { when "../type = 'interfaces'"; description "Filter down to only this list of interfaces."; key "int"; leaf int { description "A specific interface to filter against."; type if:interface-ref; } } list actions { when "../type = 'action'"; description "Filter down to only this list of actions."; key "action"; leaf action { description "One specific action code."; Gray Expires 6 February 2020 [Page 14] Internet-Draft Sampled Traffic Streaming August 2019 type action-type; } } leaf type { description "The type of filter associated."; type filter-type; mandatory true; } leaf ipv4-address { when "../type = 'ip-v4-srcip' | ../type = 'ip-v4-dstip'"; type inet:ipv4-address-no-zone; description "The IPv4 address to filter on."; } leaf ipv6-address { when "../type = 'ip-v6-srcip' | ../type = 'ip-v6-dstip'"; type inet:ipv6-address-no-zone; description "The IPv6 address to filter on."; } leaf version { when "../type = 'ip-version'"; type inet:ip-version; description "The value of the IP version number to match on."; } container frame-payload { when "../type = 'frame-payload'"; description "Frame payload fragment to match on."; leaf offset { description "Offset in octets from the start of the frame to begin the match on."; type uint16; } leaf match { description "The bytes to match on."; type binary; } } leaf frame-length { when "../type = 'frame-length'"; type uint16; description "Frame length to match on."; } Gray Expires 6 February 2020 [Page 15] Internet-Draft Sampled Traffic Streaming August 2019 } container stream-format { config false; description "This contains the packet format data that this sampling stream is sending. This is only valid after configuration. The length fields are given in bits, and are consecutive. Needed gaps should use a 'padding' element."; list fields { description "The listing of the fields that will be encapsulated and sent to the receiver."; key "name"; leaf name { type string; description "Human readable name of what this field contains."; } leaf size { type uint32 { range "0..524280"; } description "The size of this field, in bits. The value of '0' denotes a variable-sized field, which is only permitted as the last field in this definition."; } leaf type { type field-type; description "The type of this data."; } list action-mappings { description "The mapping of values to action-type codes, valid for type=action."; key "value"; when "../type='action'"; leaf value { type binary; description "The value that will appear in the header."; } leaf meaning { type action-type; description "What this value indicates."; } } list port-mappings { description "The mapping of values to interfaces, valid for type=ingress-port or type=egress-port"; key "value"; when "../type='ingress-port' | ../type='egress-port'"; leaf value { type binary; description "The value that will appear in the header."; } leaf port { type if:interface-ref; Gray Expires 6 February 2020 [Page 16] Internet-Draft Sampled Traffic Streaming August 2019 description "The port the value maps to."; } } container timestamp { description "Supplemental data for type=timestamp*, in PTP Truncated Timestamp Format. Provides the time used as the epoch for the number in the data stream."; when "../type='timestamp-nsec' | ../type='timestamp-usec' | ../type='timestamp-msec'"; leaf seconds { type uint32; description "Specifies the integer portion of the number of seconds since the epoch."; } leaf nanoseconds { type uint32; description "Specifies the fractional portion of the number of seconds since the epoch, in integer number of nanoseconds."; } } } } } } } Author's Address Andrew Gray (editor) Charter Communications 8560 Upland Drive, Suite B Englewood, CO 80112 United States Phone: +1 720 699 5125 Email: Andrew.Gray@charter.com Gray Expires 6 February 2020 [Page 17]