Transport Area Working Group J. Evans
Internet-Draft O. Pylypenko
Intended status: Informational Amazon
Expires: 23 November 2023 22 May 2023
Experience from implementing a new packet discard classification scheme
draft-evans-discardclass-00
Abstract
Router reported packet loss is the primary signal of when a network
is not doing its job. Some packet loss is normal or intended in TCP/
IP networks, however. To minimise network packet loss through
automated network operations we need clear and accurate signals of
all packets which are dropped and why. This document describes our
experience from implementing a packet loss classification scheme to
provide these signals and enable automated network mitigation of
unintended packet loss.
Status of This Memo
This Internet-Draft is submitted in full conformance with the
provisions of BCP 78 and BCP 79.
Internet-Drafts are working documents of the Internet Engineering
Task Force (IETF). Note that other groups may also distribute
working documents as Internet-Drafts. The list of current Internet-
Drafts is at https://datatracker.ietf.org/drafts/current/.
Internet-Drafts are draft documents valid for a maximum of six months
and may be updated, replaced, or obsoleted by other documents at any
time. It is inappropriate to use Internet-Drafts as reference
material or to cite them other than as "work in progress."
This Internet-Draft will expire on 23 November 2023.
Copyright Notice
Copyright (c) 2023 IETF Trust and the persons identified as the
document authors. All rights reserved.
Evans & Pylypenko Expires 23 November 2023 [Page 1]
Internet-Draft Exp. Implementing Packet Discard Class. May 2023
This document is subject to BCP 78 and the IETF Trust's Legal
Provisions Relating to IETF Documents (https://trustee.ietf.org/
license-info) in effect on the date of publication of this document.
Please review these documents carefully, as they describe your rights
and restrictions with respect to this document. Code Components
extracted from this document must include Revised BSD License text as
described in Section 4.e of the Trust Legal Provisions and are
provided without warranty as described in the Revised BSD License.
Table of Contents
1. Introduction . . . . . . . . . . . . . . . . . . . . . . . . 2
2. Problem statement . . . . . . . . . . . . . . . . . . . . . . 3
3. Classification Scheme . . . . . . . . . . . . . . . . . . . . 4
3.1. Discard Class Descriptions . . . . . . . . . . . . . . . 7
3.2. Discard Accounting Requirements . . . . . . . . . . . . . 8
3.3. Examples . . . . . . . . . . . . . . . . . . . . . . . . 9
4. Signal to Cause to Mitigation Mapping . . . . . . . . . . . . 9
5. Implementation Experience . . . . . . . . . . . . . . . . . . 9
6. Security Considerations . . . . . . . . . . . . . . . . . . . 11
7. IANA Considerations . . . . . . . . . . . . . . . . . . . . . 11
8. Terminology . . . . . . . . . . . . . . . . . . . . . . . . . 11
9. Acknowledgments . . . . . . . . . . . . . . . . . . . . . . . 11
10. References . . . . . . . . . . . . . . . . . . . . . . . . . 11
10.1. Normative References . . . . . . . . . . . . . . . . . . 11
10.2. Informative References . . . . . . . . . . . . . . . . . 11
Appendix A. Where do packets get dropped? . . . . . . . . . . . 12
Authors' Addresses . . . . . . . . . . . . . . . . . . . . . . . 12
1. Introduction
The job of a network is to transport packets. Understanding both
where and why packet loss occurs is essential to the effective
operation of networks. Router-reported packet loss is the most
direct signal for network operations to identify customer impact from
unintended packet loss. Accurate accounting of packet loss is not
enough, however, as some level of packet loss is normal in TCP/IP
networks. In automating network operations, there are only a
relatively small number of automated actions that can be taken to
mitigate impacting packet loss. Precise signal is important to
ensure the right action is taken, however, as taking the wrong action
can make problems worse.
The existing definitions of ifindiscards, ifoutdiscards, ifinerrors,
ifouterrors [RFC1213], do not provide the precision needed to be able
to identify the cause of packet loss sufficient to mitigate the
impact. From a network operators' perspective, ifindiscards can
represent both intended packet loss (i.e. packets discarded due to
Evans & Pylypenko Expires 23 November 2023 [Page 2]
Internet-Draft Exp. Implementing Packet Discard Class. May 2023
policy) and unintended packet loss. Further, these definitions are
ambiguous, in that vendors can and have implemented them differently
with different outcomes. In some implementations, ifinerrors
accounts only for errored packets which are dropped, whilst in others
it accounts for all errored packets whether they are dropped or not.
Many vendors support more discard metrics than these; where they do
they are inconsistently implemented due to an absence of clearly
defined classification and semantics for packet loss reporting.
This document describes our experience from implementing a packet
loss classification scheme across multiple hardware platforms, which
aimed to address these issues and enable automated mitigation of
unintended packet loss. Section 2 describes the problem. Section 3
defines the classification scheme and the accounting requirements
with examples. Section 4 gives examples of discard signal to cause
to auto-mitigation action mapping. Section 5 details our experience
from implementing this scheme.
2. Problem statement
Working backwards from auto-mitigation of unintended packet loss,
there are only a relative small number of potential auto-mitigation
actions, e.g.:
1. Take device, link or set of devices and/or links out of service
2. Put device, link or set of devices and/or links back into service
3. Move traffic
4. Roll-back a change
5. Escalate to network operators
Precise signal of impact is important as taking the wrong action can
be worse than taking no action. For example, taking a congested
device out of service can make congestion worse.
To be able to detect whether router reported packet loss is a problem
and determine what actions should be taken to mitigate the impact and
remediate the cause, depends on four primary features of the packet
loss signal:
1. the cause of the loss
2. the rate and magnitude of the loss
3. the duration of the loss
Evans & Pylypenko Expires 23 November 2023 [Page 3]
Internet-Draft Exp. Implementing Packet Discard Class. May 2023
4. the location of the loss
Features 2, 3 and 4 are already addressed with device level passive
monitoring statistics, e.g. obtained with SNMP [RFC1157] or NETCONF
[RFC6241]. Feature 1 is dependent on the classification scheme used
for packet loss reporting. We define a new scheme to address this
problem in the next section.
3. Classification Scheme
We define the classification scheme as a tree. At the top level we
differentiate the direction: ingress or egress. At the second level
we differentiate between traffic and discards. At the third level we
differentiate between intended discards and unintended discards by
discard class.
.
|-- interface/
| |-- ingress/
| | |-- traffic_rx/
| | | |-- v4/
| | | | |-- packets
| | | | `-- bytes
| | | |-- v6/
| | | | |-- packets
| | | | `-- bytes
| | | |-- l2/
| | | | |-- packets
| | | | `-- bytes
| | | `-- qos/
| | | |-- class_0/
| | | | |-- packets
| | | | `-- bytes
| | | |-- ...
| | | `-- class_n/
| | | |-- packets
| | | `-- bytes
| | `-- discards/
| | |-- v4/
| | | |-- packets
| | | `-- bytes
| | |-- v6/
| | | |-- packets
| | | `-- bytes
| | |-- l2/
| | | |-- packets
| | | `-- bytes
| | |-- policy/
Evans & Pylypenko Expires 23 November 2023 [Page 4]
Internet-Draft Exp. Implementing Packet Discard Class. May 2023
| | | |-- acl/
| | | | `-- packets
| | | |-- policer/
| | | | |-- packets
| | | | `-- bytes
| | | `-- null_route/
| | | `-- packets
| | |-- errors/
| | | |-- rx/
| | | | |-- l2/
| | | | | |-- frames
| | | | | |-- fec_error/
| | | | | | `-- frames
| | | | | `-- invalid_frame/
| | | | | `-- frames
| | | | `-- l3/
| | | | |-- packets
| | | | |-- checksum_error/
| | | | | `-- packets
| | | | `-- invalid_packet/
| | | | `-- packets
| | | |-- local/
| | | | |-- packets
| | | | `-- parity_error/
| | | | `-- packets
| | | `-- no_route/
| | | `-- packets
| | |-- ttl_expired/
| | | `-- packets
| | `-- no_buffer/
| | |-- class_0/
| | | |-- packets
| | | `-- bytes
| | |-- ...
| | `-- class_n/
| | |-- packets
| | `-- bytes
| `-- egress/
| |-- traffic_tx/
| | |-- v4/
| | | |-- packets
| | | `-- bytes
| | |-- v6/
| | | |-- packets
| | | `-- bytes
| | |-- l2/
| | | |-- packets
| | | `-- bytes
Evans & Pylypenko Expires 23 November 2023 [Page 5]
Internet-Draft Exp. Implementing Packet Discard Class. May 2023
| | `-- qos/
| | |-- class_0/
| | | |-- packets
| | | `-- bytes
| | |-- ...
| | `-- class_n/
| | |-- packets
| | `-- bytes
| `-- discards/
| |-- v4/
| | |-- packets
| | `-- bytes
| |-- v6/
| | |-- packets
| | `-- bytes
| |-- l2/
| | |-- packets
| | `-- bytes
| |-- policy/
| | |-- acl/
| | | `-- packets
| | `-- policer/
| | `-- packets
| |-- errors/
| | `-- tx/
| | |-- l2/
| | | `-- frames
| | `-- l3/
| | `-- packets
| `-- no_buffer/
| |-- class_0/
| | |-- packets
| | `-- bytes
| |-- ...
| `-- class_n/
| |-- packets
| `-- bytes
`-- to_cpu/
|-- packets
|-- bytes
`-- policy/
|-- acl/
| `-- packets
`-- policer/
`-- packets
Notes:
Evans & Pylypenko Expires 23 November 2023 [Page 6]
Internet-Draft Exp. Implementing Packet Discard Class. May 2023
1. The tree follows the structure <component><direction><type><sub-
type><sub-sub-type><metric>, where:
a. component can be interface|device|tunnel|to_cpu
b. direction can be ingress|egress
c. type can be traffic|discards
2. If Diffserv [RFC2475] quality of service (QOS) is not configured,
class0 is considered the default for no_buffer discards
3. from_cpu traffic is assumed to be accounted for as transit
traffic
See Appendix A for an example of where packets may be dropped in a
device.
3.1. Discard Class Descriptions
discards/policy/
These are intended discards, i.e. packets dropped due to a configured
policy. There are multiple sub-classes.
discards/policy/acl/
Discards due to packet matching an access control list (ACL)
discards/policy/policer/
Discards due to packet matching a configured policer
discards/policy/null_route/
Discards due to a packet matching a route with discard action
discards/no_route/
Discards due to a packet not matching any route
discards/no_buffer/
Discards due to no available buffer to enqueue the packet. These can
be tail-drop discards or due to an active queue management algorithm,
e.g. RED [RED93], CODEL [RFC8289]
discards/ttl_expired
There can also be multiple causes for TTL-exceed drops: i) trace-
route; ii) TTL set too low by the end system; iii) routing loops
discards/error/rx/l2/
Frames dropped because they are invalid at L2, e.g. due to bad CRC or
an invalid MAC address
Evans & Pylypenko Expires 23 November 2023 [Page 7]
Internet-Draft Exp. Implementing Packet Discard Class. May 2023
discards/error/rx/l3/
These are drops due to errors in the received packet, i.e. which
indicate an upstream problem, rather than a problem with the device
that is dropping the errored packets. There are multiple potential
errors that can cause a packet to be dropped on receipt: i) IP
checksum errors; ii) malformed frame/packets
discards/error/local/
A device may drop packets within its switching pipeline due to
internal errors, e.g. parity errors. Any discards not assigned to
the above classes are accounted here.
3.2. Discard Accounting Requirements
These requirements apply to the packets forwarded by the device, not
the packets destined to the device:
1. All packet receipt, transmission and drops MUST be reported
2. All packet receipt, transmission and drops SHOULD be attributed
to the corresponding physical or logical interface where they
occur.
3. An individual packet MUST NOT account against both the traffic
and discard classes on a single direction, i.e. ingress or egress
4. The aggregate v4, v6, and L2 traffic and discard classes MUST
account for all underlying packets received, transmitted and
dropped across all causal classes
5. The aggregate QOS traffic and discard (no buffer) classes MUST
account for all underlying packets received, transmitted and
dropped across v4, v6 and L2
6. In addition to these aggregate classes, an individual dropped
packet SHOULD only account against a single discard class
7. If there may be two drop causes for a packet, an individual
dropped packet SHOULD account against the first discard class in
order
Evans & Pylypenko Expires 23 November 2023 [Page 8]
Internet-Draft Exp. Implementing Packet Discard Class. May 2023
3.3. Examples
Assuming all the requirements are met, a good IPv4 packet received
would increment:
- interface/ingress/traffic/v4/rx/packets
- interface/ingress/traffic/v4/rx/bytes
- interface/ingress/traffic/diffserv/class_0/rx/packets
- interface/ingress/traffic/diffserv/class_0/rx/bytes
A received IPv6 packet dropped due to ttl-expired would increment:
- interface/ingress/discards/v6/packets
- interface/ingress/discards/ttl_expired/packets
An IPv4 packet dropped on egress due to no buffers would increment:
- interface/egress/discards/no_buffer/class_0/packets
- interface/egress/discards/no_buffer/class_0/bytes
4. Signal to Cause to Mitigation Mapping
Example discard signal to cause to mitigation mappings are shown in
the table below:
+------------------+---------+-------------------+-----------+--------+-----------+----------------------+
|Discard class |Direction|Cause |Rate |Duration|Unintended?|Possible actions |
+------------------+---------+-------------------+-----------+--------+-----------+----------------------+
|ErrorRxL2Discards |Ingress |Upstream device |>0(Anomaly)|O(1min) |Y |Take upstream link or |
| | |or link errror | | | |device out-of-service |
|TTLDiscards |Ingress |Tracert |<=Baseline | | |no action |
|TTLDiscards |Ingress |Convergence |>Baseline |O(1s) |Y |no action |
|TTLDiscards |Ingress |Routing loop |>Baseline |O(1min) |Y |Roll-back |
|AclDiscards |Ingress |ACL | | |N |no action |
|NullRouteDiscards |Ingress |Null route | | |N |no action |
|NoRouteDiscards |Ingress |Convergence |>0(Anomaly)|O(1s) |Y |no action |
|NoRouteDiscards |Ingress |Config error |>0(Anomaly)|O(1min) |Y |Roll-back |
|NoRouteDiscards |Ingress |Invalid destination|>0(Anomaly)|O(10min)|N |Escalate |
|ErrorLocalDiscards|Ingress |Device errors |>0(Anomaly)|O(1min) |Y |Take device |
| | | | | | |out-of-service |
|NoBufferDiscards |Egress |Congestion |<=Baseline | |N |no action |
|NoBufferDiscards |Egress |Congestion |>Baseline |O(1min) |Y |Bring capacity back |
| | | | | | |into service or move |
| | | | | | |traffic |
+------------------+---------+-------------------+-----------+--------+-----------+----------------------+
5. Implementation Experience
1. The number of classes is a compromise between: providing
sufficient detail to be able to take the appropriate actions
whilst: a) not providing too much detail which can require deeper
Evans & Pylypenko Expires 23 November 2023 [Page 9]
Internet-Draft Exp. Implementing Packet Discard Class. May 2023
understanding rather than helping to surface the problem quickly;
b) constraining the quantity of data produced where these metrics
are produced per interface.
2. There are multiple ways that we could have defined the discard
classification tree, e.g. we could have used a multi-rooted tree,
rooted in each protocol. We opted instead to define a tree where
protocol discards and cause discards are accounted orthogonally,
as this reduces the number of classes and we found it sufficient
to determine mitigation actions.
3. NoBuffer discards can be realised differently with different
memory architectures. Hence, whether a NoBuffer discard is
attributed to ingress or egress can differ accordingly. A packet
dropped due to NoBuffer discard should never be accounted for
both on ingress and on egress.
4. Most platforms account for the number of packets where the TTL
has expired, and the CPU has returned an ICMP Time Exceeded
message. In practise, however, there is often a policer applied
to traffic on the to_CPU path, which limits the number of packets
to the CPU. Implicitly, this limits the rate of TTL discards
processed by the CPU and hence it limits the number reported.
One method to account for all TTL discards, even those that are
dropped by a policer when being punted to the CPU, is to use
accounting of all ingress packets received with TTL=1, i.e.
before TTL processing.
5. Where a no_route discard is implemented with a default null
route, separate accounting is needed for any explicit null routes
configured, in order to differentiate between
interface/ingress/discards/policy/null_route/packets and
interface/ingress/discards/errors/no_route/packets.
6. It is useful to account separately for transit packets dropped by
transit ACLs/policers, and to_cpu packets dropped by ACLs/
policers which limit the number of packets to the CPU
7. It is not possible to identify a configuration error - i.e. when
intended discards are unintended - with device packet loss
metrics alone. For example, to determine if ACL drops are
intended or due to a misconfigured ACL some other method is
needed, e.g. with config validation before deployment or in
detecting a significant change in ACL drops before/after a
change.
Evans & Pylypenko Expires 23 November 2023 [Page 10]
Internet-Draft Exp. Implementing Packet Discard Class. May 2023
6. Security Considerations
There are no new security considerations introduced by this document.
7. IANA Considerations
There are no new IANA considerations introduced by this document.
8. Terminology
The key words "MUST", "MUST NOT", "REQUIRED", "SHALL", "SHALL NOT",
"SHOULD", "SHOULD NOT", "RECOMMENDED", "NOT RECOMMENDED", "MAY", and
"OPTIONAL" in this document are to be interpreted as described in
BCP 14 [RFC2119] [RFC8174] when, and only when, they appear in all
capitals, as shown here.
9. Acknowledgments
The content of this draft has benefitted from discussions with JR
Rivers, Ronan Waide and Chris DeBruin.
10. References
10.1. Normative References
[RFC2119] Bradner, S., "Key words for use in RFCs to Indicate
Requirement Levels", BCP 14, RFC 2119,
DOI 10.17487/RFC2119, March 1997,
<https://www.rfc-editor.org/rfc/rfc2119>.
[RFC8174] Leiba, B., "Ambiguity of Uppercase vs Lowercase in RFC
2119 Key Words", BCP 14, RFC 8174, DOI 10.17487/RFC8174,
May 2017, <https://www.rfc-editor.org/rfc/rfc8174>.
10.2. Informative References
[RED93] Jacobson, V., "Random Early Detection gateways for
Congestion Avoidance", n.d..
[RFC1157] Case, J., Fedor, M., Schoffstall, M., and J. Davin,
"Simple Network Management Protocol (SNMP)", RFC 1157,
DOI 10.17487/RFC1157, May 1990,
<https://www.rfc-editor.org/rfc/rfc1157>.
[RFC1213] McCloghrie, K. and M. Rose, "Management Information Base
for Network Management of TCP/IP-based internets: MIB-II",
STD 17, RFC 1213, DOI 10.17487/RFC1213, March 1991,
<https://www.rfc-editor.org/rfc/rfc1213>.
Evans & Pylypenko Expires 23 November 2023 [Page 11]
Internet-Draft Exp. Implementing Packet Discard Class. May 2023
[RFC2475] Blake, S., Black, D., Carlson, M., Davies, E., Wang, Z.,
and W. Weiss, "An Architecture for Differentiated
Services", RFC 2475, DOI 10.17487/RFC2475, December 1998,
<https://www.rfc-editor.org/rfc/rfc2475>.
[RFC6241] Enns, R., Ed., Bjorklund, M., Ed., Schoenwaelder, J., Ed.,
and A. Bierman, Ed., "Network Configuration Protocol
(NETCONF)", RFC 6241, DOI 10.17487/RFC6241, June 2011,
<https://www.rfc-editor.org/rfc/rfc6241>.
[RFC8289] Nichols, K., Jacobson, V., McGregor, A., Ed., and J.
Iyengar, Ed., "Controlled Delay Active Queue Management",
RFC 8289, DOI 10.17487/RFC8289, January 2018,
<https://www.rfc-editor.org/rfc/rfc8289>.
Appendix A. Where do packets get dropped?
The diagram below is an example of where and why packets may be
dropped in a typical single ASIC, shared buffered type device.
+----------+
| |
| CPU |
| |
+--+---^---+
from_cpu | | to_cpu
| |
+------------------------------v---+-------------------------------+
| |
+----------+ +----------+ +----------+ +----------+ +----------+ +----------+ +----------+
| | | | | | | | | | | | | |
-> Phy +--> Mac +--> Ingress +--> Buffers +--> Egresss +--> Mac +--> Phy |>
| | | | | Pipeline| | | | Pipeline| | | | |
+----------+ +----------+ +----------+ +----------+ +----------+ +----------+ +----------+
error/rx/l2 error/rx/l3 no_buffer error/tx/l3
error/local
no_route
ttl
policy/acl policy/acl
policy/policer policy/policer
null_route
Authors' Addresses
Evans & Pylypenko Expires 23 November 2023 [Page 12]
Internet-Draft Exp. Implementing Packet Discard Class. May 2023
John Evans
Amazon
Email: jevanamz@amazon.co.uk
Oleksandr Pylypenko
Amazon
Email: opyl@amazon.com
Evans & Pylypenko Expires 23 November 2023 [Page 13]