Internet DRAFT - draft-heist-tsvwg-ecn-deployment-observations
draft-heist-tsvwg-ecn-deployment-observations
Transport Working Group P. Heist
Internet-Draft
Intended status: Informational J. Morton
Expires: 9 September 2021 8 March 2021
Explicit Congestion Notification (ECN) Deployment Observations
draft-heist-tsvwg-ecn-deployment-observations-02
Abstract
This note presents data gathered at an Internet Service Provider's
gateway on the observed deployment and usage of ECN. Relevant IP
counter and flow tracking data was collected and analyzed for TCP and
other protocols.
Status of This Memo
This Internet-Draft is submitted in full conformance with the
provisions of BCP 78 and BCP 79.
Internet-Drafts are working documents of the Internet Engineering
Task Force (IETF). Note that other groups may also distribute
working documents as Internet-Drafts. The list of current Internet-
Drafts is at https://datatracker.ietf.org/drafts/current/.
Internet-Drafts are draft documents valid for a maximum of six months
and may be updated, replaced, or obsoleted by other documents at any
time. It is inappropriate to use Internet-Drafts as reference
material or to cite them other than as "work in progress."
This Internet-Draft will expire on 9 September 2021.
Copyright Notice
Copyright (c) 2021 IETF Trust and the persons identified as the
document authors. All rights reserved.
This document is subject to BCP 78 and the IETF Trust's Legal
Provisions Relating to IETF Documents (https://trustee.ietf.org/
license-info) in effect on the date of publication of this document.
Please review these documents carefully, as they describe your rights
and restrictions with respect to this document. Code Components
extracted from this document must include Simplified BSD License text
as described in Section 4.e of the Trust Legal Provisions and are
provided without warranty as described in the Simplified BSD License.
Heist & Morton Expires 9 September 2021 [Page 1]
Internet-Draft ecndep March 2021
Table of Contents
1. Introduction . . . . . . . . . . . . . . . . . . . . . . . . 2
2. Collection Details . . . . . . . . . . . . . . . . . . . . . 3
3. Observations . . . . . . . . . . . . . . . . . . . . . . . . 3
3.1. ECN Endpoint Activity . . . . . . . . . . . . . . . . . . 4
3.1.1. Client Initiation . . . . . . . . . . . . . . . . . . 4
3.1.2. Server Acceptance . . . . . . . . . . . . . . . . . . 4
3.2. RFC3168 AQM Activity . . . . . . . . . . . . . . . . . . 4
3.2.1. Known AQM Activity . . . . . . . . . . . . . . . . . 5
3.2.2. Identifying Possible AQM Activity . . . . . . . . . . 5
3.2.3. AQM False Positives . . . . . . . . . . . . . . . . . 6
3.2.4. Limitations in AQM Identification . . . . . . . . . . 6
3.3. ECN Codepoints on Non-TCP Protocols . . . . . . . . . . . 6
3.3.1. Tunneled ECN Traffic . . . . . . . . . . . . . . . . 7
3.3.2. QUIC-ECN . . . . . . . . . . . . . . . . . . . . . . 8
3.3.3. Use of the ECN Field for Historical Reasons . . . . . 8
3.3.4. Use of the ECN Field Inadvertently . . . . . . . . . 8
3.3.5. Use of the ECN Field Maliciously . . . . . . . . . . 9
4. Study Limitations and Recommendations for Future Work . . . . 9
4.1. ECN Acceptance Rate . . . . . . . . . . . . . . . . . . . 9
4.2. Not-ECT Counts by IP . . . . . . . . . . . . . . . . . . 9
4.3. ECN Marked Packets on Not-ECN-Capable TCP Flows . . . . . 9
4.4. Tunnels . . . . . . . . . . . . . . . . . . . . . . . . . 10
4.5. Non-TCP Protocols . . . . . . . . . . . . . . . . . . . . 10
4.6. Other Protocols . . . . . . . . . . . . . . . . . . . . . 10
4.7. CWR Flag . . . . . . . . . . . . . . . . . . . . . . . . 10
4.8. NS Flag . . . . . . . . . . . . . . . . . . . . . . . . . 10
5. Abbreviated Output from ecn-stats . . . . . . . . . . . . . . 10
5.1. All IP . . . . . . . . . . . . . . . . . . . . . . . . . 11
5.2. TCP initiated from LAN to WAN . . . . . . . . . . . . . . 11
5.3. Non-TCP conntrack-supported protocols initiated from LAN to
WAN . . . . . . . . . . . . . . . . . . . . . . . . . . . 14
6. IANA Considerations . . . . . . . . . . . . . . . . . . . . . 28
7. Security Considerations . . . . . . . . . . . . . . . . . . . 28
8. Affiliation . . . . . . . . . . . . . . . . . . . . . . . . . 28
9. Acknowledgements . . . . . . . . . . . . . . . . . . . . . . 29
10. Informative References . . . . . . . . . . . . . . . . . . . 29
Authors' Addresses . . . . . . . . . . . . . . . . . . . . . . . 30
1. Introduction
To help guide the evolution of ECN, there is a need for more data on
current deployment status, and observed usage of the ECN related
bits, including:
* the initiation and acceptance of ECN capable TCP flows
Heist & Morton Expires 9 September 2021 [Page 2]
Internet-Draft ecndep March 2021
* marking via CE, and feedback for TCP via the ECE and CWR flags
* codepoints set on packets for protocols other than TCP
For several weeks, we gathered data on all traffic through an
Internet Service Provider's gateway. Though some of the results are
informative, we caution that a larger, more widely reviewed and
geographically distributed survey would be needed to be
authoritative.
2. Collection Details
From December 28, 2020 to January 20, 2021, data was gathered on all
traffic into and out of the Internet gateway at FreeNet Liberec, a
cooperative WISP in an urban area of the Czech Republic. A total of
122.5 TB of incoming data and 12 TB of outgoing data was seen.
Around 660 members belong to the ISP, and 861 member IP addresses on
the LAN were considered active during data collection. Most member
IPs are used by a household of users, while others are for individual
devices and public locations.
[IPTABLES-ECN] was used to collect and analyze the data. This
consists of a script to gather the data using iptables and ipsets in
Linux, and an analysis program that produces textual output. An
abbreviated version of the output is included in Section 5. See the
referred to source repository for more details and full output.
3. Observations
Our key observations are summarized as follows, and further expanded
upon in the following sections:
* 1.44% of TCP flows attempted to initiate ECN, across 390 member
IPs (45%).
* The acceptance rate for ECN flows was likely >50%.
* 382 member IPs (44%) successfully negotiated any TCP ECN flows,
determined by receipt of any ECN SYN/ACKs.
* 71 member IPs (18.6% of ECN negotiating) saw possible AQM marking
via CE and ECE, with 38 in one of two subnets with known AQMs, and
33 from other subnets, with unknown, possible AQMs.
Heist & Morton Expires 9 September 2021 [Page 3]
Internet-Draft ecndep March 2021
* The 33 IPs that saw possible AQM marking from unknown, possible
AQMs, were from a population of 319 ECN negotiating IPs not in
subnets with known AQMs, leading to a rough estimate of 10.3% of
random paths with AQMs present.
* Nonzero ECN codepoints were observed on 0.053% of non-TCP packets,
with possible attribution to tunneled ECN and/or misuse of the ECN
field.
3.1. ECN Endpoint Activity
3.1.1. Client Initiation
Of 319.5 million TCP SYNs from LAN to WAN, 1.44% indicated ECN
capability. Of 861 active member IP addresses, 390 (45.3%) attempted
initiation for at least one ECN flow. A large proportion of the ECN
flows are thought to come from Apple devices.
3.1.2. Server Acceptance
While 4.6 million ECN TCP SYNs were seen from LAN to WAN, 3.3 million
ECN SYN-ACKs were seen in return. While it's not possible to get an
exact ECN acceptance rate from this, it appears to be reasonably
high, likely due to default acceptance on prevailing server operating
systems like Linux, FreeBSD and recent versions of Windows Server.
3.2. RFC3168 AQM Activity
There appears to be some evidence of [RFC3168] marking AQMs. Of 861
active member IP addresses:
* 382 member IPs, or 44%, successfully negotiated any TCP ECN flows
- 63 of those (16.5%) were from subnets with known AQMs deployed
- 319 (83.5%) were from subnets without a known AQM
* 90 member IPs saw any CE or ECE marks on negotiated TCP ECN flows
* 71 member IPs (18.6% of ECN negotiating) saw possible AQM
activity, of which:
- 38 are in subnets with known AQMs (60.3% of 63 in total,
representing the detection rate)
- 33 are in subnets with unknown AQMs (10.3% of 319 in total,
representing the proportion of possible AQMs on random paths)
Heist & Morton Expires 9 September 2021 [Page 4]
Internet-Draft ecndep March 2021
Some factors may affect the 10.3% proportion of possible AQMs on
random paths:
* There may be false positives and negatives in possible AQM
identification.
* We have not attempted to correct for the likelihood of missed
AQMs, for example by assuming a detection rate similar to that for
the known AQMs.
The following subsections clarify what is known AQM activity, what is
considered possible activity, and some of the limitations in
identifying AQMs.
3.2.1. Known AQM Activity
Two backhaul links have fq_codel [RFC8290] deployed, serving the
10.45.64.0/24 and 10.45.235.0/24 subnets (also see Section 8). These
are the known AQMs that account for the 38/71 member IPs that saw
possible AQM activity. It is possible that additional AQMs exist
within the known subnets with AQM deployments, however those would be
harder to identify. The source of the AQM activity from the
remaining subnets is unknown, and could be from CPE, an external
network, or both.
3.2.2. Identifying Possible AQM Activity
For detection of [RFC3168] AQM activity, we observe TCP flows. The
ECE flag seen on negotiated TCP ECN flows allows for a more accurate
confirmation of marking, since CE marks applied after traversal
through the gateway will be missed.
To classify a member IP as having seen AQM activity, we use the
following criteria:
1. Nonzero ECT(0) counts in both directions
2. AND nonzero ECE in either direction
3. AND for nonzero CE and ECE, ECE:CE ratio >= 2:1, OR subtraction
of the ECE marks from the ECE marks in the opposite direction
allows the opposite direction to meet the same criteria, as
explained in the following section.
Heist & Morton Expires 9 September 2021 [Page 5]
Internet-Draft ecndep March 2021
3.2.3. AQM False Positives
Some of the member IP addresses showed small and nearly identical
counts of CE marks in one direction and ECE flags in both directions,
which seems unlikely to be the result of genuine congestion and AQM
activity. Our hypothesis is that these are part of a
"fingerprinting" routine used by port scanners when they find an open
port.
To exclude these, we looked for cases where inbound CE mark counts
were within a 2x factor of outbound ECE flags, and discounted these
as evidence of inbound AQM activity. In these cases we also applied
a more stringent check for outbound AQM activity, by subtracting the
outbound ECE flags from the inbound ECE flags, and looking for at
least a 2x ratio between any remainder and the number of any outbound
CE marks observed.
3.2.4. Limitations in AQM Identification
It is possible that some false-positives and/or false-negatives still
remain in our classification method. Additionally, there is no way
to know if we have identified all utilized paths with AQMs present.
Detection first requires an ECN flow (1.44% of TCP SYNs requested
ECN) and for that flow to encounter congestion at a bottleneck with
an AQM enabled.
3.3. ECN Codepoints on Non-TCP Protocols
Referring to the packet counts in the _All IP_ / _Both Directions_
table in the stats output in Section 5, where M indicates megapackets
and G, gigapackets:
+========+==========+================+=========+==========+
| | TCP (X) | Conntrack (XX) | Other | Total |
+========+==========+================+=========+==========+
| All | 76.60 G | -> | 43.52 G | 120.14 G |
+--------+----------+----------------+---------+----------+
| CE | 10031 | 3.38 M | 813951 | 4.20 M |
+--------+----------+----------------+---------+----------+
| ECT(0) | 523.91 M | 9.66 M | 2.55 M | 536.12 M |
+--------+----------+----------------+---------+----------+
| ECT(1) | 63 | 6.68 M | 182928 | 6.86 M |
+--------+----------+----------------+---------+----------+
Table 1
(X) TCP ECN packet counts only for negotiated ECN flows
Heist & Morton Expires 9 September 2021 [Page 6]
Internet-Draft ecndep March 2021
(XX) UDP, ICMP, DCCP, SCTP, GRE (Conntrack All packets included in
Other)
We note the following, where TCP-ECN refers to TCP flows that have
successfully negotiated ECN support:
* TCP-ECN accounted for 97.7% of the 536 million ECT(0) marks
* 0.68% of all TCP packets were from TCP-ECN marked with a nonzero
ECN codepoint
* About 0.05% of all non-TCP packets were marked with a nonzero ECN
codepoint
* Not-TCP-ECN accounted for 99.8% of the 4.2 million CE marks
* Not-TCP-ECN accounted for virtually all of the ECT(1) marks
* 6.4 of 6.68 million ECT(1) marks were to a single member IP
Possible explanations for ECN marks on non-TCP packets are explored
further in this section.
3.3.1. Tunneled ECN Traffic
There are several different encapsulation methods used when handling
the ECN field through tunnels, as per [RFC3168] and [RFC6040]:
1. copy the ECN field from the inner to the outer packet
2. reset the ECN field on the outer packet to ECT(0)
3. set Not-ECT on the outer packet
When method 3 is used at both ends of a tunnel, we would not expect
to see ECN codepoint usage in either direction.
When methods 1 or 2 are used at both ends of a tunnel, we would
expect to see ECT(0) on both incoming and outgoing packets. We would
also expect a bias towards incoming packets, since more data is
generally downloaded than uploaded, and pure ACKs do not have ECT(0)
marks.
When method 3 is used at only one end of the tunnel, we would expect
to see ECT(0) on packets in only one direction.
We note the following:
Heist & Morton Expires 9 September 2021 [Page 7]
Internet-Draft ecndep March 2021
* Bi-directional ECT(0) marks were observed for two member IP / port
pairs, on UDP port 443 and 60001.
* Uni-directional ECT(0) marks were observed for:
- UDP port 4500 (IPSec NAT traversal [RFC3948]) with 23 member IP
addresses downstream, and 1 member IP address upstream.
- UDP port 51820 [WIREGUARD] with 2 member IP addresses
downstream.
- Numerous UDP ports in other ranges, mostly on the downstream.
While it's possible that some of the data observed was from tunneled
ECN traffic, this can't be established definitively.
3.3.2. QUIC-ECN
Since a production implementation of QUIC-ECN is known to exist, we
could see some of that traffic in the non-TCP data. One member IP /
dstport pair to udp:443 saw 4603 ECT(0) marks from the WAN, and 1882
ECT(0) marks from the LAN. This may have been QUIC-ECN, although it
can't be established definitively.
3.3.3. Use of the ECN Field for Historical Reasons
Some applications may still use historical definitions of the former
TOS byte. Although RFC791 reserved the ECN field for future use, the
now obsolete [RFC1349] defined the TOS field as four bits within the
Type of Service octet, one of which overlaps with the ECN field.
This may account for some of the observed usage of ECT(0), since the
value for "minimize monetary cost" was 0001, shifted to the left one
bit, coinciding with ECT(0).
3.3.4. Use of the ECN Field Inadvertently
Users of operating system's socket APIs wishing to set a DiffServ
codepoint may be confused as to whether or not they need to shift the
desired value left two bits before passing it in. Additionally, OS
header files have been seen with out-of-date definitions for obsolete
values in the former Type of Service octet, and obsolete definitions
from [RFC2481].
Another possible source of confusion is the TOS field values listed
in the now obsolete [RFC1349], without having been shifted. A casual
reader could see the value 0001 for "minimize monetary cost" and
think that they should use this value in the TOS byte, conflicting
with ECT(1), not realizing that:
Heist & Morton Expires 9 September 2021 [Page 8]
Internet-Draft ecndep March 2021
* [RFC1349] is obsolete
* even if it weren't obsolete, the TOS values must be shifted to the
left *by one bit*
To reduce incorrect usages of the DS field, OS header files should be
sanitized, obsolete RFCs more prominently marked as such, and API
documentation brought up to date.
3.3.5. Use of the ECN Field Maliciously
It's possible that some software is using the ECN field to gain an
advantage in Internet queues or for some other nefarious purpose.
Further analysis would be needed to determine if this is the case.
4. Study Limitations and Recommendations for Future Work
The main limitation of this study is that we use counter data at
different levels of granularity. In some cases, it would be possible
to gain more certainty by taking packet captures and analyzing
individual flows, but this can be more difficult to do on production
data, both for technical and privacy reasons.
4.1. ECN Acceptance Rate
While we captured the ratio of ECN SYNs to ECN SYN-ACKs, we do not
have an exact count of flows that were accepted or rejected. It may
be possible to do this more accurately with additional iptables rules
in [IPTABLES-ECN]. Additionally, flows are tracked as ECN capable
upon receipt of an ECN SYN/ACK, regardless of whether that SYN/ACK
was associated with an outgoing ECN SYN. This could be improved with
more state tracking.
4.2. Not-ECT Counts by IP
While we captured the nonzero ECN codepoints by IP address,
separately for TCP and conntrack-supported protocols, we do not have
a count of Not-ECT by IP address. Although it may be possible to
obtain this, it would require an ipset lookup for every packet
traversing the gateway.
4.3. ECN Marked Packets on Not-ECN-Capable TCP Flows
While we captured counts of ECN marked packets on ECN capable TCP
flows, identified upon receipt of an ECN SYN/ACK, we do not have
separate counts of TCP packets that were marked with ECN codepoints
without having negotiated ECN. This should be possible to obtain
with additional iptables rules in [IPTABLES-ECN].
Heist & Morton Expires 9 September 2021 [Page 9]
Internet-Draft ecndep March 2021
4.4. Tunnels
Tunnel protocols are challenging because of the different
encapsulation methods and protocols used. An analysis at the flow
level, rather than by IP address and destination port pairs, might be
more useful in identifying the usage of ECN over tunnels.
4.5. Non-TCP Protocols
More research is needed into the reasons for ECN codepoints being set
on non-TCP traffic. Given the relatively low volume of this traffic,
it might be practical to take packet captures of it for further
analysis.
Additionally, we are currently not able to differentiate between the
total number of packets for conntrack-supported and Other protocols.
This could be improved with some changes to [IPTABLES-ECN].
4.6. Other Protocols
While this study looked at signals by IP address for TCP and IP/port
for conntrack-supported protocols, it does not break down signals for
Other protocols by IP address. Among those protocols is IPSec ESP
packets, using IP protocol 50. The [IPTABLES-ECN] script could be
modified to create more ipsets of type hash:ip, parallel to what was
done for IP traffic as a whole, to further analyze these protocols
for tunnel activity.
4.7. CWR Flag
Counting occurrences of TCP's CWR flag could better help distinguish
between some of the false positives and negatives in AQM detection.
4.8. NS Flag
Since [RFC8311] declared that the NS (Nonce Sum) flag is again
Reserved, after its now historical use by [RFC3540], we could collect
any observed usages of this flag, to confirm that it's available for
use in practice.
5. Abbreviated Output from ecn-stats
This abbreviated output only includes LAN to WAN flows, and a small
subset of the non-TCP conntrack protocols by member IP address. For
full output, see the [IPTABLES-ECN] repository.
*Note* the IP addresses shown here have been anonymized within the
10.0.0.0/8 address space, in a way that retains the subnet structure.
Heist & Morton Expires 9 September 2021 [Page 10]
Internet-Draft ecndep March 2021
5.1. All IP
Packets, CE, ECT(0) and ECT(1) are packet counts, and use
units of M, G or T for mega, giga, or terapackets.
Total (both directions):
TCP [*] Conntrack [+] Other Total
------- ------------- ----- -----
Bytes 101.22 TB -> 33.22 TB 134.46 TB
Packets 76.60 G -> 43.52 G 120.14 G
|-CE 10031 3.38 M 813951 4.20 M
|-ECT(0) 523.91 M 9.66 M 2.55 M 536.12 M
|-ECT(1) 63 6.68 M 182928 6.86 M
WAN to LAN:
TCP [*] Conntrack [+] Other Total
------- ------------- ----- -----
Bytes 95.79 TB -> 26.65 TB 122.45 TB
Packets 41.43 G -> 30.29 G 71.72 G
|-CE 9298 3.38 M 721002 4.11 M
|-ECT(0) 480.35 M 9.62 M 1.93 M 491.91 M
|-ECT(1) 62 6.68 M 65111 6.74 M
LAN to WAN:
TCP [*] Conntrack [+] Other Total
------- ------------- ----- -----
Bytes 5.43 TB -> 6.57 TB 12.00 TB
Packets 35.17 G -> 13.23 G 48.41 G
|-CE 733 60 92949 93742
|-ECT(0) 43.56 M 40366 614623 44.21 M
|-ECT(1) 1 28 117817 117846
[*] TCP ECN packet counts only for negotiated ECN flows
Counts for non-ECN TCP flows in Other
[+] Conntrack protocols: UDP, ICMP, DCCP, SCTP, GRE
Conntrack total Bytes and Packets included in Other
5.2. TCP initiated from LAN to WAN
SYN packet count totals for active IPs:
All SYNs: 319560652
ECN SYNs: 4601118 (1.44% of all)
ECN SYN/ACKs: 3273815 (71.15% of ECN SYNs)
Heist & Morton Expires 9 September 2021 [Page 11]
Internet-Draft ecndep March 2021
ECN packet count totals for active IPs:
Direction CE ECE ECT(0) ECT(1)
--------- -- --- ------ ------
From LAN 733 502985 42903861 1
From WAN 9298 19367 479756419 62
IP address counts with TCP and ECN activity:
Active (sent >= 10 SYNs): 861 (of 1195)
Initiated any ECN flows: 390 (45.3%)
Negotiated any ECN flows: 382 (44.4%)
Saw CE or ECE on ECN flow: 90 (23.6% of ECN negotiating)
Saw ECT(1) on ECN flow: 5
IP address counts with possible AQM activity:
Negotiated any ECN flows: 382
|- in subnet with known AQM: 63 (16.5% of ECN negotiating)
|- in subnet without known AQM: 319 (83.5% of ECN negotiating)
Criteria for possible AQM activity:
nonzero ECT(0) in both directions
AND nonzero ECE in either direction
AND ECE:CE ratio >= 2:1 OR opposite direction
ECE difference meets same criteria
IPs with possible AQM activity: 71 (18.6% of ECN negotiating)
|- from known AQMs: 38 (60.3% of 63 /w known AQM)
|- from unknown, possible AQMs: 33 (10.3% of 319 w/o known AQM)
ECN flow packet counts by active IP, for nonzero CE or ECE:
Flags column:
A: possible AQM activity (see Criteria above)
K: known AQM deployment
ECT(0) CE ECE ECT(0) CE ECE
from from from from from from
IP Flags WAN WAN LAN LAN LAN WAN
-- ----- --- --- --- --- --- ---
10.45.9.88 17970 0 0 0 0 431
10.45.64.3 AK 2909975 36 13348 245614 0 45
10.45.64.4 AK 228451 0 2192 60531 0 0
10.45.64.7 AK 1119810 28 4610 15970 0 35
10.45.64.11 AK 158206 0 335 52721 0 0
10.45.64.12 AK 466173 0 14955 28006 3 0
10.45.64.13 AK 287196 0 223 17277 0 0
Heist & Morton Expires 9 September 2021 [Page 12]
Internet-Draft ecndep March 2021
10.45.64.14 AK 2264292 13 20863 362708 0 23
10.45.64.15 AK 46467 0 9 33894 0 0
10.45.64.16 AK 12238 0 1396 3339 0 0
10.45.64.17 AK 1058702 0 464 10578 0 0
10.45.64.31 AK 1137817 0 46740 114016 12 0
10.45.64.39 AK 519279 0 11019 44094 0 0
10.45.64.45 AK 229911 0 363 297811 0 0
10.45.64.47 AK 1257622 0 15731 410234 321 6041
10.45.64.59 AK 186455 0 44 71189 0 0
10.45.64.85 AK 4958 0 57 1069 0 0
10.45.64.93 AK 604641 0 16530 89847 0 0
10.45.64.103 AK 660575 0 10649 129808 0 0
10.45.64.105 AK 407561 0 2046 28347 0 0
10.45.64.112 AK 2007755 0 1135 58168 1 1
10.45.64.116 AK 427696 0 1042 30081 0 0
10.45.64.118 AK 275449 163 710 94141 0 170
10.45.64.123 AK 461008 0 3118 29852 0 0
10.45.64.125 AK 540511 0 52960 57523 49 0
10.45.64.126 AK 686366 0 12579 57480 122 0
10.45.65.7 A 959608 0 176 94920 0 0
10.45.65.16 A 1392760 0 4483 157057 0 0
10.45.65.110 A 694373 0 1530 44914 0 0
10.45.65.112 A 1140897 0 2313 171205 0 0
10.45.65.124 A 73846 5 6 24228 0 9
10.45.86.39 A 823913 1 13 135427 0 0
10.45.86.41 A 6872888 72 3228 1116750 0 0
10.45.87.32 A 62967 0 64 33255 0 0
10.45.87.45 2052604 1 0 225774 0 0
10.45.87.50 2500669 3 3 134178 0 0
10.45.87.127 A 527732 17 22 58020 0 39
10.45.101.96 1883157 155 156 270789 0 151
10.45.104.24 A 1837837 55 63 416284 0 77
10.45.107.73 A 1851297 400 416 463753 0 430
10.45.108.24 A 230952 0 0 33749 0 36
10.45.113.6 10491356 168 191 253856 0 174
10.45.113.106 A 638565 34 37 173265 0 40
10.45.114.98 1370882 1619 1792 309940 0 1739
10.45.138.66 479880 43 56 95577 0 47
10.45.140.73 6036 510 551 1918 0 520
10.45.140.74 5396418 39 46 336854 0 38
10.45.141.85 A 450018 39 50 157585 0 85
10.45.145.2 A 118843 10 15 19309 0 25
10.45.145.73 3484464 1 0 153147 0 0
10.45.153.10 714256 6 11 80712 0 0
10.45.154.82 A 2355299 22 25 263636 0 44
10.45.155.68 1849086 1 1 349722 0 0
10.45.155.71 A 9089268 144 143 660457 1 152
10.45.158.197 17618743 493 53 369921 0 0
Heist & Morton Expires 9 September 2021 [Page 13]
Internet-Draft ecndep March 2021
10.45.158.198 A 1019414 13 13 222440 0 25
10.45.176.114 A 736022 32 46 88713 0 62
10.45.176.119 A 2393601 38 47 167329 0 68
10.45.177.68 A 17191899 22 24 201187 0 27
10.45.182.75 A 615134 6 7 100995 0 13
10.45.183.117 199726 131 145 52876 6 152
10.45.183.204 2729641 8 10 851838 0 0
10.45.212.82 A 2497321 18 23 213484 0 48
10.45.229.81 A 1043941 268 2104 36517 1 0
10.45.230.25 A 4560825 3132 18481 290819 0 0
10.45.230.204 28514121 1 1 321299 0 0
10.45.231.31 A 62246 16 9 12782 0 30
10.45.234.197 265034 188 225 98664 0 153
10.45.235.6 AK 140242 0 217 42778 0 0
10.45.235.24 AK 213822 0 388 50485 0 0
10.45.235.59 AK 1017759 16 897 121453 0 30
10.45.235.89 AK 8066090 56 31899 637830 176 5630
10.45.235.90 AK 3878916 727 4278 460048 0 709
10.45.235.92 AK 15410232 151 169965 1030037 41 1784
10.45.235.94 AK 344002 0 1394 3913608 0 0
10.45.235.196 AK 2234 0 157 1323 0 0
10.45.235.199 AK 2166 0 56 676 0 0
10.45.235.200 AK 5279 0 220 2311 0 0
10.45.235.203 AK 2966 0 234 1765 0 0
10.45.235.206 AK 154701 0 3484 333 0 0
10.45.235.208 AK 5076 0 378 3240 0 0
10.45.238.75 4916805 196 262 942367 0 229
10.45.241.101 A 2838055 0 740 303980 0 0
10.45.242.72 A 81526 5 5 36884 0 11
10.45.242.146 A 894737 21 25 85268 0 44
10.45.243.69 1946622 2 3 64838 0 0
10.45.249.6 A 621444 0 2461 75061 0 0
10.45.249.34 A 180747 0 2260 128409 0 0
10.45.251.37 A 455964 39 171 18996 0 0
10.45.251.114 A 14208298 134 13794 946610 0 0
5.3. Non-TCP conntrack-supported protocols initiated from LAN to WAN
Protocols included:
UDP, ICMP, DCCP, SCTP, GRE
Active IPs:
Active IPs with ECN signals: 420
Active IP/dstport pairs with ECN signals: 24972
ECN flow packet count totals for active IPs:
Heist & Morton Expires 9 September 2021 [Page 14]
Internet-Draft ecndep March 2021
Direction CE ECT(0) ECT(1)
--------- -- ------ ------
From LAN 59 26692 28
From WAN 2838929 9562002 6632561
ECN codepoint packet counts by client IP, with selected ports:
(ports with '*' had >100 ECT(0) marks)
ECT(0) CE ECT(1) ECT(0) CE ECT(1)
from from from from from from
IP/Port WAN WAN WAN LAN LAN LAN
------- --- --- --- --- --- ---
10.45.10.0 0 0 0 201 0 0
10.45.10.4 0 0 0 14 0 0
10.45.10.5 0 0 0 20 0 0
10.45.10.6 0 0 0 9 0 0
10.45.10.7 0 0 0 8 0 0
10.45.10.8 0 0 0 39 0 0
10.45.10.11 0 0 0 8 0 0
10.45.10.12 0 0 0 2 0 0
10.45.10.42 0 0 0 6 0 0
10.45.10.61 0 0 0 2 0 0
10.45.10.70 0 0 0 44 0 0
10.45.10.71 0 0 0 5 0 0
10.45.10.73 0 0 0 7 0 0
10.45.10.77 0 0 0 13 0 0
10.45.10.81 0 0 0 10 0 0
10.45.10.82 0 0 0 8 0 0
10.45.10.83 0 0 0 3 0 0
10.45.10.95 0 0 0 59 0 0
10.45.10.96 0 0 0 39 0 0
10.45.10.129 0 403 1 0 0 0
10.45.10.196 0 0 0 80 0 0
10.45.10.197 0 0 0 63 0 0
10.45.10.201 0 0 0 3 0 0
10.45.10.204 0 0 0 25 0 0
10.45.10.227 0 0 0 40 0 0
10.45.10.228 0 0 0 7 0 0
10.45.10.244 0 0 0 14 0 0
10.45.10.245 0 0 0 7 0 0
10.45.64.3 0 0 0 100 0 0
10.45.64.4 0 0 0 31 0 0
10.45.64.6 0 0 0 2 0 0
10.45.64.7 12 126 20 8 0 0
10.45.64.10 0 0 0 29 0 0
10.45.64.11 0 0 0 67 0 0
10.45.64.12 0 0 0 6 0 0
10.45.64.13 0 0 0 35 0 0
Heist & Morton Expires 9 September 2021 [Page 15]
Internet-Draft ecndep March 2021
10.45.64.14 0 0 0 121 0 0
10.45.64.15 0 0 0 52 0 0
10.45.64.16 0 0 0 18 0 0
10.45.64.19 16 0 0 0 0 0
udp:4500 (ipsec-na.. 11 0 0 0 0 0
10.45.64.31 34129 2468 58304 27 0 0
udp:37658 0 0 4346 0 0 0
* udp:38129 24957 2468 15281 0 0 0
udp:38884 0 0 10409 0 0 0
* udp:40871 288 0 2269 0 0 0
* udp:41621 3057 0 14609 0 0 0
* udp:41744 171 0 61 0 0 0
udp:43588 0 0 6746 0 0 0
udp:45444 0 0 1292 0 0 0
* udp:45465 866 0 0 0 0 0
udp:45483 0 0 1838 0 0 0
* udp:45522 4764 0 708 0 0 0
10.45.64.39 0 0 0 75 0 0
10.45.64.45 0 0 0 50 0 0
10.45.64.47 0 0 0 11 0 0
10.45.64.51 0 0 0 2 0 0
10.45.64.59 56 1624 10 593 0 0
udp:3478 (stun) 56 1624 10 0 0 0
10.45.64.85 0 0 0 4 0 0
10.45.64.86 7 434404 3 9 0 0
udp:4400 (ds-srv) 0 29065 0 0 0 0
udp:14757 0 97175 0 0 0 0
udp:24173 0 35437 0 0 0 0
udp:29493 0 120959 0 0 0 0
udp:44495 0 41547 0 0 0 0
udp:53678 0 109978 0 0 0 0
10.45.64.89 7 50 0 4 0 0
10.45.64.93 598 2971 341 75 0 0
* udp:3478 (stun) 598 2971 341 0 0 0
10.45.64.98 0 0 32780 0 0 0
udp:6008 0 0 9234 0 0 0
udp:7008 (afs3-upd.. 0 0 23546 0 0 0
10.45.64.99 132 2094 73 0 0 0
udp:3478 (stun) 0 3 0 0 0 0
10.45.64.103 0 0 0 47 0 0
10.45.64.104 70 293 31 1 0 0
10.45.64.105 213 33440 0 7 0 0
* udp:443 (https) 213 33440 0 0 0 0
10.45.64.107 0 0 0 2 0 0
10.45.64.108 0 0 0 1 0 0
10.45.64.111 1 1 0 0 0 0
10.45.64.112 0 421 0 48 0 0
10.45.64.116 4 143 8 64 0 8
Heist & Morton Expires 9 September 2021 [Page 16]
Internet-Draft ecndep March 2021
10.45.64.118 0 0 0 77 0 0
10.45.64.121 0 2107 0 0 0 0
udp:38603 0 2100 0 0 0 0
10.45.64.123 0 0 0 13 0 0
10.45.64.124 6 0 0 0 0 0
udp:443 (https) 6 0 0 0 0 0
10.45.64.125 0 0 0 22 0 0
10.45.64.126 1 10 0 37 0 0
10.45.65.0 0 0 0 42 0 0
10.45.65.1 0 0 0 45 0 0
10.45.65.5 0 0 0 17 0 0
10.45.65.7 0 0 0 30 0 0
10.45.65.11 0 0 0 6 0 0
10.45.65.16 1686 40141 36888 505 0 0
* udp:3478 (stun) 1595 22049 4 0 0 0
udp:26808 0 0 36805 0 0 0
udp:62348 0 15738 0 0 0 0
10.45.65.17 0 4 0 0 0 0
10.45.65.66 0 17 0 94 0 0
udp:3478 (stun) 0 17 0 0 0 0
10.45.65.94 319 0 1 25 0 0
udp:3478 (stun) 0 0 1 0 0 0
10.45.65.95 0 0 0 8 0 0
10.45.65.104 0 0 0 41 0 0
10.45.65.107 12 77 2 5 0 0
10.45.65.110 0 0 0 38 0 0
10.45.65.112 39 1168 18 75 0 0
10.45.65.122 2 5 0 0 0 0
10.45.65.123 0 0 0 1 0 0
10.45.65.124 0 0 0 11 0 0
10.45.65.127 0 0 0 5 0 0
10.45.75.90 0 0 0 1 0 0
10.45.80.28 2 8 1 0 0 0
10.45.80.79 4 7 0 2 0 0
10.45.80.85 0 0 0 10 0 0
10.45.80.99 0 0 0 11 0 0
10.45.83.76 0 0 0 3 0 0
10.45.83.80 28 51 11 0 0 0
10.45.85.127 301 174 30747 68 0 0
* udp:599 (acp) 222 174 45 0 0 0
udp:6008 0 0 30702 0 0 0
* udp:60001 49 0 0 65 0 0
10.45.86.16 13 0 0 2 0 0
udp:4500 (ipsec-na.. 8 0 0 0 0 0
udp:51820 (wiregua.. 5 0 0 0 0 0
10.45.86.36 0 0 0 4 0 0
10.45.86.39 205 37619 107 50 0 0
udp:29492 0 2512 0 0 0 0
Heist & Morton Expires 9 September 2021 [Page 17]
Internet-Draft ecndep March 2021
udp:64733 0 30711 0 0 0 0
10.45.86.40 2 0 0 0 0 0
udp:443 (https) 2 0 0 0 0 0
10.45.86.43 0 11 0 532 0 0
10.45.86.68 760 3528 614 325 0 0
udp:80 (http) 0 2 0 0 0 0
10.45.87.32 12 0 0 14 0 0
10.45.87.44 709 4963 623 0 0 0
udp:80 (http) 0 1 0 0 0 0
udp:6881 3 1313 43 0 0 0
10.45.87.45 0 0 0 185 0 0
10.45.87.48 0 0 0 82 0 0
10.45.87.50 3 0 9 68 0 0
udp:4500 (ipsec-na.. 3 0 9 0 0 0
10.45.87.103 0 0 0 2 0 0
10.45.87.112 0 1 0 0 0 0
10.45.87.113 0 0 0 33 0 0
10.45.87.127 0 0 0 44 0 0
10.45.92.74 31 0 1 2 0 0
10.45.93.69 15 122 6 0 0 0
10.45.93.75 361 2945 278 4 0 0
10.45.93.79 0 0 0 8 0 0
10.45.98.71 2 8 0 0 0 0
10.45.98.72 0 1 0 40 0 0
udp:3478 (stun) 0 1 0 0 0 0
10.45.101.96 0 0 0 140 0 0
10.45.101.100 0 0 0 12 0 0
10.45.101.101 2 10 7 0 0 0
10.45.101.103 21 21899 15 0 0 0
udp:58479 0 21372 0 0 0 0
10.45.101.104 0 0 10 33 0 0
10.45.104.24 0 0 0 324 0 0
10.45.104.104 16 72 2 60 0 0
10.45.107.73 32 0 1 58 0 0
udp:4500 (ipsec-na.. 32 0 1 0 0 0
10.45.107.79 34 0 0 70 0 0
udp:443 (https) 34 0 0 0 0 0
10.45.107.81 0 4421 0 3 0 0
udp:61094 0 4421 0 0 0 0
10.45.108.3 0 0 0 1 0 0
10.45.108.4 33 5079 90 1 0 0
udp:33027 0 2978 0 0 0 0
10.45.108.13 0 0 0 14 0 0
10.45.108.24 799 5543 1059 117 0 0
* udp:40211 107 0 0 0 0 0
10.45.108.25 1 2 1 799 0 0
10.45.108.66 0 0 0 0 0 1
10.45.108.69 0 0 0 2 0 0
Heist & Morton Expires 9 September 2021 [Page 18]
Internet-Draft ecndep March 2021
10.45.108.71 28 12830 0 0 0 0
udp:34665 0 12462 0 0 0 0
10.45.108.75 0 0 6395176 38 0 0
udp:6008 0 0 1755476 0 0 0
udp:7008 (afs3-upd.. 0 0 1827173 0 0 0
udp:8008 (http-alt) 0 0 740987 0 0 0
udp:9008 0 0 809024 0 0 0
udp:10008 (octopus) 0 0 380001 0 0 0
udp:11008 0 0 578400 0 0 0
udp:12008 (accurac.. 0 0 231619 0 0 0
udp:13008 0 0 72496 0 0 0
10.45.108.76 0 0 0 2 0 0
10.45.108.77 0 0 0 31 0 0
10.45.108.80 337 1566 173 10 0 0
10.45.108.95 1 5 0 283 0 0
10.45.108.126 0 0 0 12 0 0
10.45.112.74 9 95 4 371 0 0
10.45.112.102 0 0 0 29 0 0
10.45.112.139 0 0 0 5 0 0
10.45.112.154 0 0 0 4 0 0
10.45.112.165 0 0 0 24 0 0
10.45.112.172 6333 0 0 0 0 0
* udp:443 (https) 6333 0 0 0 0 0
10.45.112.216 0 0 0 2 0 0
10.45.113.6 147184 0 0 136 0 0
* udp:4500 (ipsec-na.. 147184 0 0 0 0 0
10.45.113.7 453 0 10 52 0 0
* udp:443 (https) 309 0 0 0 0 0
* udp:4500 (ipsec-na.. 144 0 10 0 0 0
10.45.113.9 0 0 0 60 0 0
10.45.113.11 0 0 0 187 0 0
10.45.113.27 0 0 0 1 0 0
10.45.113.30 0 0 0 4 0 0
10.45.113.33 0 0 0 2 0 0
10.45.113.34 0 0 0 58 0 0
10.45.113.35 0 0 0 6 0 0
10.45.113.36 0 0 0 2 0 0
10.45.113.66 1 11 0 0 0 0
10.45.113.90 0 0 0 163 0 0
10.45.113.94 17 62 2 0 0 0
10.45.113.97 0 0 0 19 0 0
10.45.113.99 11 76 12 15 0 0
10.45.113.104 818 0 0 0 0 0
* udp:4500 (ipsec-na.. 818 0 0 0 0 0
10.45.113.106 0 0 0 10 0 0
10.45.113.119 0 178 0 313 0 0
udp:3478 (stun) 0 178 0 0 0 0
10.45.113.122 36 0 0 0 0 0
Heist & Morton Expires 9 September 2021 [Page 19]
Internet-Draft ecndep March 2021
udp:4500 (ipsec-na.. 36 0 0 0 0 0
10.45.113.124 0 0 0 201 0 0
10.45.114.8 0 3 0 0 0 0
10.45.114.10 0 0 0 3 0 0
10.45.114.42 286 12 67 3 0 0
* udp:51820 (wiregua.. 286 0 66 0 0 0
10.45.114.98 0 0 0 10 0 0
10.45.120.25 0 0 0 53 0 0
10.45.120.34 0 0 0 12 0 0
10.45.120.78 0 0 0 715 0 0
10.45.122.51 686 28190 122 66 0 0
udp:45622 0 5782 0 0 0 0
udp:59437 0 17791 0 0 0 0
10.45.124.31 1720 5946 16897 105 0 0
udp:3478 (stun) 0 6 0 0 0 0
* udp:50451 1720 0 15875 0 0 0
udp:50919 0 2428 0 0 0 0
udp:50996 0 0 1016 0 0 0
udp:57403 0 1944 0 0 0 0
10.45.124.43 0 0 0 12 0 0
10.45.124.73 37 0 0 0 0 0
udp:4500 (ipsec-na.. 37 0 0 0 0 0
10.45.124.74 0 0 0 1 0 0
10.45.124.89 0 0 0 2 0 0
10.45.124.107 142 626895 83 0 0 0
udp:24616 0 501142 0 0 0 0
udp:51123 0 124060 0 0 0 0
10.45.124.111 0 1538 166 0 0 0
udp:4748 0 1491 166 0 0 0
10.45.124.117 0 0 0 248 0 0
10.45.125.97 0 0 0 2 0 0
10.45.125.99 130 6235 29 1 0 0
udp:8609 (canon-cp.. 0 3002 0 0 0 0
10.45.125.104 0 0 0 3 0 0
10.45.125.105 0 0 0 7 0 0
10.45.136.82 0 0 0 1 0 0
10.45.136.198 0 0 0 8 0 0
10.45.136.199 68 3210 7 0 0 0
udp:22312 0 2452 0 0 0 0
10.45.136.200 0 44 1 0 0 0
10.45.137.4 4603 0 0 1882 0 0
* udp:443 (https) 4603 0 0 1882 0 0
10.45.137.21 0 0 0 118 0 0
10.45.137.27 4 0 0 63 0 0
10.45.137.29 0 0 0 0 0 1
10.45.137.46 9 154 0 6 0 0
udp:443 (https) 9 0 0 0 0 0
10.45.137.53 0 0 0 7 0 0
Heist & Morton Expires 9 September 2021 [Page 20]
Internet-Draft ecndep March 2021
10.45.137.55 0 0 1 37 0 0
10.45.137.62 5 29 1 14 0 0
udp:443 (https) 2 0 0 0 0 0
10.45.137.119 16 203825 12 4 0 0
udp:16772 0 55846 0 0 0 0
udp:25135 0 24694 0 0 0 0
udp:25476 0 66965 0 0 0 0
udp:51123 0 54265 0 0 0 0
udp:55430 0 1138 0 0 0 0
10.45.137.123 2 4190 1 1 0 0
udp:29363 0 3283 0 0 0 0
10.45.138.52 3093 18938 0 0 0 0
* udp:42420 3087 18871 0 0 0 0
10.45.138.66 0 0 0 249 0 0
10.45.138.88 43 107 10 0 0 0
10.45.138.95 0 0 0 20 0 0
10.45.140.0 0 0 0 84 0 0
10.45.140.5 0 0 0 2 0 0
10.45.140.28 0 0 0 1 0 0
10.45.140.74 0 0 0 12 0 0
10.45.140.81 0 0 0 26 0 0
10.45.140.100 143 465 37 0 0 0
10.45.140.103 0 0 0 16 0 0
10.45.140.104 0 0 0 4 0 0
10.45.140.109 0 0 0 2 0 0
10.45.140.118 0 0 0 27 0 0
10.45.140.121 0 7032 0 17 0 0
udp:49710 0 1160 0 0 0 0
udp:53984 0 2694 0 0 0 0
udp:58704 0 1597 0 0 0 0
10.45.140.122 0 3 0 0 0 0
10.45.140.123 0 0 4 0 0 0
10.45.140.127 0 0 0 15 0 0
10.45.140.133 0 0 0 0 1 0
10.45.140.169 0 0 0 59 0 0
10.45.140.171 0 0 0 14 0 0
10.45.141.2 91 0 0 12 0 0
udp:443 (https) 91 0 0 0 0 0
10.45.141.6 0 0 0 24 0 0
10.45.141.14 0 0 0 2 0 0
10.45.141.17 2 37 1 17 0 0
10.45.141.19 0 0 0 2 0 0
10.45.141.82 579 0 0 21 0 0
* udp:443 (https) 579 0 0 0 0 0
10.45.141.83 0 0 0 14 0 0
10.45.141.84 0 0 0 90 0 0
10.45.141.85 0 0 0 518 0 0
10.45.141.86 0 0 0 6 0 0
Heist & Morton Expires 9 September 2021 [Page 21]
Internet-Draft ecndep March 2021
10.45.141.87 0 0 0 2 0 0
10.45.141.103 0 0 0 57 0 0
10.45.141.106 7 190 3947 1079 0 0
udp:3478 (stun) 0 24 12 0 0 0
* udp:5001 (commplex.. 0 0 0 1072 0 0
udp:40208 0 0 3932 0 0 0
10.45.141.125 0 0 0 2 0 0
10.45.144.20 2 6 2 1 0 0
10.45.144.43 0 0 0 3 0 0
10.45.144.55 0 0 0 2 0 0
10.45.144.68 0 0 0 363 0 0
10.45.144.73 0 0 0 14 0 0
10.45.144.75 0 0 3 51 0 0
10.45.144.77 51 289 35 24 0 0
10.45.144.105 413 0 11 1 0 0
* udp:4500 (ipsec-na.. 413 0 11 0 0 0
10.45.144.139 1496 0 0 0 0 0
* udp:443 (https) 1496 0 0 0 0 0
10.45.144.197 0 0 0 102 0 0
10.45.145.2 0 0 0 15 0 0
10.45.145.26 0 0 0 44 0 0
10.45.145.39 2503039 0 0 11 0 0
udp:443 (https) 4 0 0 0 0 0
* udp:4500 (ipsec-na.. 2503035 0 0 0 0 0
10.45.145.56 0 0 0 3 0 0
10.45.145.72 0 0 0 32 0 0
10.45.145.75 3024 0 0 0 0 0
* udp:443 (https) 3024 0 0 0 0 0
10.45.145.81 8691 107114 8245 292 0 0
udp:80 (http) 0 2 0 0 0 0
* udp:6881 355 8092 672 0 0 0
udp:19517 0 1097 0 0 0 0
udp:22784 0 3441 0 0 0 0
* udp:25223 110 0 0 0 0 0
* udp:37526 139 0 0 0 0 0
* udp:40631 191 0 0 0 0 0
udp:40990 0 33415 0 0 0 0
udp:51820 (wiregua.. 0 3 0 0 0 0
10.45.145.96 0 0 0 7 0 0
10.45.145.98 0 0 0 3 0 0
10.45.145.107 0 9 0 0 0 0
10.45.145.109 0 0 0 9 35 0
10.45.145.115 0 0 0 11 0 0
10.45.146.66 52 88 7 26 0 0
10.45.146.195 0 0 0 2 0 0
10.45.146.200 1471 0 0 49 0 0
* udp:4500 (ipsec-na.. 1471 0 0 0 0 0
10.45.146.201 0 0 0 9 0 0
Heist & Morton Expires 9 September 2021 [Page 22]
Internet-Draft ecndep March 2021
10.45.153.10 0 0 0 33 0 0
10.45.153.194 2 86 2 0 0 0
10.45.154.6 0 0 0 9 0 0
10.45.154.81 0 0 0 4 0 0
10.45.154.82 0 0 0 140 0 0
10.45.154.100 0 0 0 14 0 0
10.45.154.105 0 0 0 17 0 0
10.45.154.112 0 0 0 5 0 0
10.45.154.113 1 88 2 3 0 0
10.45.154.115 0 0 0 224 0 0
10.45.155.12 0 0 0 11 0 0
10.45.155.67 0 0 0 1 0 0
10.45.155.68 0 0 0 237 0 0
10.45.155.69 0 0 0 1 0 0
10.45.155.71 0 0 0 246 0 0
10.45.155.73 0 0 0 72 0 0
10.45.155.74 0 1 0 0 0 0
udp:3478 (stun) 0 1 0 0 0 0
10.45.155.75 0 4 0 0 0 0
10.45.155.76 0 1 0 0 0 0
10.45.155.217 0 0 0 15 0 0
10.45.155.229 4 42 6 48 0 0
10.45.156.94 25 152 8 0 0 0
10.45.156.105 0 5362 0 19 0 0
udp:58796 0 5362 0 0 0 0
10.45.156.127 0 0 0 22 0 0
10.45.158.115 0 0 0 402 0 0
10.45.158.124 0 0 0 4 0 0
10.45.158.127 0 0 0 3 0 0
10.45.158.195 0 1630 3 25 0 0
udp:6881 0 1610 0 0 0 0
10.45.158.197 0 0 0 82 0 0
10.45.158.198 0 0 0 204 0 0
10.45.158.204 0 0 0 118 0 0
10.45.158.206 9 32 2 0 0 0
10.45.176.114 0 0 0 68 0 0
10.45.176.116 188 1702 191 1 0 0
10.45.176.117 0 0 0 35 0 0
10.45.176.119 9320 1028270 11302 218 0 0
udp:6881 0 91498 83 0 0 0
* udp:6900 322 0 0 0 0 0
udp:8999 (bctp) 0 405853 3 0 0 0
* udp:10556 741 0 0 0 0 0
udp:11778 0 311705 0 0 0 0
* udp:12111 274 0 0 0 0 0
udp:21606 0 5678 0 0 0 0
udp:23578 0 4281 0 0 0 0
udp:24488 0 2140 0 0 0 0
Heist & Morton Expires 9 September 2021 [Page 23]
Internet-Draft ecndep March 2021
udp:35849 0 2632 0 0 0 0
* udp:37758 212 721 0 0 0 0
udp:40954 0 27113 0 0 0 0
* udp:42012 380 26 101 0 0 0
udp:48235 0 3182 0 0 0 0
* udp:50321 2066 14226 5982 0 0 0
* udp:50838 389 0 0 0 0 0
udp:50884 0 0 2743 0 0 0
udp:51413 39 1712 0 0 0 0
udp:54457 0 3504 0 0 0 0
udp:56769 0 23761 0 0 0 0
udp:59025 0 3034 0 0 0 0
* udp:60050 3000 3961 1478 0 0 0
udp:60062 0 13672 0 0 0 0
udp:64329 0 75590 0 0 0 0
10.45.176.120 0 0 0 73 21 18
10.45.176.206 37 689 3 34 0 0
udp:3478 (stun) 37 685 3 0 0 0
10.45.176.207 8 143 0 5 0 0
10.45.176.209 12 88 1 11 0 0
10.45.176.210 10 32 4 1 0 0
10.45.176.214 25 8900 0 18 0 0
udp:6672 (vision-s.. 23 8900 0 0 0 0
10.45.176.224 1 0 0 114 0 0
10.45.176.225 120 786 137 1 0 0
10.45.176.226 0 0 0 4 0 0
10.45.176.237 4 0 0 0 0 0
udp:443 (https) 4 0 0 0 0 0
10.45.177.66 9 213 8349 0 0 0
udp:6672 (vision-s.. 0 0 8334 0 0 0
10.45.177.68 12 64 8 124 0 0
10.45.177.75 0 2 0 66 0 0
10.45.177.197 0 2 1 0 0 0
10.45.182.75 44 71 17 25 0 0
10.45.182.85 41 2612 5024 0 0 0
udp:45864 0 0 4985 0 0 0
10.45.182.136 0 0 0 8 0 0
10.45.183.117 0 0 0 15 0 0
10.45.183.199 45 1579 0 8 0 0
udp:3478 (stun) 45 1578 0 0 0 0
10.45.183.204 0 9478 0 731 0 0
* udp:4500 (ipsec-na.. 0 0 0 237 0 0
udp:22885 0 9404 0 0 0 0
10.45.183.205 0 0 1 3 0 0
udp:4500 (ipsec-na.. 0 0 1 0 0 0
10.45.183.209 3 1 0 280 0 0
10.45.183.219 0 0 0 61 0 0
10.45.203.6 0 0 0 2 0 0
Heist & Morton Expires 9 September 2021 [Page 24]
Internet-Draft ecndep March 2021
10.45.212.17 10472 25127 16430 0 0 0
* udp:62503 10452 23528 16423 0 0 0
10.45.212.27 0 0 0 1 0 0
10.45.212.29 0 0 0 30 0 0
10.45.212.51 0 0 0 2 0 0
10.45.212.82 0 1 1 28 0 0
10.45.212.84 0 0 0 2 0 0
10.45.212.199 0 0 0 1 0 0
10.45.212.202 0 0 0 4 0 0
10.45.212.205 0 0 0 299 0 0
10.45.212.207 0 0 0 85 0 0
10.45.229.75 3 0 0 0 0 0
udp:443 (https) 3 0 0 0 0 0
10.45.229.78 6694314 0 0 113 0 0
* udp:4500 (ipsec-na.. 6694314 0 0 0 0 0
10.45.229.79 0 0 0 27 0 0
10.45.229.81 0 0 0 3 0 0
10.45.229.101 0 0 0 69 0 0
10.45.229.104 128 525 128 0 0 0
10.45.229.119 0 0 0 20 0 0
10.45.230.20 0 0 0 1 0 0
10.45.230.25 10 0 72 32 0 0
udp:4500 (ipsec-na.. 10 0 72 0 0 0
10.45.230.89 495 3537 296 4 0 0
10.45.230.99 7 0 5 2 0 0
udp:4500 (ipsec-na.. 7 0 5 0 0 0
10.45.230.204 9 57 18 110 0 0
10.45.230.207 18 33 1 1 0 0
10.45.230.212 0 0 0 2 0 0
10.45.230.223 0 0 0 3 0 0
10.45.230.224 27927 93 13 0 0 0
* udp:50323 322 0 0 0 0 0
* udp:50361 128 0 0 0 0 0
* udp:52065 409 0 0 0 0 0
* udp:55236 257 0 0 0 0 0
* udp:57072 142 0 0 0 0 0
* udp:58494 170 0 0 0 0 0
* udp:59465 160 0 0 0 0 0
* udp:59659 445 0 0 0 0 0
* udp:60874 129 0 0 0 0 0
* udp:60898 102 0 0 0 0 0
* udp:61122 302 0 0 0 0 0
* udp:61312 137 0 0 0 0 0
* udp:61669 124 0 0 0 0 0
* udp:62889 24738 0 0 0 0 0
* udp:63354 122 0 0 0 0 0
* udp:63474 107 0 0 0 0 0
10.45.230.226 0 0 0 3 0 0
Heist & Morton Expires 9 September 2021 [Page 25]
Internet-Draft ecndep March 2021
10.45.230.228 0 45 0 0 1 0
10.45.230.229 682 21 3 0 0 0
* udp:4500 (ipsec-na.. 682 0 0 0 0 0
10.45.231.16 433 0 0 24 0 0
* udp:4500 (ipsec-na.. 433 0 0 0 0 0
10.45.231.21 40 256 81 0 0 0
10.45.231.31 0 0 0 32 0 0
10.45.231.53 2 46 0 0 0 0
10.45.231.61 4151 0 1 13 0 0
* udp:4500 (ipsec-na.. 4151 0 1 0 0 0
10.45.231.80 0 0 0 6 0 0
10.45.231.99 0 0 0 40 0 0
10.45.231.102 0 0 0 11 0 0
10.45.231.114 0 0 0 47 0 0
10.45.233.16 0 0 0 55 0 0
10.45.233.39 1 13 2 1 0 0
10.45.233.41 0 0 3 4 0 0
10.45.233.42 0 0 0 115 0 0
10.45.233.47 0 0 0 1 0 0
10.45.233.55 0 0 0 3 0 0
10.45.234.197 320 0 11 2 0 0
* udp:4500 (ipsec-na.. 320 0 11 0 0 0
10.45.235.6 107 454 62 6 0 0
10.45.235.11 250 0 0 0 0 0
* udp:443 (https) 249 0 0 0 0 0
10.45.235.13 0 0 0 4 0 0
10.45.235.16 24 56 3 0 0 0
10.45.235.19 0 0 0 3 0 0
10.45.235.24 0 2 0 33 0 0
10.45.235.25 2310 28152 68 17 0 0
* udp:443 (https) 2214 0 0 0 0 0
udp:6881 0 13339 0 0 0 0
udp:31708 0 4595 0 0 0 0
udp:51413 0 5367 0 0 0 0
udp:52372 0 3975 0 0 0 0
10.45.235.49 672 3165 14 0 0 0
* udp:443 (https) 672 79 0 0 0 0
udp:59418 0 3078 0 0 0 0
10.45.235.52 23 0 0 0 0 0
udp:4500 (ipsec-na.. 23 0 0 0 0 0
10.45.235.59 0 0 0 58 0 0
10.45.235.66 0 0 0 4 0 0
10.45.235.89 165 2580 23 582 0 0
* udp:3478 (stun) 165 2580 23 0 0 0
10.45.235.90 0 0 0 332 0 0
10.45.235.92 0 0 0 1007 0 0
10.45.235.93 229 3272 306 13 0 0
10.45.235.94 0 0 0 10 0 0
Heist & Morton Expires 9 September 2021 [Page 26]
Internet-Draft ecndep March 2021
10.45.238.75 0 0 0 1744 0 0
10.45.238.104 0 2576 0 7 0 0
udp:443 (https) 0 2576 0 0 0 0
10.45.239.66 40 0 5 0 0 0
udp:4500 (ipsec-na.. 40 0 5 0 0 0
10.45.239.219 1 0 0 18 0 0
udp:443 (https) 1 0 0 0 0 0
10.45.240.86 0 0 0 5 0 0
10.45.241.57 216 66079 437 0 0 0
udp:4500 (ipsec-na.. 0 0 21 0 0 0
udp:33522 0 37844 0 0 0 0
udp:37859 0 27536 0 0 0 0
10.45.241.94 0 0 0 44 0 0
10.45.241.98 0 0 0 4 0 0
10.45.241.101 68946 10 2 120 0 0
* udp:4500 (ipsec-na.. 68942 0 0 0 0 0
10.45.241.121 0 0 0 2 0 0
10.45.242.72 0 0 0 4 0 0
10.45.242.81 0 0 0 14 1 0
10.45.242.144 0 0 0 5 0 0
10.45.242.146 0 0 0 30 0 0
10.45.242.161 143 134 2297 139 0 0
* udp:4500 (ipsec-na.. 115 0 4 0 0 0
udp:27032 0 78 2293 0 0 0
10.45.243.13 13877 63 1 0 0 0
* udp:20911 13853 0 0 0 0 0
10.45.243.41 12 0 0 14 0 0
udp:443 (https) 12 0 0 0 0 0
10.45.243.69 0 0 0 66 0 0
10.45.243.71 0 28 0 2 0 0
udp:80 (http) 0 28 0 0 0 0
10.45.243.109 0 2008 0 7 0 0
udp:41697 0 2002 0 0 0 0
10.45.248.33 2 8 0 10 0 0
udp:3478 (stun) 2 8 0 0 0 0
10.45.248.94 0 0 0 11 0 0
10.45.248.118 0 0 0 2 0 0
10.45.249.6 0 0 0 1502 0 0
10.45.249.34 25 0 0 154 0 0
udp:443 (https) 25 0 0 0 0 0
10.45.249.99 68 558 88 0 0 0
10.45.249.104 0 0 0 7 0 0
udp:4500 (ipsec-na.. 0 0 0 6 0 0
10.45.250.89 0 0 0 5 0 0
10.45.251.37 0 0 0 19 0 0
10.45.251.110 9 72 1 0 0 0
10.45.251.119 0 0 0 23 0 0
10.45.253.59 0 0 0 1 0 0
Heist & Morton Expires 9 September 2021 [Page 27]
Internet-Draft ecndep March 2021
udp:4500 (ipsec-na.. 0 0 0 1 0 0
10.45.253.61 0 0 0 53 0 0
10.45.253.84 121 0 0 16 0 0
* udp:443 (https) 121 0 0 0 0 0
10.45.253.93 0 0 0 4 0 0
10.45.253.100 0 0 0 142 0 0
10.45.253.121 0 0 0 2 0 0
10.45.254.94 0 0 0 12 0 0
10.45.255.90 1 125 0 0 0 0
10.45.255.97 0 0 0 36 0 0
ECN codepoint packet counts for selected ports:
ECT(0) CE ECT(1) ECT(0) CE ECT(1)
from from from from from from
Port WAN WAN WAN LAN LAN LAN
---- --- --- --- --- --- ---
icmp:port-unreachable 6632 40795 3539 404 0 0
icmp:host-unreachable 1171 2575 43 22990 0 0
icmp:ttl-zero-during.. 65 2 66 0 0 0
icmp:network-unreach.. 321 4 0 0 0 0
ipencap:0 0 0 0 1 0 0
udp:53 (domain) 0 403 1 0 0 0
udp:80 (http) 0 33 0 0 0 0
udp:443 (https) 20006 36095 0 1882 0 0
udp:599 (acp) 238 261 59 0 0 0
udp:1024-3457 [81] 100 618 9 34 59 28
udp:3478 (stun) 2498 31725 394 0 0 0
udp:3553-4492 [19] 1 29449 0 0 0 0
udp:4500 (ipsec-nat-t) 9422229 0 151 244 0 0
udp:4548-51819 [8177] 62692 2291117 6604184 1072 0 0
udp:51820 (wireguard) 291 3 66 0 0 0
udp:51821-65535 [9371] 45758 405849 24049 65 0 0
6. IANA Considerations
This document has no IANA actions.
7. Security Considerations
There are no known security considerations introduced by this note.
8. Affiliation
The author wrote the script used to deploy fq_codel to the two
backhaul subnets mentioned in Section 3.2.
Heist & Morton Expires 9 September 2021 [Page 28]
Internet-Draft ecndep March 2021
9. Acknowledgements
Thanks go to:
* Adam Pribyl, for gathering data at the FreeNet gateway
* Jonathan Morton and Rodney Grimes, for helping to analyze the
results
* FreeNet Liberec, for allowing access for data collection
10. Informative References
[IPTABLES-ECN]
Heist, P.G., "iptables-ecn GitHub Repository",
<https://github.com/heistp/iptables-ecn/>.
[RFC1349] Almquist, P., "Type of Service in the Internet Protocol
Suite", RFC 1349, DOI 10.17487/RFC1349, July 1992,
<https://www.rfc-editor.org/info/rfc1349>.
[RFC2481] Ramakrishnan, K. and S. Floyd, "A Proposal to add Explicit
Congestion Notification (ECN) to IP", RFC 2481,
DOI 10.17487/RFC2481, January 1999,
<https://www.rfc-editor.org/info/rfc2481>.
[RFC3168] Ramakrishnan, K., Floyd, S., and D. Black, "The Addition
of Explicit Congestion Notification (ECN) to IP",
RFC 3168, DOI 10.17487/RFC3168, September 2001,
<https://www.rfc-editor.org/info/rfc3168>.
[RFC3540] Spring, N., Wetherall, D., and D. Ely, "Robust Explicit
Congestion Notification (ECN) Signaling with Nonces",
RFC 3540, DOI 10.17487/RFC3540, June 2003,
<https://www.rfc-editor.org/info/rfc3540>.
[RFC3948] Huttunen, A., Swander, B., Volpe, V., DiBurro, L., and M.
Stenberg, "UDP Encapsulation of IPsec ESP Packets",
RFC 3948, DOI 10.17487/RFC3948, January 2005,
<https://www.rfc-editor.org/info/rfc3948>.
[RFC6040] Briscoe, B., "Tunnelling of Explicit Congestion
Notification", RFC 6040, DOI 10.17487/RFC6040, November
2010, <https://www.rfc-editor.org/info/rfc6040>.
Heist & Morton Expires 9 September 2021 [Page 29]
Internet-Draft ecndep March 2021
[RFC8290] Hoeiland-Joergensen, T., McKenney, P., Taht, D., Gettys,
J., and E. Dumazet, "The Flow Queue CoDel Packet Scheduler
and Active Queue Management Algorithm", RFC 8290,
DOI 10.17487/RFC8290, January 2018,
<https://www.rfc-editor.org/info/rfc8290>.
[RFC8311] Black, D., "Relaxing Restrictions on Explicit Congestion
Notification (ECN) Experimentation", RFC 8311,
DOI 10.17487/RFC8311, January 2018,
<https://www.rfc-editor.org/info/rfc8311>.
[WIREGUARD]
"WireGuard web site", <https://www.wireguard.com>.
Authors' Addresses
Peter G. Heist
463 11 Liberec 30
Czech Republic
Email: pete@heistp.net
Jonathan Morton
Kokkonranta 21
FI-31520 Pitkajarvi
Finland
Phone: +358 44 927 2377
Email: chromatix99@gmail.com
Heist & Morton Expires 9 September 2021 [Page 30]
