The ISP Column
A column on all things Internet
This is a followup article to Counting DNSSEC that reflects some further examination of the collected data. This time I'd like to describe some additional thoughts about the experiment, and some revised results in our efforts to count just how much DNSSEC is being used out there.
And for those looking for just the answers, here's a quick summary of the recount. It appears that as of September 2012 when this experiment was performed some 1.7% of the visible DNS resolvers in the Internet are performing DNSSEC validation, and some 1.6% of all end client systems are exclusively using DNSSEC validating resolvers.
And now the details…
Firstly, I should briefly recap on the experiment itself. We used an online advertisement delivery system as a means of enrolling end user systems to perform a simple DNSSEC capability experiment. Many online advertisement systems support dynamic content, and in this Adobe Flash was used with the code configured to perform the necessary dynamic support for the measurement exercise. We configured the ad to generate two unique URLs for the user's browser to fetch. The URLs are of the form:
The 's' and 'u' fields are dynamically generated, and the combination is unique for each user that is presented with an impression of the ad. This means that every client will generate a query for resolution of a unique DNS name, so that any caching of the outcome of the DNS query for one instance of this experiment will not be used by any other instance of this experiment, even if the users share the same DNS resolver. This form of dynamic DNS label generation also eliminates any URL object caching.
In this experiment we have used two subdomains, both of which are DNSSEC signed, and each zone consists of a single wildcard. The only difference between the two subdomains lies in the DNSSEC configuration. In the case of one subdomain (t5.dotnxdomain.net) the DS records are correctly configured in the dotnxdomain.net zone, while in the case of the other subdomain (t6.dotnxdomain.net) the DS records are deliberately altered. The intended consequence is that DNSSEC validation of domain names in t5.dotnxdomain.net will succeed, while DNSSEC validation in the other subdomain, t6.dotnxdomain.net, will fail.
The experiment script includes a 10 second timer. At the expiration of this timer the client will perform a GET where the parameters to the GET record the locally measured status of the fetches of the above two URLs. This result GET is directed to a URL whose DNS name part is not DNSSEC-signed.
The experiment was in operation for 17 days, from the 10th of September 2012 until the 27th of September 2012:
|Advertisement Placement Report:||4,965,129|
|DNS Query Log: Unique Identifiers:||3,816,822|
|Web Query Logs: Unique Identifiers:||2,831,780|
The difference between the advertisement placement report and the DNS query log indicates that some 23% of the ads were aborted before the control script had a chance to execute the first DNS query for the experiment. Some 985,042 experiments (or 26% of the DNS-active experiments) were started, but terminated before any web fetches were executed by the client (the difference between the DNS query log total and the Web query log total). In total some 57% of the placed advertisements produced results that were recorded in the web logs.
A further 1,244,299 experiments, or 44% of all the experiments that contacted the web server, did not complete the experiment and did not download the result URL after 10 seconds, or did not download either of the test URLs within the 10 second wait period before downloading the result URL. Both of these are attributable to the user terminating the advertisement before the download of the test URLs had completed.
This left 1,587,481 experiments that downloaded at least one of the test URLs and communicated their results back to the server on the expiration of the 10 second timer. The *.d.t5.dotnxdomain.net is called here "Valid", and the *.e.t6.dotnxdomain.net is termed "Invalid". The breakdown of the combination of fetches of these two objects is as follows:
|Web Queries:||Valid AND Invalid||1,438,291||90%|
|Valid and NOT Invalid||90,138||6%|
|NOT Valid and Invalid||59,052||4%|
There were 1,438,291 experiments that download both the DNSSEC-valid and invalid URLs, or 90% of the completed experiments. Some 90,138 experiments, or 6% of the total set of completed experiments that loaded the DNSSEC-Valid URL and did not download the DNSSEC-invalid URL. However, there were a further 59,052 experiments, or 4% of the total set, that behaved contrary to DNSSEC, downloading the DNSSEC-invalid URL, but not downloading the DNSSEC-valid URL.
This data points to some considerable level of variability in browser behavior. All these experiments reported back after 10 seconds as to their status, but it appears that some clients (at least 4%) are taking longer than 10 seconds to complete the 2 download tasks, and the experiment was terminated by the user prior to the completion of the experiment. If the number of incomplete experiments is equally distributed between the two cases of retrieving one but not the other URL, then it appears to indicate that some 2% of experiments did not download the DNSSEC-invalid URL because of the negative DNSSEC-validation outcome, while the other 4% did not perform the download because the experiment did not run to completion. However, this does not appear to be a very satisfactory form of analysis for the numbers, and the 2% outcome of DNSSEC-validating users appears to be a highly approximate calculation.
Is there a way to interpret the log files to provide a better estimate of DNSSEC use in the Internet?
A more methodical approach is to work through the DNS logs to see if we can assemble the set of DNSSEC-validating resolvers, then compare this proposed set to the web logs to see if the web logs contradict the tentative results from the DNS log analysis.
So the first question is: How can we tell if a DNS resolver is performing DNSSEC validation?
This is an example of the log from the local DNS authoritative name server when a DNSSEC-validating resolver generates queries for the experiment
15:50:27.130 queries: client 68.x.y.z#62436 (t10000.u1675001815.s1347893426.i767.v6022.d.t5.dotnxdomain.net): query: t10000.u1675001815.s1347893426.i767.v6022.d.t5.dotnxdomain.net IN A –ED 15:50:27.327 queries: client 68.x.y.z#45855 (t5.dotnxdomain.net): query: t5.dotnxdomain.net IN DS -ED 15:50:27.523 queries: client 68.x.y.z#45824 (t5.dotnxdomain.net): query: t5.dotnxdomain.net IN DNSKEY –ED 15:50:27.720 queries: client 68.x.y.z#47318 (dotnxdomain.net): query: dotnxdomain.net IN DNSKEY -ED
When the client performs the initial A query with the EDNS0 and DO bits set then the response will include the requested A RR, but will also include the RRSIG RRs, or the signature data, and also the relevant NSEC records and their signatures. The client will then attempt to validate these signatures, and to do so it needs the signing key values, or DNSKEY RR values for the domain. To validate this DNSKEY record it will need the DS RRs for the subdomain, obtained from the parent zone. To validate the RRSIG entries that were retrieved in the DS query it will then need the DNSKEY of the parent zone. This will, in turn, require a fetch of the DS RRs from its parent zone, and so on to the root zone.
The inference to be drawn from the logs of this instance of the test is that if a client is using a DNSSEC-validating DNS resolver, then we should see a DNSKEY query from the resolver.
However it is possible for resolvers to be "chained" so that a number of resolvers can lie behind a resolver (as shown in Figure 1). In such a case the "hidden" validating resolver will generate a DNSKEY query, which will be forwarded through a common Non-validating resolver as a distinct query. If we are using a single rule that a visible Validating Resolver generates DNSKEY queries then we will falsely assume that the common non-validating resolver is a DNSSEC-validating resolver. Is there a way to distinguish between these forms of resolver configurations?
The first approach to perform this differentiation of resolver types is to look at the time difference between the first query seen from a resolver and the first DNSKEY query. If the resolver that is querying the authoritative nameservers is itself a DNSSEC-validating resolver then a DNSKEY query will follow the A query "almost immediately", while the non-validating resolver will only pass on the DNSKEY query when a "hidden" DNSSEC-validating resolver performs DNSSEC validation.
The distribution of the elapsed time between the first query seen from each resolver and the DNSKEY query is shown in the following figure. The figure shows a pronounced peak at a delay of 200ms (the x axis uses a logarithmic scale in this figure), and there is a visible waning after 3 seconds.
This allows us to refine our assumptions of how to distinguish between a visible DNSSEC-validating resolver and a non-validating resolver to include the consideration that a DNSSEC validating resolver should query for a DNSKEY RR for the dotnxdomain.net domain within 3 seconds of the first query seen from this resolver.
In this experiment we saw 126,780 resolvers, and while 3,367 resolvers performed DNSKEY queries, only 2,277 performed this query within 3 seconds of the resolver's initial query. It seems reasonable to conclude that the remaining 1,090 visible resolvers are non-DNSSEC validating resolvers that may have DNSSEC-validating resolvers chained behind them in some manner.
There is a second filter we can use as well to apply to this candidate set of DNSSEC-validating resolvers. If we look at the each individual experiment (using the unique id generated for each experiment), the DNS logs will reveal which resolvers were used by the end host to resolve the experiment's domain names into IPv4 addresses. DNSSEC-validating resolvers do not return the requested resource record in the case of DNSSEC validation failure. Instead. they return a SERVFAIL error code.
At this stage we have a set of 2,277 visible resolvers whom we suspect are DNSSEC-validating resolvers. We also have a collection of 1,497,343 experiments where the invalid URL was retrieved by the client. If we filter these experiments, and select only those experiments where the client used a single DNS resolver, and further filter this set to retrieve only those experiments where this client loaded the invalid URL, then we have grounds to believe that this single resolver is not a DNSSEC-validating resolver. Some 154 visible DNS resolvers fall into this category.
We now have 2,123 visible DNS resolvers that appear to be performing DNSSEC validation out of a total of 126,780 visible DNS resolvers, or some 1.6% of all visible resolvers.
The initial estimate of 2,316 DNSSEC-validating resolvers out of a total pool of 57,267, or 4.0% of visible resolvers was somewhat optimistic. A revised estimate is somewhat lower, with 2,123 DNSSEC-validating resolvers seen, from a total pool of 126,780 visible resolvers.
What proportion of DNS resolvers are DNSSEC-capable?
2,123 out of 126,780, or 1.7% of the visible DNS resolvers were observed to perform DNSSEC validation
We can also look at the location of these DNS resolvers in terns of the country in which they are located. The Regional Internet Registries all regularly publish address allocation summary reports that include a mapping of IP address to country code. This allows us to map the IP address of the DNS resolver to a country where the address has been associated from the RIRs' reports. There are a large number of resolvers used by just 1 or 2 client systems, and a smaller number of resolvers used in some form of infrastructure mode where many clients use the same resolver. It appears reasonable to weight each resolver's DNSSEC validating capability by the number of unique clients seen who use that resolver, and use the DNSSC validating resolver weighted count as a percentage of the total weighted resolver draft for each country. From this data we can color a map of the world with the amount of DNSSEC-validating resolvers in each country, as show in Figure 3, below. (The data used to generate this map can be found at http://labs.apnic.net/dnssec/resolvers_by_cc_2.txt). The 10 countries with the highest levels of weighted DNSSEC resolvers are shown in Table 1. It should be noted that while the experiment covered some 750,000 individual experiments, the distribution of the clients who executed this test was not uniformly spread across all countries. The level of uncertainty in the per country data varies according to the number of tests that were performed by clients in each of these countries.
|15||24640||380||1320940||142958||10%||US||United States of America|
Table 1: Ranking of 15 Countries with the highest DNSSEC Resolver capability
(Countries with >= 1000 experiment ids)
What about the very largest of these DNS resolvers? The following table lists these largest resolvers and their ability to perform DNSSEC validation. Of the largest 26 individual resolvers we saw in this exercise just 1 set of these resolvers that undertook DNSSEC validation, located in AS7922, and operated by Comcast.
|no||976241||AS4766||KIXS-AS-KR Korea Telecom||Republic of Korea|
|no||472735||AS15169||GOOGLE - Google Inc.||USA|
|no||330663||AS3462||HINET Data Communication Business Group||Taiwan|
|no||294053||AS3786||LGDACOM LG DACOM Corporation||Republic of Korea|
|no||274418||AS5384||Emirates Telecommunications||United Arab Emirates|
|no||228905||AS4134||CHINANET-BACKBONE No.31,Jin-rong Street||China|
|no||194865||AS9318||HANARO-AS Hanaro Telecom Inc.||Republic of Korea|
|no||145429||AS4837||CHINA169-BACKBONE CNCGROUP China169||China|
|yes||140211||AS7922||Comcast Cable Communications||USA|
|no||120063||AS4788||TM Net, Internet Service Provider||Malaysia|
|no||113965||AS3356||LEVEL3 Level 3 Communications||US|
|no||107524||AS9050||RTD ROMTELECOM S.A||Romania|
|no||100527||AS45595||PKTELECOM-AS-PK Pakistan Telecom Company||Pakistan|
|no||87825||AS6799||OTENET-GR (Hellenic Telecommunications)||Greece|
|no||86182||AS7470||TRUEINTERNET-AS-AP TRUE INTERNET Co.,Ltd.||Thailand|
|no||85917||AS17676||GIGAINFRA Softbank BB Corp.||Japan|
|no||83349||AS4713||OCN NTT Communications Corporation||Japan|
|no||82146||AS8781||QA-ISP Qatar Telecom (Qtel) Q.S.C.||Qatar|
|no||78339||AS9737||TOTNET-TH-AS-AP TOT Public Company Limited||Thailand|
|no||75510||AS9299||Philippine Long Distance Telephone||Philippines|
|no||71499||AS15557||LDCOMNET Societe Francaise Radiotelephone||France|
|no||69071||AS45758||TRIPLETNET-AS-AP TripleT Interne||Thailand|
|no||58219||AS36692||OPENDNS - OpenDNS, LLC||USA|
Table 2: Ranking of 26 Largest DNS Resolvers by their DNSSEC Resolver capability
The next table shows the 20 largest DNSSEC-validating resolvers.
|yes||140211||AS7922||COMCAST-7922 - Comcast Cable Communication||USA|
|yes||11355||AS5466||EIRCOM Eircom Limited||Ireland|
|yes||9327||AS3301||TELIANET-SWEDEN TeliaSonera AB||Sweden|
|yes||9005||AS22047||VTR BANDA ANCHA S.A.||Chile|
|yes||7390||AS16276||OVH OVH Systems||France|
|yes||5313||AS28573||NET Servicos de Comunicao S.A.||Brazil|
|yes||3762||AS7657||VODAFONE-NZ-NGN-AS Vodafone NZ Ltd.||New Zealand|
|yes||3684||AS23700||BM-AS-ID PT. Broadband Multimedia, Tbk||Indonesia|
|yes||3448||AS15735||DATASTREAM-NET GO p.l.c.||Malta|
|yes||3411||AS2519||VECTANT VECTANT Ltd.||Japan|
|yes||3177||AS29562||KABELBW-ASN Kabel BW GmbH||Germany|
|yes||2927||AS4134||CHINANET-BACKBONE No.31,Jin-rong Street||China|
|yes||2180||AS28725||CZ-EUROTEL-AS AS of Eurotel Praha||Czech Republic|
|yes||1897||AS39651||COMHEM-SWEDEN Com Hem Sweden||Sweden|
|yes||1849||AS11992||CENTENNIAL-PR - Centennial de Puerto Rico||Puerto Rico|
|yes||1832||AS12912||ERA Polska Telefonia Cyfrowa S.A.||Poland|
|yes||1809||AS12301||INVITEL Invitel Tavkozlesi Zrt.||Hungary|
Table 3: Table 3 – Ranking of 20 Largest DNSSEC-Validating Resolvers
The full list of the resolvers' DNSSEC capability, per originating AS number can be found at http://labs.apnic.net/dnssec/resolvers_by_origin_as.txt. Of note is the diversity of countries in this list.
Let's now turn our attention from the resolvers to those clients who use these resolvers, and look at the clients and DNSSEC The web logs allow us to link the resolvers' DNSSEC capability to individual end host systems. This allows us to derive a measurement of the level of coverage of DNSSEC validation capability for end users. In the previous report we took the optimistic view that if any of the resolvers used by a client appeared to perform DNSSEC validation then we were prepared to list the client as performing DNSSEC validation. This resulted in a measurement of 69,560 out of 770,934 experiments, or 9.0% of the clients. We can improve this definition by taking a stricter view, namely what proportion of clients are seen to use only those resolvers that perform DNSSEC-validation. This is similar to the question of what proportion of clients will be unable to load a URL where the DNS label fails DNSSEC validation. As indicated at the start of this article the web logs indicate that some 6% of clients load the DNSSEC-valid URL but do not load the DNSSEC-invalid URL. But, also as noted above, the browser behavior introduces a significant level of uncertainty into these results, and the clients that appear to obey DNSSEC-validation outcomes use a mix of resolvers that both do and do not appear to perform DNSSEC validation.
Perhaps the question can be rephrased differently: What proportion of clients exclusively use DNSSEC-validating resolvers:
What proportion of users are using DNSSEC-validating DNS resolvers?
27,838 out of 1,717,906, or 1.6% of the end host systems were observed to perform DNSSEC validation.
The final query relates to the location of the users. for this experiment we used the mapping of IP address to country codes as published by the RIRs and were able to map users to countries.
Where are these users?
Of the 207 unique country codes that were seen in this experiment, some 105 countries contributed 1000 or more experiments. The 25 countries that contributed 1,000 or more experiments with the highest proportion of DNSSEC use is shown in the following table:
| % DNSSEC-|
|15.75%||9,149||58,074||United States of America|
Table 5: Ranking of 25 Countries with the highest DNSSEC client use
Once again is it possible to feed this data into a map of the world and paint each country with a color that denotes the level of coverage of DNSSEC. This is shown in Figure 4. (The data used to generate this map can be found at http://labs.apnic.net/dnssec/hosts_by_cc_2.txt)
Rather than by country it is also possible to generate the list of DNSSEC-using clients by originating AS. Using a filter of obtaining a minimum of 50 tested clients per originating AS, we obtain the following table of the 25 AS's that have the highest proportion of DNSSEC-using clients.
|AS Name, Country|
|1||AS44143||97.54%||119||122||VIPMOBILE-AS Vip mobile d.o.o., Serbia|
|2||AS27831||97.26%||71||73||Colombia Movil, Colombia|
|3||AS44034||97.03%||261||269||HI3G Hi3G Access AB, Sweden|
|4||AS28725||96.83%||61||63||CZ-EUROTEL-AS AS of Eurotel Praha, Czech Republic|
|5||AS15600||96.49%||55||57||FINECOM Finecom Telecommunications AG, Switzerland|
|6||AS20776||96.26%||180||187||OUTREMER-AS Outremer Telecom, France|
|7||AS12912||94.93%||712||750||ERA Polska Telefonia Cyfrowa S.A., Poland|
|8||AS31343||94.30%||248||263||INTERTELECOM Intertelecom Ltd, Ukraine|
|9||AS29518||91.87%||113||123||BREDBAND2 Bredband2 AB, Sweden|
|10||AS5466||90.86%||1631||1795||EIRCOM Eircom Limited, Ireland|
|11||AS38484||90.79%||69||76||VIRGIN-BROADBAND-AS-AP Virgin Broadband VISP, Australia|
|12||AS22047||88.06%||2066||2346||VTR BANDA ANCHA S.A., Chile|
|13||AS11992||87.83%||570||649||CENTENNIAL-PR - Centennial de Puerto Rico, Puerto Rico|
|14||AS3737||87.74%||93||106||PTD-AS - PenTeleData Inc., USA|
|15||AS17711||87.40%||111||127||NDHU-TW National Dong Hwa University, Taiwan|
|16||AS3301||86.25%||508||589||TELIANET-SWEDEN TeliaSonera AB, Sweden|
|17||AS3245||85.19%||46||54||DIGSYS-AS Digital Systems Ltd, Bulgaria|
|18||AS41833||83.78%||62||74||MOSCANET Moscanet (WISE), Lebanon|
|19||AS8473||82.26%||102||124||BAHNHOF Bahnhof Internet AB, Sweden|
|20||AS7922||80.43%||8855||11010||COMCAST-7922 - Comcast Cable Communications, Inc., USA|
|21||AS4704||80.27%||118||147||SANYO Information Technology Solutions Co., Ltd., Japan|
|22||AS5713||80.09%||744||929||SAIX-NET, South Africa|
|23||AS41749||80.00%||100||125||NETCOMPUTERS-AS Net & Computers SRL, Romania|
|24||AS24852||79.44%||85||107||VINITA VINITA Internet Services, Lithuania|
|25||AS1257||76.16%||409||5372||TELE2, European Union|
Table 5 – Ranking of 25 ASs with the highest DNSSEC client use
The complete set of data of DNSSEC use by hosts per originating AS can be found at http://labs.apnic.net/dnssec/hosts_by_as_2.txt.
Where are we with DNSSEC?
While the most optimistic estimate is that some 9% of clients use a collection of DNS resolvers where one or more of this resolver set appear to undertake DNSSEC validation, if we make the qualifying conditions a little more precise we get a different number.
The views expressed are the author’s and not those of APNIC, unless APNIC is specifically identified as the author of the communication. APNIC will not be legally responsible in contract, tort or otherwise for any statement made in this publication.
GEOFF HUSTON B.Sc., M.Sc., has been closely involved with the development of the Internet for many years, particularly within Australia, where he was responsible for the initial build of the Internet within the Australian academic and research sector. He is author of a number of Internet-related books, and has been active in the Internet Engineering Task Force for many years.