TOC 
Anti-Spam Research Group - IRTFC. Lewis
Internet-DraftNortel Networks
Intended status: BCPM. Sergeant
Expires: September 25, 2008MessageLabs, Inc
 March 24, 2008


Guidelines for Management of DNS Blacklists for Email
draft-irtf-asrg-bcp-blacklists-01

Status of this Memo

By submitting this Internet-Draft, each author represents that any applicable patent or other IPR claims of which he or she is aware have been or will be disclosed, and any of which he or she becomes aware will be disclosed, in accordance with Section 6 of BCP 79.

Internet-Drafts are working documents of the Internet Engineering Task Force (IETF), its areas, and its working groups. Note that other groups may also distribute working documents as Internet-Drafts.

Internet-Drafts are draft documents valid for a maximum of six months and may be updated, replaced, or obsoleted by other documents at any time. It is inappropriate to use Internet-Drafts as reference material or to cite them other than as “work in progress.”

The list of current Internet-Drafts can be accessed at http://www.ietf.org/ietf/1id-abstracts.txt.

The list of Internet-Draft Shadow Directories can be accessed at http://www.ietf.org/shadow.html.

This Internet-Draft will expire on September 25, 2008.

Abstract

The rise of spam and other anti-social behavior on the Internet has led to the creation of shared blacklists and whitelists of IP addresses or domains. This memo discusses guidelines for management of public DNS blacklists (DNSBLs).

The document will seek BCP status. Comments and discussion of this document should be addressed to the asrg@ietf.org mailing list.



Table of Contents

1.  Introduction
    1.1.  DNS-Based Reputation Systems
    1.2.  Guidance for DNSBL Users
    1.3.  Requirements Language
2.  Background
    2.1.  Transparency
        2.1.1.  Listing/Delisting Criteria SHOULD Be Easily Available
        2.1.2.  Audit Trail SHOULD be maintained
    2.2.  Listings and Removals
        2.2.1.  Listings SHOULD Be Temporary
        2.2.2.  A Direct Non-Public Way to Request Removal SHOULD Be Available
        2.2.3.  Removals SHOULD Be Prompt
        2.2.4.  SHOULD Have Similar Criteria for Listing and Delisting
        2.2.5.  Removals SHOULD Be Possible in Absence of the List Administrator
3.  Operational Issues
    3.1.  DNSBL Query Root Domain SHOULD be a Subdomain
    3.2.  DNSBLs SHOULD be Adequately Provisioned
    3.3.  DNSBLs SHOULD Provide Operational Flags
    3.4.  Shutdowns MUST Be Done in a Graceful Fashion
    3.5.  Listing of Special and Reserved IP Addresses MUST be disclosed
    3.6.  Use of Collateral Damage MUST Be Disclosed
    3.7.  Considerations for DNSBLs Listing Insecure Hosts
        3.7.1.  MUST NOT scan without provocation
        3.7.2.  Re-scan Periods SHOULD be Reasonable
4.  Security Considerations
5.  References
    5.1.  Normative References
    5.2.  Informative References
Appendix A.  Acknowledgements
§  Authors' Addresses
§  Intellectual Property and Copyright Statements




 TOC 

1.  Introduction



 TOC 

1.1.  DNS-Based Reputation Systems

Due to the rising amount of spam and other forms of network abuse on the Internet, many community members and companies began to create and maintain DNS-based reputation systems ("DNSBL") of IP addresses and domains identified as problem sources of email. These lists also have been known as blocklists and blacklists. These lists are used for filtering email. DNSBLs are either public or private. A public DNSBL makes its data available to any party seeking information about data on the list, but a private DNSBL is used solely by an organization for its own use and the data is not made available publicly. There are also commercial DNSBLs.

The first publicly available DNSBL using the Domain Name System (DNS) for distributing reputation data about email senders emerged in 1997, shortly after spam became a problem for network operators and email administrators. This pioneer DNSBL focused on identifying known spam sources situated at static IP addresses. Due to the broad adoption of this DNSBL, it had a devastating impact on static spam sources. Consequently, abusers found other methods for distributing their spam such as relaying messages through unsecured email servers or flawed formmail scripts on web pages. Additional DNSBLs were developed by others in order to address these changing tactics, and today more than 700 DNSBLs are in operation.

These DNSBLs vary widely in integrity, strategy, methodology and stability. While listing criteria can sometimes be quite controversial, this document deliberately does not discuss the rightness or wrongness of any criteria. We assert that DNSBL operators are free to choose whatever listing criteria they wish, as long as they're clear to their users what they are. It is the responsibility of the DNSBL user to ensure that the listing criteria and other aspects of a DNSBL meets their needs.

This document is also intended to provide guidance to DNSBL administrators so that they may be able to identify what features users would be interested in seeing as part of a high-quality, well- managed DNSBL, e.g., a clear listing and delisting policy to which the DNSBL administrator adheres strictly. This document is intended to be normative rather than prescriptive: it seeks to characterize the features of a well-managed DNSBL rather than setting out rules for how DNSBLs should be operated.

This document is not intended as a protocol specification of DNSBL queries. See [DNSBL‑EMAIL] (Levine, J., “DNS-based Blacklists and Whitelists for E-Mail,” November 2005.).



 TOC 

1.2.  Guidance for DNSBL Users

When choosing to adopt a DNSBL, an administrator should keep the following questions in mind:

  1. What is the intended use of the list?
  2. Does the list have a web site?
  3. Are the list's policies stated on the web site?
  4. Are the policies stated clearly and understandably?
  5. Does the web site function properly, e.g., hyperlinks?
  6. Are web pages for removal requirements accessible and working properly?
  7. How long has the list been in operation?
  8. What are the demographics and quantity of the list's user base? In other words, do other sites like my own use this DNSBL?
  9. Are comparative evaluations of the list available?
  10. What do your peers or members of the Internet community say about the list?

It is the responsibility of the system administrators who adopt one or more DNSBLs to evaluate, understand, and make a determination of which DNSBLs are appropriate for the sites they administer. If you are going to allow a third party to make blocking decisions for you, you MUST understand the policies and practices of those third parties because responsibility for blocking decisions remain ultimately with you, the system administrator. A DNSBL does not prevent anyone from receiving email or any other Internet service. A DNSBL *user* prevents service by means of a DNSBL.

DNSBL administrators are merely expressing an opinion through the publication of a DNSBL, and it is their absolute right to do so free of legal encumbrance, even in violation of this BCP. However, it is through abiding by the guidelines set forth in this BCP that the administrators of a DNSBL may gain the trust of their users.

These guidelines only address public DNSBLs and do not apply to private access DNSBLs.



 TOC 

1.3.  Requirements Language

The key words "MUST", "MUST NOT", "REQUIRED", "SHALL", "SHALL NOT", "SHOULD", "SHOULD NOT", "RECOMMENDED", "MAY", and "OPTIONAL" in this document are to be interpreted as described in RFC 2119 (Bradner, S., “Key words for use in RFCs to Indicate Requirement Levels,” March 1997.) [RFC2119].



 TOC 

2.  Background

The Anti Spam Research Group (ASRG) was chartered to address the spam problem. The ASRG charter includes:

"codification of best current practices in spam management"

This note falls within that category by listing guidelines for management of public blacklists. This document will seek BCP status.

NOTE:
This document is a product of the Anti-Spam Research Group (ASRG) of the IRTF. As per section 3 of RFC 2014 (Weinrib, A. and J. Postel, “IRTF Research Group Guidelines and Procedures,” October 1996.) [RFC2014]IRTF groups do not require consensus to publish documents. Therefore readers should be aware that this document does not necessarily represent the consensus of the entire ASRG.
NOTE:
This document is intended to evolve, based on comments from the Anti-Spam Research Group (ASRG) mailing list. It is certain that the current draft is incomplete and entirely possible that it is inaccurate. Hence, comments are eagerly sought, preferably in the form of suggested text changes, and preferably on the ASRG mailing list, at <asrg@ietf.org>.


 TOC 

2.1.  Transparency

A DNSBL SHOULD carefully describe the criteria which are the cause for adding, and the criteria for removing an IP address or domain name on the list. Such listing and delisting criteria SHOULD be presented in a clear and readable manner easily accessible to the public on the DNSBL's web site. A DNSBL MUST abide by its stated listing and delisting criteria. Entries that do not meet the published criteria MUST NOT be added to the DNSBL.

In other words, be direct and honest and clear about the listing criteria, and make certain that only entries meeting the published criteria are added to the list. For example, some DNSBL operators have been known to include spite listings in the lists they administer. There is nothing inherently wrong with this practice so long as it is clearly disclosed. For example, a DNSBL described as listing open relays only MUST NOT include IP addresses for any other reason. This transparency principle does not require DNSBL administrators to disclose the precise algorithms and data involved in a listing.

Availability of documentation concerning a DNSBL SHOULD NOT be dependent on the continued operation of DNS for DNSBL queries.

In other words, if the DNSBL documentation is at "http://dnsbl.example.com", the documentation for the web site should not become unavailable if the DNSBL query name servers are not available (or shutdown). See Section 3.1 (DNSBL Query Root Domain SHOULD be a Subdomain).



 TOC 

2.1.1.  Listing/Delisting Criteria SHOULD Be Easily Available

Listing and delisting criteria for DNSBLs SHOULD be easily available and SHOULD be located in a place clearly marked in its own section of the web site affiliated with the DNSBL.

DNSBLs often publish their listing criteria along with additional technical information about using the blacklist. This additional technical information can confuse end users, so a separate page or section on its own SHOULD be dedicated to detailing why a specific entry appears in the DNSBL.



 TOC 

2.1.2.  Audit Trail SHOULD be maintained

A DNSBL SHOULD maintain an audit trail for all listings and it is RECOMMENDED that it is made publicly available in an easy to find location, preferably on the DNSBL's web site. Please note that making audit trail data public does not entail revealing all information in the DNSBL administrator's possession relating to the listing; e.g., a DNSBL administrator MAY make the audit trail data selectively accessible in such a way as to not disclose information that might assist spammers, such as the contents of an email received by a DNSBL's spam trap.



 TOC 

2.2.  Listings and Removals



 TOC 

2.2.1.  Listings SHOULD Be Temporary

With the exception of DNSBLs that are based on data that does not change, such as those that list the IP addresses associated with a specific country or geographic region, all listings SHOULD be temporary so that an entry will time out at some point in the future. In most cases it is appropriate to expire in days or a few weeks, and 6 months is a sensible maximum (absent additional detections).

Note that all listings being temporary does not mean that some listings will not remain after the initial timeout period. If the DNSBL administrator determines that the conditions for listing still exists, then the timer for determining timeouts can be renewed.



 TOC 

2.2.2.  A Direct Non-Public Way to Request Removal SHOULD Be Available

Discussions about whether a DNSBL should remove an entry MAY include activity in a public forum. Methods for processing removal requests through private, direct exchanges, such as person-to-person email or a combination of web page requests and email responses, SHOULD be available. As a minimum, the DNSBL SHOULD have a web page that has a removal request function (separate from the page describing listing criteria as per Section 2.1.1 (Listing/Delisting Criteria SHOULD Be Easily Available)). The DNSBL SHOULD also make available an email address to handle issues other than blocking issues.

The DNSBL administrator MUST NOT use the list in question in such a way that removal requests would be blocked, and, moreover, SHOULD make unfiltered mailboxes available in order to allow affected users to submit their requests. In some cases it is impractical not to filter email to role accounts due to the amount of spam those mailboxes receive. If filtering should be necessary in such circumstances, filtering methods with virtually non- existent false positive rates SHOULD be chosen.



 TOC 

2.2.3.  Removals SHOULD Be Prompt

Requests for removal SHOULD be honored without question. Although this approach allows people to submit a request and have any listed IP address removed immediately, it does not prevent the DNSBL administrator from re-listing the IP address at a later time (for example: subject to seeing more spam, or re-checking the IP address security). A re-listing MAY result in a longer timeout until the listing expires before it is eligible for removal. Bounded exponentially extended periods is a good choice for listing timeout.

Assuming the above is not possible and no listing reasons remain, removal at anyone's request SHOULD be prompt. Removal requests SHOULD be acted upon within a period of 24 hours, and SHOULD be handled within three (3) days.

Most DNSBLs can effectively use a "no questions asked" removal policy because by their very nature they will redetect or relist problems almost immediately. They can mitigate more organized attempts to "game" the system by elementary checking and rate- limiting procedures, increasing lockout periods, rescans etc. Furthermore, a few IP addresses more or less do not make a significant difference in the overall effectiveness of a DNSBL. Moreover, a "no questions asked" removal policy provides the huge benefit of a swift reaction to incorrect listings.

As an example, one popular DNSBL uses a "no questions asked" removal policy, but does perform rate-limiting and malicious removal detection.

Another important consideration supporting a "no questions asked" self-removal policy is that it forestalls conflicts between DNSBL administrators and organizations whose IP addresses have been listed. Such a policy also can be an effective deterrent to legal problems.



 TOC 

2.2.4.  SHOULD Have Similar Criteria for Listing and Delisting

The criteria for being removed from a DNSBL SHOULD bear a reasonable relationship to the factors which were the cause of the addition to the DNSBL. If a listed entity fulfills all published requirements for removal from a DNSBL, then the DNSBL administrator SHOULD NOT impose any additional obstacles to remove a given entry from the DNSBL. There SHOULD NOT be any extra rules for de-listing other than the ones listed in the published listing criteria.

In addition, it is RECOMMENDED that all listings SHOULD be temporary as described in Section 2.2.1 (Listings SHOULD Be Temporary). For temporary listings, it is not necessary to correct the causes of the listing in order to be removed from the DNSBL.



 TOC 

2.2.5.  Removals SHOULD Be Possible in Absence of the List Administrator

If removals cannot be automated (e.g., via robot re-testing) then the DNSBL SHOULD have multiple administrators so that a removal request can be processed if the principal list administrator is on vacation or otherwise unavailable.



 TOC 

3.  Operational Issues



 TOC 

3.1.  DNSBL Query Root Domain SHOULD be a Subdomain

By virtue of using domain names, a DNSBL is a hierarchy with a root anchored in the global Internet. The DNSBL "query root" SHOULD be below the registered domain, so that the DNSBL information is not conflated with domain housekeeping information (e.g., name server or MX records) for the domain. By using this approach, DNSBL queries would take the form of "<query>.dnsbl.example.com" rather than "<query>.example.com". Further, this sub-tree should have its own name servers. Thus, the DNSBL query root has its own zone file containing the DNSBL information, and the registered domain has its own name servers containing the information (MX records etc.) for the domain. This approach facilitates clear delineation of function as well as orderly DNSBL shutdown because the DNSBL nameserver records can be specified separately from the domain's principal nameservers.



 TOC 

3.2.  DNSBLs SHOULD be Adequately Provisioned

The DNSBL should have sufficient name server capacity to handle the expected loading, and have sufficient redundancy to handle normal outages.

If the DNSBL offers zone transfers (in addition to or instead of standard DNSBL query mechanisms), it should be sufficiently provisioned to handle the expected loading.

Note that some DNSBLs have been subject to distributed denial of service attacks. Provisioning should take the likelyhood of this into account.



 TOC 

3.3.  DNSBLs SHOULD Provide Operational Flags

Most DNSBLs follow a convention of entries for IPs in 127.0.0.0/8 to provide online indication of whether the DNSBL is operational. Many DNSBLs arrange to have a query of 127.0.0.2 return an A record indicating that the IP is listed, and a query of 127.0.0.1 return no A record (NXDOMAIN). When both of these indicators are present, this indicates that the DNSBL is functioning normally. See [DNSBL‑EMAIL] (Levine, J., “DNS-based Blacklists and Whitelists for E-Mail,” November 2005.).

Operational flag usage and meaning SHOULD be published on the DNSBL's web site.



 TOC 

3.4.  Shutdowns MUST Be Done in a Graceful Fashion

A number of DNSBLs have shut down operations in such a way as to list the entire Internet, sometimes without warning. These were usually done this way to force DNSBL users (mail administrators) to adjust their DNSBL client configurations to omit the now inoperative DNSBL and to shed the DNS query load from the registered domain name servers. Popular DNSBLs are in use by 10s of thousands of sites, are not well monitored, and often not compliant with DNSBL query conventions (e.g.: will treat any A record return as being "listed", instead of specific 127/8 A record returns) hence shutdowns can be quite destructive to all email flow if not done properly.

The DNSBL operator MUST issue impending shutdown warnings (on the DNSBL web site, appropriate mailing lists, newsgroups, vendor newsletters etc), and indicate that the DNSBL is inoperative using the signalling given in Section 3.3 (DNSBLs SHOULD Provide Operational Flags).

Only after these warnings have been issued for a significant period of time (RECOMMENDED: one or more months), should the DNSBL operator finally shutdown the DNSBL.

The shutdown procedure should have the following properties:

  1. MUST NOT list the entire Internet
  2. SHOULD shed the DNSBL query load from the DNSBL name servers, permitting the registered domain name to continue being useable.
  3. SHOULD, perhaps through increased delays, indicate to the Mail administrator that the DNSBL is no longer functional.
  4. The base domain name SHOULD be registered indefinately, so as to ensure that the domain name doesn't represent a "booby trap" for future owners, and/or provide a means by which a new owner could list the entire Internet.

One way of satisfying the points 1-3 above is to change the DNS name servers for the DNSBL to point at "TEST-NET" addresses (see RFC3330 (IANA, “Special-Use IPv4 Addresses,” September 2002.) [RFC3330]). The below suggested [BIND] (Internet Systems Corporation, “ISC BIND,” .) declarations will cause a DNSBL query to query name servers in TEST-NET addresses, which will result in a significant delay, but not return any A records except in very unusual circumstances.

BIND-equivalent DNS declarations for DNSBL shutdown.

dnsbl.example.com.  604800  IN  NS  u1.example.com.
dnsbl.example.com.  604800  IN  NS  u2.example.com.
dnsbl.example.com.  604800  IN  NS  u3.example.com.

... [as many as you like]

u1.example.com.     604800  IN  A   192.0.2.1
u2.example.com.     604800  IN  A   192.0.2.2
u3.example.com.     604800  IN  A   192.0.2.3

... [as many as you like]

Assumes DNSBL is named "dnsbl.example.com". Replace "example.com" and "dnsbl.example.com" as appropriate for the DNSBL

NOTE:
Of course, the above shutdown procedure cannot be implemented if Section 3.1 (DNSBL Query Root Domain SHOULD be a Subdomain) is not followed.


 TOC 

3.5.  Listing of Special and Reserved IP Addresses MUST be disclosed

The DNSBL MAY list loopback, RFC 1918 (Rekhter, Y., Moskowitz, R., Karrenberg, D., Groot, G., and E. Lear, “Address Allocation for Private Internets,” February 1996.) [RFC1918], LINK-LOCAL class D/E, and any other permanently reserved or special-use IP addresses. Such use MUST be disclosed in the documentation related to the DNSBL.

A functioning DNSBL MUST NOT list 127.0.0.1



 TOC 

3.6.  Use of Collateral Damage MUST Be Disclosed

Some DNSBLs have adopted a policy of using "collateral damage" as an intentional element of the list's operation. When collateral damage is applied, a DNSBL listing is expanded to include IP addresses under the control of a provider even though those IP addresses are not the source of abusive email. The theory is that customers impacted by collateral damage will apply pressure to the provider to take action against the customer which is the cause of the original listing.

Collateral damage as a DNSBL policy is highly controversial, and discussion of the appropriateness of this policy is beyond the scope of this document. However, if a DNSBL has policies and practices that include the imposition of collateral damage, such policies MUST be disclosed.



 TOC 

3.7.  Considerations for DNSBLs Listing Insecure Hosts

Some DNSBLs list IP addresses of hosts that are insecure in various ways (e.g. open relays, open proxies). The following recommendations for such DNSBLs may not be relevant to other types of DNSBLs.



 TOC 

3.7.1.  MUST NOT scan without provocation

DNSBLs MUST NOT automatically probe for insecure systems without provocation. There is little agreement in the community as to whether or not such activity should be allowed, so this BCP errs on the side of caution.

Therefore, scanning MUST be be targeted, rather than broad-based, where a given scan is motivated by a specific reason to have concern about the address being scanned. Examples of such reasons include delivery to a spam trap address, receipt of a user complaint, or periodic testing of an address that is already listed.



 TOC 

3.7.2.  Re-scan Periods SHOULD be Reasonable

If the DNSBL administrator re-scans a host in order to determine whether the listing SHOULD timeout or not, the re-scan period SHOULD be reasonable. Automated scanning SHOULD NOT occur more often than once every 24 hours.



 TOC 

4.  Security Considerations

Any system manager that uses DNSBLs is entrusting part of his or her server management to the parties that run the lists. A DNSBL manager that decided to list 0/0 (which has actually happened) could cause every server that uses the DNSBL to reject all mail. Conversely, if a DNSBL manager removes all of the entries (which has also happened), systems that depend on the DNSBL will find that their filtering doesn't work as they want it to.

If a registered domain used for a DNSBL is allowed to lapse, or the DNSBL user spells the DNSBL domain name incorrectly, the system manager's server management is now subject to an entirely different party than was intended. Further, even if there is no malicious intent, some DNSBL query clients will interpret any A record being returned as being listed.

Like all DNS-based mechanisms, DNSBLs are subject to various threats outlined in RFC 3833 (Atkins, D. and R. Austein, “Threat Analysis of the Domain Name System (DNS),” August 2004.) [RFC3833]



 TOC 

5.  References



 TOC 

5.1. Normative References

[RFC1918] Rekhter, Y., Moskowitz, R., Karrenberg, D., Groot, G., and E. Lear, “Address Allocation for Private Internets,” BCP 5, RFC 1918, February 1996 (TXT).
[RFC2014] Weinrib, A. and J. Postel, “IRTF Research Group Guidelines and Procedures,” BCP 8, RFC 2014, October 1996 (TXT, HTML, XML).
[RFC2119] Bradner, S., “Key words for use in RFCs to Indicate Requirement Levels,” BCP 14, RFC 2119, March 1997 (TXT, HTML, XML).
[RFC3330] IANA, “Special-Use IPv4 Addresses,” RFC 3330, September 2002 (TXT).
[RFC3833] Atkins, D. and R. Austein, “Threat Analysis of the Domain Name System (DNS),” RFC 3833, August 2004 (TXT).


 TOC 

5.2. Informative References

[BIND] Internet Systems Corporation, “ISC BIND.”
[DNSBL-EMAIL] Levine, J., “DNS-based Blacklists and Whitelists for E-Mail,” November 2005 (txt).
[RFC2026] Bradner, S., “The Internet Standards Process -- Revision 3,” BCP 9, RFC 2026, October 1996 (TXT).


 TOC 

Appendix A.  Acknowledgements

We would like to thank John R. Levine, Alan Murphy and Dave Crocker for their insightful comments.

We would also like to thank Yakov Shafranovich and Nick Nicholas for editing previous versions of this document.



 TOC 

Authors' Addresses

  Chris Lewis
  Nortel Networks
Email:  clewis@nortel.com
  
  Matt Sergeant
  MessageLabs, Inc
Email:  msergeant@messagelabs.com


 TOC 

Full Copyright Statement

Intellectual Property