Network Working Group R. Whittle Internet-Draft First Principles Intended status: Experimental March 07, 2010 Expires: September 8, 2010 Ivip (Internet Vastly Improved Plumbing) Architecture draft-whittle-ivip-arch-04.txt Abstract Ivip (Internet Vastly Improved Plumbing) is a Core-Edge Separation solution to the routing scaling problem, for both IPv4 and IPv6. It provides portable address "edge" address space which is suitable for multihoming and inbound traffic engineering (TE) to end-user networks of all types and sizes - in a manner which imposes far less load on the DFZ control plane than the only current method of achieving these benefits: separately advertised PI prefixes. Ivip includes two extensions for ITR-to-ETR tunneling without encapsulation and the Path MTU Discovery problems which result from encapsulation - one for IPv4 and the other for IPv6. Both involve modifying the IP header and require most DFZ routers to be upgraded. Ivip is a good basis for the TTR (Translating Tunnel Router) approach to mobility, in which mobile hosts retain an SPI micronet of one or more IPv4 addresses (or IPv6 /64s) no matter what addresses or access network they are using, including behind NAT and on SPI addresses. TTR mobility for both IPv4 and IPv6 involves generally optimal paths, works with unmodified correspondent hosts and supports all application protocols. Status of this Memo This Internet-Draft is submitted to IETF in full conformance with the provisions of BCP 78 and BCP 79. Internet-Drafts are working documents of the Internet Engineering Task Force (IETF), its areas, and its working groups. Note that other groups may also distribute working documents as Internet- Drafts. Internet-Drafts are draft documents valid for a maximum of six months and may be updated, replaced, or obsoleted by other documents at any time. It is inappropriate to use Internet-Drafts as reference material or to cite them other than as "work in progress." The list of current Internet-Drafts can be accessed at http://www.ietf.org/ietf/1id-abstracts.txt. Whittle Expires September 8, 2010 [Page 1] Internet-Draft Ivip Architecture March 2010 The list of Internet-Draft Shadow Directories can be accessed at http://www.ietf.org/shadow.html. This Internet-Draft will expire on September 8, 2010. Copyright Notice Copyright (c) 2010 IETF Trust and the persons identified as the document authors. All rights reserved. This document is subject to BCP 78 and the IETF Trust's Legal Provisions Relating to IETF Documents (http://trustee.ietf.org/license-info) in effect on the date of publication of this document. Please review these documents carefully, as they describe your rights and restrictions with respect to this document. Code Components extracted from this document must include Simplified BSD License text as described in Section 4.e of the Trust Legal Provisions and are provided without warranty as described in the BSD License. Whittle Expires September 8, 2010 [Page 2] Internet-Draft Ivip Architecture March 2010 Table of Contents 1. Introduction . . . . . . . . . . . . . . . . . . . . . . . . . 5 2. Brief description of Ivip . . . . . . . . . . . . . . . . . . 7 3. The routing scaling problem and other goals for an architectural enhancement . . . . . . . . . . . . . . . . . . 11 4. Summary of Ivip's architectural choices . . . . . . . . . . . 14 5. Goals . . . . . . . . . . . . . . . . . . . . . . . . . . . . 16 5.1. IPv4 and IPv6 . . . . . . . . . . . . . . . . . . . . . . 16 5.2. Portability, multihoming and TE for billions of end-user networks . . . . . . . . . . . . . . . . . . . . 16 5.3. Modular separation of the control of mapping from the CES architecture itself . . . . . . . . . . . . . . . . . 18 5.4. Simple ITRs and ETRs with little or no communication between them . . . . . . . . . . . . . . . . . . . . . . . 19 5.5. Maximise the flexibility with which ITRs and ETRs can be located . . . . . . . . . . . . . . . . . . . . . . . . 20 5.6. Mobility . . . . . . . . . . . . . . . . . . . . . . . . . 20 5.7. Elimination of encapsulation and PMTUD problems . . . . . 21 5.8. No requirement for new host functionality . . . . . . . . 23 5.9. Full benefits to all adopters irrespective of level of adoption . . . . . . . . . . . . . . . . . . . . . . . . . 24 5.10. Business incentives to deploy new infrastructure . . . . . 24 5.11. Maintenance of existing levels of security and robustness . . . . . . . . . . . . . . . . . . . . . . . . 25 5.12. Avoiding the need for any one server to store or receive the complete mapping database . . . . . . . . . . 26 5.13. Eliminating unfair burdens . . . . . . . . . . . . . . . . 27 6. Non-goals . . . . . . . . . . . . . . . . . . . . . . . . . . 29 6.1. Isolation between core and edge networks is not required . . . . . . . . . . . . . . . . . . . . . . . . . 29 6.2. Full adoption not required . . . . . . . . . . . . . . . . 29 6.3. Mapping changes need not be free of financial cost . . . . 30 6.4. No attempt to cope with partially reachable ETRs . . . . . 31 6.5. No attempt to mix IPv4 and IPv6 . . . . . . . . . . . . . 33 6.6. Not Locator - Identifier Separation . . . . . . . . . . . 33 7. Architectural Choices . . . . . . . . . . . . . . . . . . . . 35 7.1. Core-Edge Separation rather than Elimination . . . . . . . 35 7.1.1. Core-Edge Elimination (CEE) architectures . . . . . . 35 7.1.2. Core-Edge Separation (CES) architectures . . . . . . . 38 7.2. Nearby authoritative query servers . . . . . . . . . . . . 39 7.3. Real-time mapping distribution . . . . . . . . . . . . . . 41 7.4. SPI address management . . . . . . . . . . . . . . . . . . 41 7.5. IP in IP encapsulation . . . . . . . . . . . . . . . . . . 44 7.6. MHF initially or in the long term to avoid encapsulation and PMTUD problems . . . . . . . . . . . . . 44 7.7. Outer header address is that of the sending host . . . . . 44 7.8. IPTM (ITR Probes Tunnel MTU) PMTUD management . . . . . . 45 Whittle Expires September 8, 2010 [Page 3] Internet-Draft Ivip Architecture March 2010 8. Architectural Elements . . . . . . . . . . . . . . . . . . . . 48 8.1. ITRs . . . . . . . . . . . . . . . . . . . . . . . . . . . 48 8.1.1. Types of ITR and their addresses . . . . . . . . . . . 48 8.1.2. DITRs - Default ITRs in the DFZ . . . . . . . . . . . 49 8.1.3. Modified Header Forwarding - MHF-only ITRs . . . . . . 50 8.1.4. Encapsulation and PMTUD management . . . . . . . . . . 50 8.1.5. Mapping lookup and caching . . . . . . . . . . . . . . 52 8.1.6. ITFH - ITR Function in Host . . . . . . . . . . . . . 55 8.1.7. ITRs auto-discovering local query servers . . . . . . 55 8.2. ETRs . . . . . . . . . . . . . . . . . . . . . . . . . . . 56 8.2.1. In servers or dedicated routers . . . . . . . . . . . 56 8.2.2. ETRs in ISP networks . . . . . . . . . . . . . . . . . 56 8.2.3. ETRs at the end-user network site . . . . . . . . . . 56 8.2.4. MHF ETR functionality - EAF and PLF . . . . . . . . . 57 8.2.5. ETR functionality for encapsulation . . . . . . . . . 58 8.3. QSRs - Resolving Query Servers . . . . . . . . . . . . . . 58 8.4. QSCs - caching query servers . . . . . . . . . . . . . . . 59 8.5. MHF - Modified Header Forwarding . . . . . . . . . . . . . 60 8.5.1. EAF - ETR Address Forwarding for IPv4 . . . . . . . . 60 8.5.2. PLF - Prefix Label Forwarding, for IPv6 . . . . . . . 61 8.6. TTR Mobility . . . . . . . . . . . . . . . . . . . . . . . 62 9. Security Considerations . . . . . . . . . . . . . . . . . . . 64 10. IANA Considerations . . . . . . . . . . . . . . . . . . . . . 65 11. Informative References . . . . . . . . . . . . . . . . . . . . 66 Appendix A. Acknowledgements . . . . . . . . . . . . . . . . . . 69 Author's Address . . . . . . . . . . . . . . . . . . . . . . . . . 70 Whittle Expires September 8, 2010 [Page 4] Internet-Draft Ivip Architecture March 2010 1. Introduction Version 03 (2010-01-13) of this Ivip-arch ID was a freshly written document which is shorter than the original from 2007. Some terminology has been changed and the presentation is optimised for people who are involved in the RRG. Please see [I-D.whittle-ivip-glossary] for definitions of some terms and acronyms. Please refer to the RRG mailing list and http://www.firstpr.com.au/ip/ivip/ for the latest developments. This Version 04 includes significant changes to Ivip's mapping system. The DRTM (Distributed Real Time Mapping) system [I-D.whittle-ivip-drtm] removes the need for "Replicators", or for any server to carry the full Ivip mapping database. While DRTM is discussed in this Ivip-arch ID, please see the Ivip-drtm ID for a full description of this system, and how it enables the introduction of scalable routing solutions and global mobility with the initiative and investments being made by organisations which need not be ISPs. The Ivip (pr. "Eye-vip") project began in June 2007 and in early 2010 is one of the four Core-Edge Separation (CES) architectures being considered by the RRG (IRTF Routing Research Group) [I-D.irtf-rrg-recommendation] - the others being IRON-RANGER, LISP [I-D.ietf-lisp] and TIDR [I-D.adan-idr-tidr]. For my overall assessment of the proposals submitted to the RRG, and for my arguments for why Ivip is the most suitable for further IETF development, please see http://www.ietf.org/mail-archive/web/rrg/current/msg06162.html ("Recommendation suggestion from RW" 2010-03-04) My discussion of the other proposals can be found in the RRG Archives of January and February 2010. I publicly disclose and discuss all Ivip developments as rapidly as possible in order to gain support and constructive critiques - and in the hope that any novel ideas will remain free from patent encumbrances. This ID is intended for readers who are broadly familiar with the routing scaling problem and RRG discussions and who have, ideally, familiarised themselves with LISP. This ID provides not only a general description of Ivip, but the rationale for architectural choices which distinguish Ivip from other approaches. Some aspects of Ivip's architecture are discussed in greater detail in separate documents: The DRTM (Distributed Real Time Mapping) system Whittle Expires September 8, 2010 [Page 5] Internet-Draft Ivip Architecture March 2010 [I-D.whittle-ivip-drtm] describes the new approach to Ivip's real- time mapping system, which uses multiple typically "nearby" full database query servers provided directly or indirectly by MABOCs. The TTR approach to mobility is described in [TTR Mobility]. The IPv4 approach to Modified Header Forwarding (MHF) is described in detail in [I-D.whittle-ivip-etr-addr-forw]. The IPv6 approach is described in [PLF for IPv6] and the best summary of its operation can be found at the end of the ~10k word Ivip Conceptual Summary and Analysis: [Ivip Summary and Analysis] . Ivip's approach to Path MTU Discovery, when ITRs tunnel using encapsulation, is discussed in [PMTUD-Frag]. Whittle Expires September 8, 2010 [Page 6] Internet-Draft Ivip Architecture March 2010 2. Brief description of Ivip Ivip (Internet Vastly Improved Plumbing) is a Core-Edge Separation solution to the routing scaling problem, for both IPv4 and IPv6. It provides portable address "edge" address space which is suitable for multihoming and inbound traffic engineering (TE) to end-user networks of all types and sizes - in a manner which imposes far less load on the DFZ control plane than the only current method of achieving these benefits: separately advertised PI prefixes. The new "edge" subset of the global unicast address space which is used in this fashion is called SPI (Scalable Provider Independent) space. End-user networks divide their SPI space into "micronets", each with a common mapping to a single ETR (Egress Tunnel Router) address. Micronets have arbitrary starting points and integer lengths - in units of IPv4 addresses or, for IPv6, /64 prefixes. When an ITR (Ingress Tunnel Router) receives a packets which are addressed to an SPI address. After looking up the mapping of the micronet which covers the destination address, the ITR tunnels the traffic packet to the ETR specified in that mapping - and the ETR delivers the packet to the end-user network. A Mapped Address Block (MAB) is a DFZ-advertised prefix of global unicast address space which is typically divided up into many separate micronets - such as hundreds to hundreds of thousands of micronets, each of which can be used via any ISP. The total set of all MABs constitutes the "edge" (SPI) subset of the global unicast address range. The remainder is known as "core" space. A MAB is managed by an MABOC (MAB Operating Company). MABOCs may be end-user networks and the micronets their MABs contain may be used solely for that end-user network - but each micronet can be mapped to any ETR in the world. More typically, MABOCs will lease the SPI space to large numbers of end-user networks on a commercial basis, rather than use it themselves. The mapping of each micronet is controlled directly by the end-user network which owns or leases the portion of SPI space the micronet is within - or by another organization appointed by this end-user network. Multihoming end-user networks would typically contract a separate company to change the mapping of their micronets, in response to the reachability of their network through their two or more ETRs and according to the network's inbound TE requirements. DITRs (Default ITRs in the DFZ) are required for handling packets sent to SPI addresses from hosts in networks without ITRs. The one or more DITRs at a DITR site advertise in the DFZ the MABs the site Whittle Expires September 8, 2010 [Page 7] Internet-Draft Ivip Architecture March 2010 supports, which is typically a subset of all MABs in the Ivip system. ITRs other than DITRs request mapping for SPI addresses from local Resolving Query Servers (QSRs) in their own network or in their ISP's network. They may do this directly or through one or more levels of caching query servers - QSCs. QSRs are caching query servers which query multiple, distributed, authoritative query servers (QSAs) which are typically "nearby", such as within a few thousand km. QSAs are located at a number of widely dispersed sites, such as 5 to 50, where DITRs are located and run by, or for, these MABOCs. Each QSA is authoritative for only a subset of all MABs - the set supported by that DITR site. Each QSR uses a DNS-based mechanism and an additional protocol to discover two or more typically "nearby" QSAs for each MAB. Since each QSA handles mapping requests for multiple MABs, this means the number of such QSA's each QSR needs to discover is much less than the number of MABs. The number of MABs is much less than the number of end-user networks using SPI space - and the number of micronets is greater than this number, since each end-user network may have many micronets. End-user networks or their appointees generate real-time mapping changes using facilities provided by the MABOC which manages the MAB the micronet is located within. Most mapping changes will be to change the ETR address of an existing micronet. Other mapping changes will redefine how an end-user network's SPI space is divided into separate micronets. MABOCs will typically charge their customers for each mapping change. These mapping changes are transmitted in real-time from the MABOC to the organisation which runs the DITR-sites with DITRs which advertise this MAB. The mapping changes are received and incorporated into a real-time updated full mapping database for this MAB, in one or more QSAs at each site. One or more of these QSAs handle mapping queries from the DITRs at the site and one or more handle mapping queries from QSRs in typically nearby ISP and end-user networks. Any QSR can send queries to any QSA, but would normally choose nearby ones. QSAs can give feedback in mapping replies concerning how busy they are, with suggestions of other QSAs to use instead. So there is natural load-sharing with multiple QSAs being spread around the world and dynamic load-balancing between them according to actual loads. Since no one QSA or DITR-site is required to handle the full set of MABs, since each DITR-site organization controls its own real-time push of mapping to its sites, and since there can be any number of DITR-sites and any number of DITR-site operating companies, there are Whittle Expires September 8, 2010 [Page 8] Internet-Draft Ivip Architecture March 2010 no obvious scaling limits on the number of micronets the entire system can handle, or the frequency of mapping updates to those micronets. If a given global set of DITR-sites hits some kind of scaling limit in these respects, then the total load can be handled by more such systems of DITR-sites. QSRs too can be installed in larger numbers in a busy ISP or end-user network if the query demand exceeds the capacity of one such server. Each QSR can automatically discover very large numbers (tens to hundreds of thousands) of QSAs, and each QSA will typically handle dozens to hundreds or perhaps thousands of MABs. While there is no assurance of nearby QSAs, MABOCs will generally want to have numerous widely dispersed DITR sites, each with QSAs for two reasons. Firstly to ensure the DITRs tunnel packets without been too far from the path between the sending host (in a network without ITRs) and the ETR. Secondly to encourage ISPs and larger end-user networks to install ITRS and use the QSAs - since this will result in shorter paths for packets and less load on the DITRs. ISPs and end-user networks do not absolutely need to install ITRs. However ISPs will be motivated to install them (and therefore install several QSRs for them to send mapping queries to) for two reasons. Firstly, to ensure the ISP's customer's SPI-addressed packets are tunneled reliably, rather than relying on DITRs. Secondly, when their customers send SPI-addressed packets to SPI-using end-user networks which are also customers of the ISP, if the ISP has its own ITRs, then these packets do not leave the ISP's network. Without ITRs, they would leave the network via an expensive upstream link, be tunneled by a DITR and return via the same or a different upstream link. Since end-user networks can run their own ETRs on existing PA address space they get from their ISP, the only thing an ISP needs in order to allow such a network to use SPI space is to accept outgoing packets for forwarding when they have SPI source addresses. All other initiatives and investments - including the provision of multiple widely dispersed DITRs, QSAs and the real-time push of mapping changes to these - is undertaken by the MABOCs who profit by renting SPI space to their end-user customers. A MABOC need not be an ISP. Ivip includes two extensions for ITR-to-ETR tunneling without encapsulation and the Path MTU Discovery problems which result from encapsulation - one for IPv4 and the other for IPv6. Both involve modifying the IP header and require most DFZ routers to be upgraded. Ivip is a good basis for the TTR (Translating Tunnel Router) approach Whittle Expires September 8, 2010 [Page 9] Internet-Draft Ivip Architecture March 2010 to mobility, in which mobile hosts retain an SPI micronet of one or more IPv4 addresses (or IPv6 /64s) no matter what addresses or access network they are using, including behind NAT and on SPI addresses. TTR mobility for both IPv4 and IPv6 involves generally optimal paths, works with unmodified correspondent hosts and supports all application protocols. TTR Mobility is described in: [TTR Mobility] Whittle Expires September 8, 2010 [Page 10] Internet-Draft Ivip Architecture March 2010 3. The routing scaling problem and other goals for an architectural enhancement For a fuller account of my understanding of the routing scaling problem, and other problems which should be considered when devising an architectural enhancement to the Internet, please see http://www.ietf.org/mail-archive/web/rrg/current/msg06099.html ("Scalable routing problem & architectural enhancements" 2010-02-23) and http://www.ietf.org/mail-archive/web/rrg/current/msg06162.html ("Recommendation suggestion from RW" 2010-03-04). The most visible aspect of the routing scaling problem can be summarised as there being practical problems and unfair cost-burdens due to the growth in the number of PI prefixes end-user networks advertise in the DFZ. Advertising PI prefixes is currently the only method of providing portability, multihoming and inbound traffic engineering (TE) for end-user networks. The same problem exists in principle for IPv4 and IPv6, but only IPv4 has a problem at present. The less visible part of it is the large number of end-user networks who are unable to gain these benefits due to the costs and other barriers to obtaining their own address space and advertising it in the DFZ. Part of the reason for these costs and barriers is the push-back against this practice, due to concerns about the burden each PI prefix places on the DFZ control plane. Another part is the cost and other difficulties of obtaining the minimum amount of space which can be advertised in the DFZ - currently 256 IPv4 addresses as a /24 prefix. The burden placed on the interdomain routing system (often referred to loosely as the Default-Free Zone - DFZ) by the prefixes advertised by ISPs is generally thought not to be a problem. So the challenge is to find a way of providing address space and new methods of routing so that the portability, multihoming and TE needs of potentially millions or billions of end-user networks can be served in a "scalable" manner: efficiently, robustly and without unfair burdens falling on anyone, such as those who operate the DFZ routers. The unfair, unsustainable, burden is caused by the number of separately advertised PI prefixes of end-user networks today - and the rate at which these prefixes have their point of advertisement changed. (Also, if an end-user network changes the type of advertisement frequently, such as with more or less ASNs, this too is a burden.) Please see http://www.ietf.org/mail-archive/web/rrg/current/msg06163.html ("Geoff Huston's BGP/DFZ research" 2010-03-05) for up-to-date analysis of trends in the number of prefixes and in the problems caused by changes to those prefixes. Whittle Expires September 8, 2010 [Page 11] Internet-Draft Ivip Architecture March 2010 The most important part of the burden is on the DFZ's "BGP control plane". This is partly the inter-router BGP traffic and the overall behaviour of routers - particularly any difficulty which the excessive number of prefixes causes in the system converging to good enough best-paths in the event of an outage. It is also the burden of CPU effort and storage in the RIB of each router. This includes the effort of writing changes to the FIB when RIB information changes. Also, FIBs may have their ability to handle packets temporarily disabled while new information is written. The actual number of prefixes each DFZ router has to handle is a major part of the problem, though the total RIB burden also depends on how many neighbours each router has. The number of prefixes in the FIB is a serious burden too, but it is widely believed that this is not the most important problem. Any solution which only helps reduce the number of prefixes the FIB must handle is not really a solution to the problem. The number of prefixes advertised in the DFZ is the most obvious and directly costly part of the routing scaling problem - analogous to the tip of an iceberg. The larger, harder-to-measure, part of the problem is the unknown number of end-user networks which want or need portability, multihoming and/or inbound TE but which cannot obtain it at present, due to the costs and other barriers to gaining address space and advertising it as PI prefixes. In order to provide portability etc. to millions or perhaps billions of end-user networks in a scalable manner, it follows that the DFZ routers must not have to consider the prefixes of each individual network in their RIB or FIB. Consequently, the Core-Edge Separation class of scalable routing architectures work by providing a special subset of the global unicast address space, which is suitable and attractive for providing end-user networks with portability, multihoming and TE, but which places only very slight burden on the DFZ compared to the burden each PI prefix places today. (Core-Edge Elimination architectures have a different approach, which is discussed below in "Architectural Choices - Core-Edge Separation rather than Elimination". Support for mobility has not generally been considered part of the routing scaling problem. However, mobility is prominently mentioned in the RRG Charter. With the proliferation of cellphones, VoIP, other IP applications it is reasonable to assume that in the future - such as by 2020 - most hosts will be mobile devices, generally running on limited battery power and relying on wireless links which are frequently slow, unreliable and/or expensive. Mobility is arguably an extreme form of portability and/or Whittle Expires September 8, 2010 [Page 12] Internet-Draft Ivip Architecture March 2010 multihoming. To embark on a major architectural enhancement for scalable routing, in a manner which did not support billions of mobile devices, would make little sense. While provision of mobility is frequently assumed not to be related to interdomain routing, it is prominent in the RRG's Charter. The TTR (Translating Tunnel Router) Mobility architecture [TTR Mobility] is a new approach to global mobility, for both IPv4 and IPv6 - and is an extension of a CES architecture such as Ivip. In the TTR Mobility architecture, each mobile device is generally considered to be a separate end-user network. An entire corporation's network, or that of a large university, is also an "end-user network". So in the following discussion, this term could mean a wide variety of things - far beyond the small subset of end- user networks whish are currently able to gain and advertise PI space. Whittle Expires September 8, 2010 [Page 13] Internet-Draft Ivip Architecture March 2010 4. Summary of Ivip's architectural choices Ivip is based on some unique architectural choices, including: ITRs (Ingress Tunnel Routers) receiving mapping changes in real-time; typically "nearby" QSA authoritative query servers which are "full- database" for at least one MAB, but typically a significant fraction of all MABs; migration to Modified Header Forwarding (MHF) to avoid encapsulation and its PMTUD (Path MTU Discovery) difficulties; and (when encapsulation is used) the use of the sending host's address as the outer header's source address, so that ETRs can easily enforce ISP BR (Border Router) source address filtering on decapsulated packets. The following description assumes that Ivip will be introduced with encapsulation, with long-term migration to MHF. However, it is possible that by the time the introduction date is set that most DFZ routers will have firmware based FIBs, and so could be easily upgraded to support MHF. In that case, ITRs and ETRs could be much simpler, since they would not need to handle encapsulation or PMTUD management. Below, Ivip is generally assumed to be introduced as a single system for the purposes of solving the routing scaling problem. However, multiple independent systems along the lines of Ivip (with encapsulation) could also be introduced without need for standardisation for the purpose of supporting commercial TTR Mobility services. The adoption of an architectural enhancement to improve routing scalability is frequently assumed to depend largely or entirely on ISPs making the initial investment. However, with DRTM, this need not be the case. DRTM enables SPI space to be leased to end-user networks - with full support for portability, multihoming and inbound TE for all their communications - with the investment and initiative being taken by organisations which may not be ISPs. These are the MABOCs - MAB Operating Companies - who lease the space in each MAB they control to typically thousands to hundreds of thousands separate end-user networks. An SPI-adopting end-user network can run its own ETR on the existing PA space it obtains from each of its one or more ISPs. ISPs need make no investment to allow this to proceed - but they must forward the outgoing packets from these SPI-adopting networks which have SPI source addresses. DRTM removes the need for the Replicators, full-database query servers in ISP networks and "Missing Payload Servers" which are described in the more recent ID: Ivip Fast Payload Replication Whittle Expires September 8, 2010 [Page 14] Internet-Draft Ivip Architecture March 2010 [I-D.whittle-ivip-fpr]. However, within DRTM, there remains an option to have the caching QSR (Resolving Query Servers) be full database for one, multiple or perhaps all MABs, and to use a small (such as between several nearby ISPs) Replicator system as part of fanning out mapping updates from DITR-sites to these "full-database" Map Resolvers. DRTM does not currently specify how the organisations which run DITR sites reliably and securely deliver the real-time mapping to each such site. This is an internal matter for these organisations and the potentially multiple MABOCs they receive this mapping information from. It is possible that Replicators could be part of these arrangements too. With TTR mobility, the MN (Mobile Node) can be in any access network at all, including behind one or more layers or NAT and including being on SPI space in an end-user network which has adopted SPI space. In all cases, the MN needs no support from the network it is currently connected to, since the MN establishes a two-way tunnel to the TTR and sends its SPI source address outgoing packets to the TTR for forwarding. So TTR mobility is a scalable routing solution which requires no investment or support from ISPs, and in which the initiative and investment comes from TTR Mobility companies, which need not be ISPs. Whittle Expires September 8, 2010 [Page 15] Internet-Draft Ivip Architecture March 2010 5. Goals 5.1. IPv4 and IPv6 Ivip is intended to solve the routing scaling problem (as described in the introduction), for IPv4 and IPv6, for very large numbers of end-user networks - where this includes a single MN (Mobile Node) within the definition of "end-user network". Much of Ivip is identical in principle for both Internets. However the mapping information for IPv6 is lengthier and there are other differences, such as in Path MTU Discovery (PMTUD) when encapsulation is used, and in the IPv4 and IPv6 approaches to MHF which remove the need for encapsulation. 5.2. Portability, multihoming and TE for billions of end-user networks Ivip is intended to provide scalable address space for billions of end-user networks - for both IPv4 and IPv6. The new kind of address space - SPI (Scalable Provider Independent) space - is suitable for end-user networks to use in a portable fashion, meaning they can keep this space when choosing another ISP for Internet connectivity. There is an assumed upper bound of order 10^7 on the number of non- mobile end-user networks. This is on the basis of a population of 10^10 and there being typically no more than one organization per 10^3 people which needs portability, multihoming and/or inbound TE enough to invest in a second ISP service and whatever else is required to achieve these goals. Brian Carpenter suggested the same thing (http://www.ietf.org/mail-archive/web/rrg/current/msg05801.html 2010-01-27). Given the growing ubiquity of cell-phones and the desire to give them IP connectivity with mobility, including session survival when changing access networks, it is reasonable to assume an upper bound of order 10^10 on the number of "mobile end-user networks". This order 10^10 upper bound has been discussed on in the RRG and no-one has suggested a routing scaling solution with mobility should aim for any greater number of end-user networks. In Ivip's mapping system and in its ITRs, no distinction is made between end-user networks which are mobile or non-mobile, so the total number 10^10 is the upper bound on number of micronets for the Ivip system to handle. In IPv4, since the smallest micronet is a single IPv4 address and there are only 3.7 billion global unicast addresses in total, from which the "edge" SPI addresses can be drawn, it follows that for Ivip in IPv4, there can be no more than probably 3 x 10^9 micronets. Whittle Expires September 8, 2010 [Page 16] Internet-Draft Ivip Architecture March 2010 Portability of the end-user network address space which is used to identify hosts, routers and networks is an absolute requirement of scalable routing. Even if a network could reliably and inexpensively renumber all its hosts and routers, and change all its configuration files which contained such addresses, it would never be able to reliably and securely alter all the other places where these identifying addresses reside in other networks. These includes the use of these addresses in referrals, existing communication sessions, config files of VPNs and hard-coded (however questionably) into firmware and software. Another example of the need for portability is end-user networks which host services for other organisations - typically their customers - in a way that the IP addresses of the network's hosts appear in the DNS zone files of these other organizations. For the network to have to renumber its network, such as to use PA space from another ISP, would require costly, error- prone and carefully timed updates to zone files of all these other organizations. Assuming the end-user network has two or more ISPs, SPI space will also support multihoming and inbound traffic engineering. In the following, "TE" refers to "inbound traffic engineering" - the ability to steer incoming traffic streams between two or more ISPs. (Outbound TE is simply a matter of sending outgoing packets out whichever ISP link is desired.) Ivip's approach to TE differs from that of other CES architectures. It is potentially finer-grained, more flexible and more able to respond to rapid changes in traffic patterns. The goal of scalable routing is to scalably provide portability, multihoming and TE to all networks which want or need it. However, it is reasonable to assume that most home and SOHO networks, and some smaller factory and office networks, will remain happy with the reliability of their single-provider service, and will not concerned about portability when choosing another ISP. A small number of end-user networks will have multiple sites or some other reason to split their SPI space into multiple micronets, but in any realistic scenario involving billions of such networks, the great majority of such networks will be a single site or device, with little or no need for TE or greater address space than a single IPv4 address or an IPv6 /64. Therefore, it is reasonable to expect that most of these billions of networks will require only a single micronet of SPI addresses. So, for these scenarios of billions of end-user networks, the total number of separately mapped micronets of SPI address space will be only marginally greater than the number of end-user networks. Whittle Expires September 8, 2010 [Page 17] Internet-Draft Ivip Architecture March 2010 5.3. Modular separation of the control of mapping from the CES architecture itself Ivip's real-time mapping system means that the tunneling behaviour of all ITRs can be controlled directly. The mapping consists of a single ETR address, so Ivip ITRs do not need to make any choices between multiple ETRs for the purposes of multihoming service restoration or TE. The non-Ivip CES architectures do not provide real-time mapping to ITRs, and therefore need to have the ITRs perform their own multihoming reachability testing and decision- making, to choose which of several ETRs to tunnel packets to. Control of the tunneling behaviour of Ivip ITRs rests entirely outside the Ivip system. It is the responsibility of end-user networks to control this mapping at all times - and many end-user networks are likely to delegate this responsibility to a company they hire for this purpose. Exactly how end-user networks make their decisions about mapping - and how, for instance, a Multihoming Monitoring (MM) company might detect ETR failure, and alter mapping accordingly - is entirely separate from Ivip's mapping system, ITRs and ETRs. This appointment of another organization to control the mapping of one or more of an end-user network's micronets would involve a private, flexible, arrangement between an end-user network and the MM company it hires to continually probe the network's reachability via its two or more ETRs. This means the frequency and type of probing, and the decision-making algorithms, can be completely open-ended and subject to development and customisation - without any constraints or need for changes in the RFCs which define Ivip. With TTR Mobility, the mapping of the micronet which the MN uses would be controlled by the TTR Company, rather than the end-user or the MN itself. This modular separation of the detection and decision-making functions from the CES architecture is good engineering practice and ensures that the Ivip subsystem can be used flexibly, including for purposes not yet anticipated. Other CES techniques monolithically integrate the following functions into the core-edge separation architecture itself - primarily by specifying exactly how all ITRs must behave regarding: reachability testing to ETRs, or of networks through ETRs, or with ETRs reporting reachability of end-user networks to ITRs by some means; multihoming failure detection based on these; decisions about how to choose between ETRs to restore service; and how to implement TE. This would add greatly to the complexity of the system itself, make it harder to introduce new methods of testing reachability etc. and restrict all end-user networks to relying on the necessarily restricted set of Whittle Expires September 8, 2010 [Page 18] Internet-Draft Ivip Architecture March 2010 functions which can reasonably be built into all ITRs. 5.4. Simple ITRs and ETRs with little or no communication between them With encapsulation, the only time ITRs engage in two-way communication is when probing the Path MTU to the ETR, by using a special pair of packets which carry a larger traffic packet than has previously been successfully received by the ETR from this ITR. The ETR then responds to the ITR and the ITR acknowledges this. Apart from this, ITRs do not communicate with anything but their local query servers - directly with their local QSRs (Resolving Query Servers) or indirectly with these, via one or more levels of QSC caching query servers. ETRs do not communicate with any part of the Ivip system except for ITRs, and then only for this PMTUD management function. If MHF is used rather than encapsulation, there is no need for ITRs to communicate with ETRs - so ITRs only communicate with QSCs and QSRs - and ETRs do not communicate at all. Consequently ETRs and ITRs can be simple functions in existing routers or in standalone servers. The ITR function can also be implemented in the sending host (ITFH), though this is not advisable if the sending host is on a slow, unreliable, link such as a wireless link. ETRs must be on conventional global unicast addresses ("core" addresses) - not on SPI ("edge") addresses. ITRs can be on both kinds of address. Ivip may in the future include an option for an ITR or ITFH to set up a two-way persistent tunnel to its one or more local query servers, which would allow an ITR function to be behind one or more layers of NAT. This "tunnel" could be as simple as TCP or SCTP from the ITR, or ITFH, to each query server, with keepalive packets. It is important to make ITRs as simple as possible, in order that they may be inexpensive and therefore, if desired, more numerous - so as to reduce the load on each one. ETRs are simpler than ITRs, since they simply decapsulate packets with a comparison between outer and inner source addresses and do not look up or cache mapping information. Ivip with encapsulation uses simple IP-in-IP encapsulation. There is no special header and no other data piggybacked onto traffic packets. This minimises encapsulation overhead and reduces the complexity of both ITRs and ETRs. Other CES architectures use their own headers to carry extra information with each traffic packet, with that header behind a UDP header. These other architectures also require ITRs to determine reachability to multiple ETRs. Whittle Expires September 8, 2010 [Page 19] Internet-Draft Ivip Architecture March 2010 5.5. Maximise the flexibility with which ITRs and ETRs can be located Ivip ITRs can be located in the sending host, in the sending-host's end-user network (which may be an ISP network or an end-user network using either SPI or conventional PI space) or in the ISP network which the host's end-user network connects to the Net through. If there is no such ITR, the packet will enter the DFZ and be forwarded to the nearest (in BGP terms) DITR (Default ITR in the DFZ, previously known as OITRD for Open ITR in the DFZ). ETRs can be located in ISP networks with a link to each end-user network they serve. ETRs can also be located at the end-user network end of a link from an ISP, and so be physically located at the end- use site. In both cases, their address must be a conventional "core" global unicast address (usually from one of the ISP's prefixes) - not an SPI ("edge") address or behind NAT. 5.6. Mobility One of Ivip's goals is to support mass adoption of IP mobility, since this will surely a major facet of the future of Internet communications. It would make no sense to introduce one set of architectural changes to solve the routing scaling problem as it appears today, and then have to devise and introduce a second set to provide for billions of mobile devices. Ivip is a good basis for the TTR approach to mobility, and would be attractive to deploy for this reason alone. It is frequently assumed that in order for a CES architecture to support mobility, the Mobile Node (MN) must be its own ETR. LISP-MN makes this assumption. So does draft-jen-mapping-00 [I-D.jen-mapping] - a critique of which is [Critique of draft-jen-mapping-00]. TTR mobility does not involve mapping changes every time the MN gains a new physical address, since it continues to use the same one or more TTRs as its one or more ETRs. Mapping changes are needed when the MN uses a new TTR. This is desirable after the MN moves a large distance, such as 1000km or more, but it not absolutely needed. An MN can still work with a TTR which is on the other side of the world - albeit with longer latency and greater chance of packet loss. Although the TTR approach to mobility could be used with other CES architectures, Ivip is a better basis for TTR mobility than other CES architectures such as LISP. None of these other proposals provide a method of ITRs gaining updated mapping within a few seconds, as Ivip does. With Ivip's real-time mapping system, the Mobile Node (MN) can Whittle Expires September 8, 2010 [Page 20] Internet-Draft Ivip Architecture March 2010 begin using a new, nearby, TTR within seconds and, most importantly, within a few seconds no ITR will be tunneling packets to the previous, and now more distant, TTR. Therefore the MN can promptly end the tunnel to the previous TTR and use the new TTR exclusively. Without this real-time mapping, the MN would need to retain tunnels to one or more previous TTRs for as long as the mapping system takes to ensure no ITRs are tunneling packets to them. This might take 10 to 30 minutes or more for the non-Ivip CES architectures. TTR Mobility is not required to solve today's routing scaling problem. It may be regarded as separate to Ivip, because it could be used with other CES architectures. However, it is best to consider TTR Mobility as a natural extension of the basic Ivip architecture, which does not place any constraints on the basic architecture other than that its mapping system will need to scale to billions of (mostly mobile, handheld device) end-user networks. 5.7. Elimination of encapsulation and PMTUD problems When ITRs use encapsulation to tunnel traffic packets to ITRs, there are serious problems with Path MTU Discovery (PMTUD) for the sending host. If the packet with its encapsulation header is too long for the next hop link of a router between the ITR and ETR, then there needs to be a mechanism by which the sending host receives a valid ICMP Packet Too Big message, with an MTU value which will result in an encapsulated packet of the correct length. The PTB generated by the router in the tunnel path will not be suitable for the sending host. It is challenging to solve this problem securely and without unreasonable amounts of state in the ITR. Ivip's solution - ITR Probes Path MTU [PMTUD-Frag] - involves extra complexity and state in ITRs and to a lesser extent in ETRs. This, and the transmission overhead of the encapsulation header (particularly heavy with IPv6 VoIP packets) makes it desirable to either avoid encapsulation entirely, or to introduce Ivip with encapsulation, but in the long- term change to an alternative system which lacks these problems. Ivip has two techniques, known collectively as Modified Header Forwarding (MHF) which replace encapsulation as the ITR to ETR tunneling technique. They are: 1. ETR Address Forwarding (EAF) - for IPv4. [I-D.whittle-ivip-etr-addr-forw] Whittle Expires September 8, 2010 [Page 21] Internet-Draft Ivip Architecture March 2010 2. Prefix Label Forwarding (PLF) - for IPv6. [PLF for IPv6]. If Ivip is introduced with encapsulation, all ITRs and ETRs should be capable of supporting MHF. At some date in the future, the DFZ routers will be upgraded to support this, probably without any significant cost. Ideally, it would be possible to establish Ivip from the outset without encapsulation. This would save having to develop the more complex ITR and ETR functions required by encapsulation - especially the PMTUD functionality. It would also eliminate the need to design a transition arrangement. I have not been able to reliably determine what proportion of current DFZ routers have firmware-based FIBs. Any such router could be upgraded with a firmware update in order to support MHF. As the years pass, there is an increasing probability that that most or essentially all DFZ routers could be upgraded in this way, for very little cost. Initial deployment with MHF is a goal, with the alternative goal being eventual transition to MHF. For any near-term introduction of Ivip, such as to introduce TTR mobility services or simply to provide SPI space to non-mobile end- user networks, the organizations initiating these services will be unable to have all or perhaps any DFZ routers upgraded in time to start their services. Since these services, especially TTR mobility services, appear to be commercially attractive in the near-term, the most likely outcome is that Ivip will be introduced with encapsulation. If so, it is vital that all ITR and ETR software be updatable so that a future transition to MHF can be performed reliably and completely. MHF involves some restrictions on the location of ETRs. For IPv4, only 30 bits are available for specifying the ETR address. However, an alternative which I have not yet fully explored is to define a new protocol type with its own header to replace the IPv4 header. In the new header at least 31 bits could be found - and probably 32. If 32 could be found , then the following paragraph would become irrelevant. This 30 bit MHF ETR address forwarding arrangement is incompatible with the initially desirable arrangement where any "core" address can be used for an ETR. There is further work to do on this problem - but the solution is probably to avoid it with a new header format as noted above. If a large number of end-user networks established their ETRs on a variety of addresses, such as the IP addresses of their existing single PA address services, then it may not be possible to have them alter these addresses in time for the Whittle Expires September 8, 2010 [Page 22] Internet-Draft Ivip Architecture March 2010 transition to MHF. For instance, in an extreme case, four separate end-user networks may run four separate ETRs on four contiguous addresses 11.22.33.16, 11.22.33.17, 11.22.33.18 and 11.22.33.19. Yet the current 30 bit IPv4 MHF technique can only tunnel packets to addresses specified with 30 bit precision - which covers all four addresses. A workaround would be for a router at this ISP to perform a second lookup on the destination address of these tunnelled packets and to forward them to the correct service directly. 5.8. No requirement for new host functionality It is a primary goal not to require any new host functionality - in stacks or applications. However, as an option, the ITR function can be integrated into sending hosts when this is desired. Mobile hosts using the TTR Mobility approach will have a little extra functionality, which could be implemented in the stack or perhaps outside it, as a separate piece of software. The IP stack itself and all applications remain unchanged and communicate with all other hosts, mobile or not, using current IPv4 and IPv6 protocols and addressing. One reason for avoiding the need for new host functionality is to enable the system to be widely enough adopted to solve the routing scaling problem, given the constraints imposed by the need for voluntary adoption. [Constraints-Voluntary] Another more fundamental reason is to ensure there is no extra burden on hosts, which would be particularly a problem for hosts which are on slow, expensive and unreliable links. This includes hosts on 3G wireless links - and in the foreseeable future it is reasonable to expect this to be true of the majority of hosts. While many people are attracted to the idea of hosts doing more, and leaving the network to be simple, there are objections to this. I intend to write these up as an ID, but for now they are on a web-page and in RRG discussions. [Host-Responsibilities] See also http://www.ietf.org/mail-archive/web/rrg/current/msg06162.html ("Recommendation suggestion from RW" 2010-03-04). In summary, it is highly undesirable for a new architecture to require all hosts to do more routing and addressing management than they currently do: just DNS lookups. The delays which are inherent any such arrangement are highly undesirable and the way these delays are worsened by one or both hosts being on high latency, unreliable, wireless links is particularly objectionable. Also, it is desirable not to enforce extra complexity or communication requirements on all hosts, since many of them will be constrained by battery power Whittle Expires September 8, 2010 [Page 23] Internet-Draft Ivip Architecture March 2010 limitations. 5.9. Full benefits to all adopters irrespective of level of adoption Ivip provides the full benefits of portability, multihoming and inbound TE to all end-user networks which adopt its SPI space. In order to do this, packets from hosts in networks which lack ITRs must be forwarded to an ITR and tunneled to the correct ETR. This is achieved by placing a number of ITRs in the DFZ. These are known as DITRs (Default ITRS in the DFZ) and were previously known as OITRDs (Open ITRs in the DFZ). When Ivip was first announced [Ivip-2007-06-15] these were named (erroneously): "Anycast ITRs in the DFZ". By placing DITRs widely around the Net, path lengths from any sending host to the ETR are minimised. LISP Proxy Tunnel Routers (PTRs) perform the same function. [I-D.lewis-lisp-interworking] For a scalable routing solution to be widely enough adopted, it must provide compelling benefits to all adaptors, including the earliest. Without DITRs, PTRs or their equivalent, only a small fraction of packets being sent to an end-user network would use the new system - those sent in networks with ITRs. Yet the goal is for all adopters to use the new form of addressing entirely, and so not to have to use the existing unscaleable "advertise PI prefixes in the DFZ" approach to portability, multihoming and TE. 5.10. Business incentives to deploy new infrastructure Some scalable routing proposals involve no additions to the network - just the adoption of new functionality in the end-user networks which use it. These are generally "Core-Edge Elimination" (CES) architectures. [C-E-Sep-Elim] No such proposal meets the constraints imposed by the need for widespread voluntary adoption. Firstly, most or all of them involve changes to host stacks and applications, which is impractical in the absence of compelling motivation for the authors of this software to make such major changes. Secondly all such proposals only provide portability, multihoming and TE benefits for packets sent from other networks which have adopted the scheme. Therefore, only if all networks adopted it would any one network be able to abandon its current routing and addressing arrangements. The benefits of scalable routing in a global sense, and for each adopter, the abandonment of unscaleable alternative routing and addressing arrangements, are only achieved after full (or almost full) adoption Whittle Expires September 8, 2010 [Page 24] Internet-Draft Ivip Architecture March 2010 by all networks. Yet there is insufficient direct incentive for early adopters for even a fraction of networks to adopt it. A CES architecture with DITRs, PTRs or some equivalent functionality provides full benefits to all adopters, and so is capable of being widely enough adopted to solve the routing scaling problem. Scalable routing benefits accrue in direct proportion to the number of adopting networks. The problem can be substantially solved by widespread adoption. Complete adoption is desirable, but not at all required. CES architectures do not require any changes to hosts - to stacks or applications. They do however involve the creation of at least two items of infrastructure which are typically global in reach, before any end-user network can use the system. Before DRTM, Ivip required a single coordinated global mapping distribution system, though the DITR systems could be operated by or for particular MABOCs (MAB Operating Companies) and need not cope with all the MABS in the Ivip system. Now, with DRTM, there is no need for a single global mapping distribution system. There will be multiple such systems, each handling a subset of the MABs. Ivip's technical structure lends itself to business models in which those who construct and run these two types of infrastructure can do so on a potentially profitable basis, by charging end-user networks according to the use they make of the mapping system and of the DITRs. The DRTM arrangements [I-D.whittle-ivip-drtm] involve MABOCs (MAB Operating Companies) or TTR mobility companies establishing and running (or contracting other organizations to establish and run) multiple DITR-sites. At each DITR-site there are DITRs and QSAs supporting the subset of MABs which are run by the one or more MABOCs the DITR-site is run for. Please see the DRTM ID for more information on how this system can develop without direct investment by ISPs, with MABOCs taking the initiative and making the investment in reaching out with DITR-sites to sending hosts in networks without ITRs, and with QSAs at those sites to help nearby ISPs run their own ITRs and QSRs. 5.11. Maintenance of existing levels of security and robustness All scalable routing schemes complexify the Internet - so it is unlikely that the goals of not degrading security and robustness to any degree can be fully realized. Only once Ivip is fully designed and carefully analysed can there be a realistic estimation of the security and robustness problems it will entail. It is a goal of Ivip to minimise and ideally to eliminate any such Whittle Expires September 8, 2010 [Page 25] Internet-Draft Ivip Architecture March 2010 degradation. Ivip's approach to handling the PMTUD problems inherent in encapsulation is intended to be secure against attacks - such as from spoofed ICMP Packet Too Big messages. Ivip is the only CES architecture to provide an inexpensive method of ETRs enforcing the source address filtering ISPs may impose on packets arriving at their Border Routers (BRs). Such filtering is imposed to prevent outside attackers spoofing the address of any host inside the ISP's network - and includes dropping packets with private (RFC 1918) source addresses. This is achieved by the simple arrangement of the ITR using the sending host's address as the outer header source address in all the encapsulated packets in the tunnel to the ETR. ETRs simply compare the inner source address with the outer, and drop any decapsulated packets where the two differ. (With encapsulation, when the ITR occasionally probes the PMTU to an ETR, it sends an additional packet with the source address being that of the ITR, but this does not alter the ETR's ability to enforce BR source address filtering.) This also works well with packets tunneled from ITRs inside the ISP network. Please see the section "ETR support for ISP border router source address filtering" in "Recommendation suggestion from RW" (http://www.ietf.org/mail-archive/web/rrg/current/msg06162.html or any later version of this) for a discussion of why it appears to be impossible for LISP ETRs to enforce this BR source address filtering. This approach - of the ETR dropping inner packets whose source address does not match the source address in the outer header - is only for encapsulation. When MHF is used, there is no need for ETRs to perform any such task, since the original packet is sent across the DFZ, with the sending host's source address in the IP header - so BR filtering occurs normally and the ETR never receives a packet which violates these filtering rules. 5.12. Avoiding the need for any one server to store or receive the complete mapping database With DRTM, QSAs store the complete mapping database for one or typically many MABs, and so require real-time feeds of mapping updates for those MABs. At boot time, they need to be able to download snapshots of the databases and bring that information up-to- date with the updates sent since the snapshot was made, before the database can be used to answer mapping queries. The same procedure would be executed if the QSA ever lost sync with the feed of mapping updates. Whittle Expires September 8, 2010 [Page 26] Internet-Draft Ivip Architecture March 2010 However, there is no requirement that any one QSA handle all the MABs. There is no prohibition of this - for instance if a DITR-site handles every MAB in the Ivip system, this will be perfectly allowable. Its just that the system is intended to work with multiple sets of DITR-sites, with the DITR-sites of each set handling a subset of the MABs. To whatever extent there are scaling limits to the number of micronets a DITR-site and its one or more DITRs and QSAs can handle, this does not pose a problem for the scaling of the entire Ivip system, since the total load can be handled by multiple such DITR-sites. QSRs can handle many sets of DITR sites - so there is no obvious limit to the scaling of the entire system. Before DRTM, each ISP with ITRs had to install two or more "QSDs" (full database query servers - the term is no longer part of Ivip). These were full-database for all MABs and so required real-time feeds of all mapping updates for all MABs. This presented a scaling problem and an unfair burden on the ISP if its customers rarely or never sent packets to micronets for which a large number of updates were sent, or never sent packets to whole MABs which the QSD still had to store and receive updates for. (These statements about ISPs also apply to any end-user network with ITRs which chooses to install its own QSR, or previously QSD, rather than use those of its one or more ISPs.) With DRTM, QSDs are replaced by QSRs - caching Resolving Query Servers. So there is no need for ISPs to maintain a server which is full-database for any MAB. This greatly reduces scaling problems. It will remain an option for a QSR to be full-database for one or more MABs - and in principle for it to be full-database for all MABs, in which case it would function just like the now-obsolete QSD. However, AFAIK, there will be no need to do this - since caching-only QSRs should scale well and cope with the largest imaginable numbers of micronets. 5.13. Eliminating unfair burdens Prior to DRTM, Ivip had a "non-goal" of eliminating unfair burdens. This was because with full-database QSDs (as discussed above) it could not be ruled out that an ISP would face expenses running its one or more QSDs which in part depended on there being some large number of micronets, or large number of changes to micronets, which the ISP never gained any benefit from - because these did not affect packets sent by its customers. This unhappy situation is no longer a part of Ivip. Ivip's goal is to eliminate "unfair burdens", but no scalable routing system is likely to achieve this entirely. Whittle Expires September 8, 2010 [Page 27] Internet-Draft Ivip Architecture March 2010 An example of an "unfair burden" which remains with DRTM is that each QSR needs to automatically discover two or more typically "nearby" QSAs for every MAB in the Ivip system. Yet perhaps the QSR and the ITRs which depend upon it will never send packets to some of these MABs. This is unfair, but it is much less of a problem than before DRTM, where the QSD would need to store all the micronets of such MABs and receive all the updates to them as well. Ivip's goal is to minimise unfair burdens and to eliminate them where possible. It should be able to achieve a huge improvement over the problem which lies at the heart of today's routing scaling problem - the unfair burden imposed on all DFZ router operators by the addition of each PI prefix by any end-user network in the world which is able to obtain the space and advertise it in the DFZ. Whittle Expires September 8, 2010 [Page 28] Internet-Draft Ivip Architecture March 2010 6. Non-goals 6.1. Isolation between core and edge networks is not required At least one CES architecture - APT (which is no longer being developed) - appeared to have a goal of completely separating (really "isolating") core networks from edge networks. In this scenario, only ISPs would have core addresses and all end-user networks (or perhaps all end-user networks which needed portability, multihoming and TE) had edge addresses. Then, in theory, it would be possible to prevent any host in an edge network from sending packets to the core - which was supposed to provide some security benefits. Ivip has no such goal. For a discussion of my attempt to understand this aspect of APT, and how this may have affected the ways in which some people use and think about the term "Core-Edge Separation", please see: "CES & CEE: GLI-Split; GSE, Six/One Router; 2008 sep./ elim. paper (v3)" (http://www.ietf.org/mail-archive/web/rrg/current/msg06110.html 2010- 02-24, or any later version). 6.2. Full adoption not required Ivip does not rely for its benefits (improvements to routing scalability, or the benefits for end-user networks) on complete adoption of SPI (edge) space by all end-user networks, or by the subset of them which want or need portability, multihoming and TE. Ideally, for scalability, the only prefixes advertised in the DFZ would be those of ISPs (including those used to serve many end-user networks with PA space) and the relatively small number of prefixes which encompass the SPI space. "Relatively small" is in comparison to the very large number of micronets these prefixes contain and to the likewise very large numbers of end-user networks which are using this SPI space. The full benefits for end-user networks which adopt SPI space - portability, multihoming and TE - do not depend at all on how many other end-user networks adopt SPI space. The benefit of routing scalability depends on how many end-user networks which need or want portability, multihoming and TE actually do adopt SPI space, rather than the two undesirable alternatives of either not getting these benefits, or getting them by the unscaleable method of advertising conventional PI prefixes in the DFZ. In order to maximise routing scalability, the more end-user networks which adopt SPI space, the better. But there is no need or intention Whittle Expires September 8, 2010 [Page 29] Internet-Draft Ivip Architecture March 2010 to have them all adopt it. A satisfactory outcome for scalable routing would be for some or many of the end-user networks which currently advertise PI prefixes in the DFZ to continue doing so - and for the great majority of all other end-user networks which want or need portability, multihoming and TE to use SPI space instead. 6.3. Mapping changes need not be free of financial cost It appears that the designers of other CES architectures have a goal of mapping changes being free of financial cost. This is not a goal of Ivip. Ivip is the only CES architecture to contemplate or assume that mapping changes will be paid for - by the end-user network whose micronet of SPI space the mapping applies to. All other proposals avoid financial costs such as this. In the case of the global query server systems - LISP-CONS, LISP-ALT and TRRP there is no need for payment, since changing the mapping has no direct impact beyond the authoritative query server(s) in which the mapping is changed. (Unless there are provisions for sending mapping changes to particular ITRs which might need it, which may be a part of LISP.) Ivip's arrangement for charging end-users for each mapping change, and for each change to the way their SPI space is divided into micronets, is intended to achieve two outcomes. Firstly, the payment - which goes to the MABOC - helps the MABOC cover its costs of maintaining multiple DITR-sites, each with their QSA authoritative query servers. Each such change involves data transmission to these sites and may involve QSAs sending Cache Update commands to queriers (QSRs) to which mapping for the micronet has "recently" been sent in a map reply message, or in a Cache Update message. This is fully described in the DRTM ID. It is also vaguely possible that if there are really large numbers of updates, ISPs and other networks with ITRs and QSRs may object to handling all these Cache Updates without some payment by the MABOC from whose QSAs they are sent. So it is vaguely possible that MABOCs may need to use some of these fees to encourage ISPs to accept these Cache Updates. Such frequent updates are most likely to arise from end-user networks doing short-timescale inbound TE changes - and they will do this as long as the cost of the mapping changes is lower than the benefit they derive from the inbound TE, which may be substantial. Whittle Expires September 8, 2010 [Page 30] Internet-Draft Ivip Architecture March 2010 Secondly, this fee per mapping change inhibits end-user networks from making so many mapping changes unless they have a suitably strong reason to do so. This will lighten the load on the MABOC's systems, including especially the DITRs and QSAs it either runs, or pays another organization to run. The cost of changes should be low enough to be a trivial issue in the rare events of multihoming service restoration and portability to another ISP. The cost should also be low enough to make reasonably frequent changes for TE attractive, when it allows significantly better utilization of multiple links to ISPs. It should also be low enough to present no problems for TTR Mobility, whenever mapping changes due to the MN moving more than about 1000km. 6.4. No attempt to cope with partially reachable ETRs Ivip's use of a single ETR address in the mapping is different from the use of multiple ETR addresses in the mapping information of all other CES architectures. This gives rise to a potential benefit of those other schemes which is not a goal of Ivip. Ivip ITRs all over the Net tunnel packets which are addressed to any particular micronet to a single ETR at any one time. (This is ignoring perhaps a second or less when the mapping is changed, and some ITRs receive the Cache Update message from their QSC or QSR query server earlier than others.) It is up to the multihoming end- user network to ensure that the mapping changes in a manner which maximises the connectivity of its network during a multihoming service restoration event. For instance, an end-user network has two ISPs ISP-A and ISP-B, and can map its one or more micronets to either ETR-A or ETR-B. Whether the ETRs are in the ISP or at the end-user site is not important. ETR-A's connection to the rest of the Net is via ISP-A and ETR-B's is via ISP-B. In this example, only one micronet is considered, but the same principles apply with multiple micronets. When both ISPs and ETRs are working well - that is to say when the end-user network is reachable via both ETRs - the end-user network may have the mapping set to ETR-A. If an external monitoring company (contracted by the end-user network) detects that the end-user network is no longer reachable via ETR-A, then it will issue a mapping change so that the micronet is mapped to ETR-B instead. As long as ETR-B is connected to the end-user network and is reachable from any router in the DFZ, then this is a perfectly good outcome: full connectivity is restored within a few seconds of the mapping change being issued. Whittle Expires September 8, 2010 [Page 31] Internet-Draft Ivip Architecture March 2010 However, if ETR-B is unreachable from some subset of the DFZ routers (and therefore from a subset of sending hosts in end-user and ISP networks) AND this subset of DFZ routers can reach the end-user network via ETR-A, then Ivip cannot ensure complete connectivity, since the end-user network is not reachable to all hosts in all networks through just one ETR or the other. (Actually, practical connectivity only concerns the fraction of DFZ routers and other networks with hosts which are currently sending packets to this end- user network - but the ideal is that the end-user network is always reachable from all other networks.) Other CES architectures such as LISP have a potential advantage in this scenario, since it is possible that all the ITRs which are currently sending packets may be able to discern the reachability of the two ETRs (or, if LISP is ever able to do this: determine the reachability of the end-user network through the two ETRs) and adapt their tunneling by choosing an ETR which enables the packets to get to the end-user network. In this circumstance, the non-Ivip CES architectures would be able to restore full connectivity when Ivip could not. However, this set of circumstances - both ETRs being partially reachable and the patterns of reachability being complementary so from anywhere in the Net, at least one was reachable - is likely to be a transient state, since the DFZ routers will rapidly adapt their best-paths to restore full connectivity to both ISPs and their ETRs. Also, it cannot be assured or assumed that the non-Ivip ITRs would choose the reachable ETR fast enough to take advantage of such a situation. Nonetheless, it is possible that a non-Ivip ITR may be able to detect non-reachability of a particular ETR when the Ivip approach would not. This is because with Ivip, multihomed end-user networks will typically contract another company to continually probe the reachability of their network through their two or more ETRs - and that company will do so from a finite number of servers in particular parts of the Net. There may be an outage affecting ITRs which are handling packets addressed to this end-user network which does not affect the set of servers the multihoming monitoring company is using - so that company will not detect the problem affecting these traffic handling ITRs. In that case, the non-Ivip approach would be superior - if the non-Ivip ITR could detect the outage and correctly chose another ETR through which the end-user network was reachable. With Ivip, end-user networks will be able to choose between many Multihoming Monitoring (MM) companies and each company would have a range of options for how frequent the reachability probing occurs, how many servers in the DFZ are used to probe the path via each ETR Whittle Expires September 8, 2010 [Page 32] Internet-Draft Ivip Architecture March 2010 and how decisions should be made if there appears to be a reachability problem. A MM company with probing servers scattered widely around the Net should be able to detect most reachability problems experienced by in any part of the DFZ, but it can't necessarily detect every one. How the MM company decides which outages to respond to, with a mapping change, is a matter for the company and the end-user network to decide. Ivip's external, user-supplied, detection of reachability problems and creation of mapping changes can be the subject of ongoing innovation and choice, with the intention that it be more effective at restoring full connectivity than the individual, isolated, efforts of non-Ivip ITRs - which have a difficult task reliably and inexpensively testing reachability of the end-user network via various ETRs. This is particularly the case if tens or hundreds of thousands of ITRs are tunneling to one ETR. Such non-Ivip ITRs may not actually probe reachability of ETRs with ping or the like, but rely on ICMP messages due to traffic packets not reaching the ETR. A difficulty with this (again for non-Ivip ETRs) is that ICMP messages may be lost or may not always be generated if there is an outage. Furthermore, it would be costly for these ITRs to be able to securely distinguish genuine ICMP messages from spoofed ICMP messages. 6.5. No attempt to mix IPv4 and IPv6 Ivip for IPv4 is intended to be a free-standing system completely independent of Ivip for IPv6. An IPv4 ITR could be implemented in the same server or router as an IPv6 ITR - just as ITR, ETR and query server functions could be performed in the one device. Likewise, the DITR-site systems of DITRs, QSAs and the mapping distribution systems inside each system or DITR-sites for IPv4 and IPv6 are intended to be separate and independent - but there's nothing to prevent one server being used for both the IPv4 and IPv6 systems. 6.6. Not Locator - Identifier Separation There is considerable terminological inexactitude regarding the use of the term "Loc/ID Separation". True Locator - Identifier separation involves hosts handling packets using two objects of different types, usually called Locator and Identifier, which therefore are in different namespaces. The Locator is usually regarded as an "address" but the Identifier is not. If both types of object are numeric and a Locator and an Identifier were numerically identical they would refer to different things because this numeric value has different meanings in each namespace. Whittle Expires September 8, 2010 [Page 33] Internet-Draft Ivip Architecture March 2010 Further discussion of the meaning of "namespace" is at: [Namespace] . HIP and ILNP [I-D.rja-ilnp-intro] are examples of Locator / Identifier Separation. LISP (Locator/Identifier Separation Protocol), Ivip, APT, TRRP and TIDR are not. An architecture which uses FQDNs as Identifiers and IP addresses (always PI, to ensure scalability) as Locators is also an example of true Loc/ID separation - for instance Name-Based Sockets [Vogt-2009]. LISP, Ivip and other CES architectures do not present hosts with separate Locator and Identifier addresses. The host sees only IP addresses, which perform both functions simultaneously - just as they do without Core-Edge Separation. ITRs are the only devices which treat packets differently if their destination address is in the "edge" subset of the global unicast address range. The full arguments about why Core-Edge Separation cannot correctly be construed as "Locator / Identifier Separation" are at: [loc-id-sep-vs-ces]. For further discussion and why LISP is misnamed, please see the following RRG messages from early 2010: msg05864, msg05865, msg06110 and msg06190. Whittle Expires September 8, 2010 [Page 34] Internet-Draft Ivip Architecture March 2010 7. Architectural Choices 7.1. Core-Edge Separation rather than Elimination 7.1.1. Core-Edge Elimination (CEE) architectures Core-Edge Elimination (CEE) involves hosts dealing with two kinds of entity for dealing with other hosts and to write into packet headers in order that they will get to their desired destination: Identifiers and Locators. The simplest adaptation of existing protocols is to retain IP addresses as Locator addresses and develop a separate namespace for the Identifier addresses. Some CEE architectures only modify the stack of each host, and use unmodified IPv6 applications. Other require modified stacks and applications. Each host retains its one or more Identifiers, no matter which one or more Locator addresses it is using. The Locator addresses are global unicast addresses which are supplied by ISPs as PI space. The simplest form of multihomed end-user network would gain a PI prefix from each of its ISPs and each of its hosts would use one address from each prefix as a Locator address. Each such prefix is part of a larger (in terms of number of addresses - shorter in terms of prefix length) prefix the ISP advertises in the DFZ. The ISP can split one such advertised prefix into many smaller (longer) prefixes for multiple end-user networks. This solves the routing scaling problem because the total number of large (short) prefixes advertised by all ISPs is scalable, whereas - if not for the CEE architecture - the number of PI prefixes advertised in the DFZ by multihoming end-user networks would be an unacceptable burden on all DFZ routers and on the entire DFZ BGP control plane. Applications connect to other hosts solely in terms of their Identifier addresses. It is the task of each host's stack (or perhaps its applications) to adapt to changes in other hosts' Locators, and to inform other hosts which need to know about this host's changed Locators. The Identifier may be numeric or have some other form, and there is typically a DNS mapping from FQDNs to one or more Identifier addresses, just as there are to IP addresses today. Some key points about Core-Edge Elimination architectures include: 1. Identifiers are from a completely different namespace than Locators. If both are numeric, and a Locator is numerically equal to an Identifier, there can be no confusion about the separate entities each refers to, since the Identifier is interpreted in a different namespace from that used for Whittle Expires September 8, 2010 [Page 35] Internet-Draft Ivip Architecture March 2010 Locators. Therefore, if IP addresses are used as Locators, IP addresses cannot be used as Identifiers. 2. Host stacks are responsible for choosing which of a correspondent host's Locators to send a packet to. This work is not done by network elements, such as routers. (However some CEE architectures may have routers alter part or all of the outgoing destination address, or perhaps source address, to exert-network centric control over traffic flows.) 3. While there is typically a global, decentralised mapping system by which hosts can use another host's Identifier (perhaps in combination with one of its Locators) to look up that host's complete set of one or more Locators, the network itself remains simple and hosts take on more responsibilities than they have with existing IP protocols. This is regarded as a virtue by many people, and represents an extension of TCP/IPs "dumb network, smart end-points" approach, especially when compared to the telephone network. 4. Since applications need to work with a different kind of address element than an IP address for establishing and maintaining communications with other hosts, the host stack, its API and applications themselves need to be substantially rewritten in order to be able to work with a CEE architecture - unless the system supports unmodified applications in some way. 5. While it may be possible to slowly introduce such an architecture, the benefits of portability, multihoming and TE only apply to packets sent between hosts using the new system - so substantial benefits to adopters only occur when all, or essentially all, hosts have been upgraded to the new system. 6. CEE architectures are subject to the critique that the extra management packets which hosts must send and receive as part of the new system is likely to create extra costs, delays and/or unreliability compared to current IP techniques. 7. This critique can be extended to argue that mobile hosts, due to their typically slow, not-necessarily reliable and potentially costly wireless links are especially impacted by these new responsibilities. 8. Core-Edge Elimination architectures typically do not apply to IPv4 and so are based on IPv6 or on entirely new arrangements. If CEE was used for IPv4, it would not be practical due to the inherent inefficiency of its use of global unicast address Whittle Expires September 8, 2010 [Page 36] Internet-Draft Ivip Architecture March 2010 space. In IPv4, any end-user network which needs a /24 of address space for its hosts would require a /24 from each of its upstream ISPs. So all multihomed end-user networks would consume at least twice the space they need - which is not practical with IPv4's address shortage. Points 4 and 5 constitute insurmountable barriers to the adoption of CEE architectures, since adoption must be very widespread, within a period of years, rather than decades, and since adoption must occur on a voluntary basis. [Constraints-Voluntary] Point 6 is an argument that while CEE architectures are theoretically elegant and simple, the facts of delay and loss of packets across global query server systems such as DNS - or whatever mapping system is used to securely determine the full set of Locators which can be used for a host with a given Identity - will contribute to delays in sending application packets. (All CEE architectures to date involve global query server systems with just one or a few authoritative query servers. None involve "nearby" or "local" authoritative query servers, which is the only way to avoid excessive delays and risks of packet loss.) Also, if the two hosts have to exchange management packets with each other, for authentication purposes, before any application packets can be sent, then this will slow down the establishment of communications - especially if the hosts are far apart, on high latency links or if packets are lost. Point 7 implies that in order to create a network which performs best, given the vagaries of slow and unreliable last-mile links, all hosts should not have to perform these additional Routing and Addressing management functions - that such functions be handled by better-connected devices, such as routers in ISPs' data-centers. [Host-Responsibilities] The only existing routing scaling problem is in the IPv4 Internet. In early 2010 the IPv4 DFZ has about 300k prefixes with a doubling time of about 4.5 years. The IPv6 DFZ has about 855 prefixes - 1/350th the IPv4 number. Even if IPv6 prefix numbers had a doubling time of 1.0 years, it would be mid 2018 before the number reached current IPv4 levels - which are not yet unworkable. IPv6 adoption rates have consistently disappointed IETF expectations. Despite the run-out of unallocated IPv4 space, there is no sign yet that large numbers of existing users can have their Internet needs adequately served via IPv6 addresses alone. For the reasons described in points 4 to 8, Ivip instead adopts a Core-Edge Separation approach. Whittle Expires September 8, 2010 [Page 37] Internet-Draft Ivip Architecture March 2010 7.1.2. Core-Edge Separation (CES) architectures Ivip uses a Core-Edge Separation (CES) Architecture. CES does not involve the creation of new namespaces and does not require any changes to host stacks or applications. A subset of the global unicast address space is converted to a new type of address which, in Ivip, is known as Scalable PI (SPI) space. The addresses which remain once this new, scalable, "edge" subset of the global unicast space is separated out is known as "core" address space. (In LISP, the "edge" subset is known as EID (Endpoint Identifier) and the remainder is known as RLOC (Routing Locator). However it is a mistake to think of these as being "Identifiers" and "Locators" or to think that LISP has anything to do with the Locator / Identifier Separation naming model.) This subset will consist of a growing number of prefixes, each of which is known as a MAB (Mapped Address Block). Each MAB is advertised in the DFZ by as many DITRs as are at DITR-sites which support this MAB. (The QSAs at those sites are also authoritative query servers for the MABs the site supports.) Within each MAB, the SPI space can be divided up amongst many (thousands to potentially millions) of separate end-user networks. If a network gains more than one basic unit of address space - an IPv4 address or an IPv6 /64 prefix - it can divide this space into multiple separately mapped "micronets". As more and more space is converted for use as SPI space, this "edge" space will grow to become a significant fraction of the total global unicast space. There must always be some conventional, "core", non- SPI, space, since ETRs must be located on such addresses. There are many uses of space within ISPs which do not need to be on SPI space - including the large numbers of IPv4 addresses, or in the future IPv6 /64s, which are used for individual home and SOHO customers. Each such customer gets what is effectively a small (long) prefix of PI space, which is suitable for their purposes because they do not want or need portability, multihoming or TE. As noted in the non-goals section, Ivip does not require or aim for complete conversion of all end-user networks to SPI space. Many will be happy with existing PI arrangements, and some larger existing end- user networks with their own (unscaleable) PA prefixes will probably retain their current arrangements. Nonetheless, SPI space is intended to be attractive to all end-user networks, including the largest corporations, universities and government departments. Whittle Expires September 8, 2010 [Page 38] Internet-Draft Ivip Architecture March 2010 CES involves the progressive repurposing of existing address space. It does not involve the creation of any separate namespaces. "Separation" in "Locator/Identifier Separation" means separate namespaces. Only CEE architectures implement "Locator / Identifier Separation". CES can be introduced gradually, and with DITRs (or their LISP equivalent - PTRs) the benefits of portability, multihoming and TE can be supported for all packets sent to the adopting end-user network. Therefore 100% of traffic receives these benefits, in contrast to CEE architectures where only the subset of traffic originating from other upgraded networks has these benefits. Assuming a CES architecture does not significantly reduce performance, robustness or security - and if it provides significant and immediate benefits to all adopters - then it meets the constraints due to the need for widespread voluntary adoption. [Constraints-Voluntary] All CES architectures I am aware of do not require hosts to perform additional work to manage routing and addressing. So no CES architecture is subject to the critique which applies to CEE architectures, particularly with reference to mobile hosts: [Host-Responsibilities]. The historical roots of Core-Edge Separation architectures can be found in the mid-1990s - Steve Deering's "Map & Encap" for IPv4 [Deering-1996], Robert Hinden's "New Scheme for Internet Routing and Addressing (ENCAPS) for IPNG" (RFC 1955) and the 1992 crocker-ip- encaps-01.txt. 7.2. Nearby authoritative query servers Probably the greatest challenge for a CES architecture is how to ensure ITRs can securely, reliably and rapidly obtain the mapping they need in order to be able to decide which ETR to tunnel a packet to. There are four basic approaches to this problem: 1. The complete global set of mapping changes is sent to each ITR, which maintains an up-to-date copy of the full mapping database. 2. Local full-database query servers are located in ISP networks and potentially in end-user networks in which ITRs are based. The complete global set of mapping changes is sent to each such query server, which maintains an up-to-date copy of the Whittle Expires September 8, 2010 [Page 39] Internet-Draft Ivip Architecture March 2010 full mapping database. ITRs query one or more of these and so obtain mapping quickly and reliably. 3. ITRs in an ISP network (or in an end-user network) send queries to local caching query servers - directly to a QSR or indirectly via one or more levels of QSC. Both these types of server are caching query servers and are "local" in that they are in the same ISP network, or if the ITR is in an end-user network are either in that end-user network or in the networks of its one or more ISPs. QSRs are the interface between the ITRs and the authoritative query servers which are not local - but which are typically "nearby". (See following text of a definition of "nearby".) 4. No site or device stores a complete copy of the global mapping database. Instead, there is a global network by which ITRs can send query to the authoritative query server for the particular micronet of addresses which match the destination address of the packet the ITR needs to tunnel. The only architecture to propose option 1 was LISP-NERD. This is widely regarded as scaling poorly with large numbers of end-user networks. LISP-NERD was to be retired, but a new version 07 ID appeared in early January 2010. [I-D.lear-lisp-nerd] APT used option 2. Ivip before DRTM (that is, before March 2010) also used option 2 - the local full database query servers were called QSDs. In APT, they were also called Default Mappers, and also handle the encapsulation of some packets. Ivip with DRTM uses option 3. The definition of "nearby" follows shortly. All other CES architectures to date use option 4. The most prominent examples are LISP-CONS [I-D.meyer-lisp-cons], LISP-ALT and TRRP. In option 3, "nearby" means something like within a few thousand km. In fibre, 200km involves approximately 1ms delay. So if the authoritative query server is 2000km away, the propagation delay in SiO2 sets a lower bound to the response time of 20ms. It is assumed that if the ITR buffers any packets it has no mapping for but gets the mapping within some time like 50ms or perhaps 100ms, then this constitutes an insignificant delay in the establishment of initial communications for all applications and human users. Therefore, "nearby" means close enough not to involve significant delay or risk of packet loss. "Typically nearby" means that except for unusual error conditions - assuming MABOCs are looking after the interests of their SPI-leasing customers well, by placing multiple DITR-sites with Whittle Expires September 8, 2010 [Page 40] Internet-Draft Ivip Architecture March 2010 their QSAs in widely spread locations around the Net - that ITRs will usually be able to send packets within 50ms or so, which is assumed to be an insignificant delay. QSC, QSR and QSAs will all have response times, but it is reasonable to assume these will normally be a few ms, considering the enormous four-core 3GHz clock CPU power which inexpensive COTS servers now possess. The global query server network approach has obvious advantages in terms of there being no hardware-imposed limit to the number of query servers or end-user networks which can be supported. Furthermore, changes to mapping impose no direct burden on any other devices - whereas for option 1 or 2, information must be sent to potentially hundreds of thousands of devices all around the world. However, global query server systems pose apparently insoluble problems of delay and potential unreliability - due the delays and risk of packet losses which are inherent in their global nature. Furthermore it seems to be impossible to make these systems scale to the very large numbers of EIDs required for ubiquitous mobile adoption. [LISP-ALT-Critique] Typically "nearby" full-database query servers is the clear choice for Ivip because ITRs will normally not delay any packets to a significant degree and because this system avoids the avoid scaling problems which arise from any server being required to store the full mapping database of all MABs, and the need for a single, coordinated, mapping distribution system to drive these servers. 7.3. Real-time mapping distribution By getting mapping changes to all ITRs which need it (all ITRs handling packets addressed to the micronet whose mapping just changed) in real-time - within a few seconds at most - Ivip achieves several major benefits. Firstly, the mapping information can be more compact, since only a single ETR address is needed. Secondly, ITRs can be much less complex, and do not need to do any reachability testing. Thirdly, the real time control of all ITRs which is given to end-user networks modularly externalises the reachability, multihoming service restoration and TE decision making systems from the CES architecture itself. 7.4. SPI address management Traditional IP techniques divide address space into binary boundary prefixes. Ivip uses traditional prefixes for the largest unit of SPI space - the "Mapped Address Block" (MAB). The smaller divisions of Whittle Expires September 8, 2010 [Page 41] Internet-Draft Ivip Architecture March 2010 this do not use prefixes or binary boundaries. The units of dividing SPI space are IPv4 addresses and IPv6/64s. A MAB is a prefix of address space which is devoted to use as SPI space. The single MAB is advertised in the DFZ, by all the DITRs at DITR-sites which support this MAB. These DITRs attract packets addressed to any address in the MAB. (It would also be possible to load share the MAB between multiple DITRs, each advertising a segment of it, but in general complete MABs will be advertised.) For instance, an IPv4 MAB may be 11.22.0.0/16. A MAB might have previously been conventional PI space of an end-user network, and may now be used exclusively by this end-user network. In this case, it will presumably be used to serve the needs of many sites within this network, so achieving routing scaling by removing the need to advertise each such smaller (longer) prefix in the DFZ. In this case, the end-user network is the MABOC of this MAB, and it does not lease any of the space to any other organizations. Most MABs will be operated by MABOCs which are specialised companies - perhaps ISPs but not necessarily. The MABOC typically acquires rights to multiple prefixes of global unicast space, advertises each of them in a global system of DITRs and then leases out smaller portions of the MABs, on an annual basis, to a large number of end- user networks. Each end-user network leases a section of the MAB - a User Address Block (UAB). One end-user network might lease multiple non- contiguous UABs in the one MAB, and multiple UABs in multiple MABs. For simplicity, the following discussion assume they rent a single UAB, such as: 11.22.33.84 to 11.22.33.95 inclusive. This is an 18 IP address UAB. UABs could be as small as a single IPv4 address or IPv6 /64 or could be very large, including as large as the MAB itself. The end-user network which rents this UAB is responsible for generating mapping changes to suit its needs - and for multihoming would typically hire a Multihoming Monitoring (MM) company and give them the credentials required to control the mapping via whatever mechanism the MABOC provides. The end-user network can split their UAB up as they wish into typically smaller sections, known as "micronets". (Bill Herrin first used this term in TRRP.) A micronet is a contiguous set of any number of IPv4 addresses or IPv6 /64s which fit within the one UAB. This 18 IP address UAB could be used as a single 18 IP address micronet, or it could be split in any way - such as into as many as 18 single IP address micronets. Whittle Expires September 8, 2010 [Page 42] Internet-Draft Ivip Architecture March 2010 Each micronet is covered by a single Ivip mapping - it is mapped to a single ETR address. MABs and micronets are important to ITRs and most of the mapping system. UABs are not needed for these, but are an administrative construct of SPI space which an end-user network is authorised to change the mapping for. The MABOC would provide a method by which the end-user network, or some other company it authorises, can change the mapping and the division of the UAB into micronets quickly and securely. This would involve the end-user network having complete control, but being able to give a username and password to another party such as the MM (Multihoming Monitoring) company, by the MM company could control the mapping of some or all of the end-user's UAB space. The technical and administrative arrangements for this are not described at present, but as the Ivip system comes closer to being standardized, it would be desirable to provide a standard protocol or interface by which end-user networks or their appointees could issue mapping changes, rearrange the division of UABs into micronets etc. Also, it would be desirable to have a standardised way that an end- user network could allow its appointee to control the mapping for individual micronets within its UAB. If this was universally adopted by all MABOCs, then multihoming monitoring systems would only need to work with this one system for controlling the mapping of micronets. For each mapping change and each change to the division of the UAB into micronets, the end-user network would typically incur a fee from the MAB company. The MAB company would charge fees for leasing the UAB space, and for the load placed on the DITRs which cover this MAB. The MAB company may run its own DITRs - and their associated QSAs - or may contract this out to another company which specialises in this service. It will be an important part of the MAB company's service to locate DITRs in all corners of the Net, to ensure good load sharing between them and to minimise the total path length from the sending host to whichever ETR the end-user network chooses to map their Micronet to. Likewise, the load-sharing between the QSAs at these sites, and the desirability of having QSAs "nearby" to the QSRs in ISP and other networks all over the world. This flexible integer-based approach to dividing SPI space is intended to maximise the efficiency with which it is can be used. Since a single physical site, such as a branch office, may be able to operate perfectly well on one or a few IPv4 addresses, or on a single IPv6 /64, a seemingly small UAB of 18 IPv4 addresses could be used to Whittle Expires September 8, 2010 [Page 43] Internet-Draft Ivip Architecture March 2010 serve the needs of as many branch offices. Each such site could be multihomed with two or more local ISPs. As fresh expanses of IPv4 space disappear, there will be continuing pressure to slice and dice the address space more finely so it can be used by more and more ISPs and end-user networks. However, the convention in the DFZ is not to propagate prefixes longer than /24. This 256 IP address granularity inherent in the current arrangement leads to considerable underutilization of space. With SPI address able to be sliced and diced freely in the smallest possible increments, a much greater utilization can be expected, in a scalable fashion, than is possible with current techniques. 7.5. IP in IP encapsulation When encapsulation is used, there is a simple IP-in-IP header. There is no need for ITRs to communicate with ETRs, except for the purpose of PMTUD management. So, when the ITR tunnels traffic packets ordinarily (in all cases except for the special Path MTU measurement protocol, which is only used rarely) there is no need for a UDP header to enclose a special header with extra information. Architectures with slow mapping distribution and which therefore require ITRs to choose between multiple ETRs typically require the ITRs and ETRs to communicate - but this is not needed for Ivip. 7.6. MHF initially or in the long term to avoid encapsulation and PMTUD problems Both the IPv4 and IPv6 headers have un-used bits which can be employed to direct the packet from ITR to ETR. This path is primarily across the DFZ but typically includes routers inside ISP and end-user networks. These routers need to be upgraded - and in the long-term this can be done without significant cost, simply by building the new capabilities into new routers and implementing it in firmware updates. 7.7. Outer header address is that of the sending host When encapsulation is used, it seems natural to use the ITR's address as the outer header's source address. This is consistent with traditional tunneling, and ensures the ITR gets any ICMP messages, including especially Packet Too Big (PTB) messages. There are two problems with this conventional approach, which is used by LISP and other CES architectures. Firstly, it is very expensive for the ITR to securely respond to PTB messages. Secondly, this approach means that any ISP BR filtering (dropping) of incoming packets according to their source address will not affect the packets Whittle Expires September 8, 2010 [Page 44] Internet-Draft Ivip Architecture March 2010 at the BRs and must be replicated in the ETR. For more than a few such blocked prefixes, this is extremely expensive too - and we want ETRs to be as simple as possible. The answer is to have the ITR use the sending host's source address in the outer header of the encapsulated packet. All ITRs will therefore generate packets with identical inner and outer source addresses. ISP BR filtering will drop the packets with source addresses matching any prefix inside the ISP's network and the ETR will never need to handle such packets. The ETR needs to enforce this in the case where an attacker sends a packet to the ETR, with an inner packet having a banned source address and the outer header having a source address which is allowable. This enforcement is achieved by the ETR performing simple logic on each decapsulated packet: If its source address does not match the outer header's source address, the packet is dropped. This arrangement of the outer source address being that of the sending host requires a novel approach to Path MTU Discovery management. 7.8. IPTM (ITR Probes Tunnel MTU) PMTUD management As long as encapsulation is used, there needs to be a method of informing sending hosts, via traditional RFC 1191 techniques of what length packet to send, so that once encapsulated, these packets may reach, but not exceed the MTU of the path between the ITR and ETR. This is true of any CES architecture which uses encapsulation. It is a complex topic and there is a solution, but it requires considerable thought and significant complexity in all ITR and ETR. PMTUD management occurs naturally via RFC 1191 mechanisms for DF=1 traffic packets if the router with the too-small MTU is between the sending host and the ITR, or between the ETR and the destination host. Without encapsulation - with MHF - packet lengths are not increased in the ITR to ETR "tunnel", and the modified routers in this path will convert a too-long packet back to its original IP header format, before passing it to the ICMP PTB algorithm. The difficult task is to make PMTUD work for the path between the ITR and ETR, where the original packet is encapsulated. I intend to write up IPTM in an ID. For now, the fullest description is on a web page. [PMTUD-Frag] Here is an overview of the process, which is much the same for IPv4 and IPv6. This system involves restrictions on the length of IPv4 DF=0 (fragmentable) packets which are accepted by this system. It is Whittle Expires September 8, 2010 [Page 45] Internet-Draft Ivip Architecture March 2010 reasonable to expect applications not to generate such packets, which place a serious burden on the network of they are too long. Google servers have been observed sending 1470 byte DF=0 packets. [DFZ-unfrag-1470] Such companies could presumably be persuaded to refrain from sending DF=0 packets altogether by the time a scalable routing solution is deployed. In the long-term, with EAF in place of encapsulation for IPv4, fragmentable packets addressed to SPI addresses will be dropped by all ITRs. A simple approach to PMTUD management would be to choose some packet length, marginally below 1500 bytes and require all ITRs to accept only packets which are the encapsulation overhead number of bytes shorter than this. Longer packets would cause the ITR to generate a PTB and the sending host would send a suitably shortened packet instead. This would be simple and perform reasonably well in today's DFZ, where the Path MTU can reasonably be assumed to be 1460 bytes or more. However, such a scheme would fail to take advantage of jumboframe sized MTUs whenever they appear in the DFZ. ITR to ETR MTUs of around 9k bytes are likely to become more and more prevalent as more routers adopt Gigabit Ethernet interfaces, which handle these large packets. The encapsulated packet has the sending host's source address. If such a packet reached a router with a next hop MTU which was longer than the packet, the router would transmit a PTB to the sending host. However, the sending host should ignore it, since the destination address in the enclosed packet headers will be that of the ETR, not of the destination host - and the rest of the enclosed headers will not match the packet it sent. Also, the MTU figure in the PTB is higher than the figure the sending host needs to adhere to. So the challenge is for the ITR to generate RFC 1191 PTBs when necessary, in an inexpensive and secure manner, whilst adapting to potentially higher or lower MTUs to the ETR due to routing path changes - while making full use of jumboframe paths if and when they exist. Security in this case means being immune to spoofed PTBs - a single one of which could greatly reduce the MTU for all traffic from the ITR to a given ETR for at least ten minutes. A careful decision will be made to assign a value such as 1200 bytes to a globally agreed constant MPMTU (Minimum Path MTU). Once set, this value must remain agreed to indefinitely. A BCP would require all DFZ routers, and all routers between the DFZ and any ITR or ETR (and of course the links between these) to handle packets of this length. Whittle Expires September 8, 2010 [Page 46] Internet-Draft Ivip Architecture March 2010 Any packets, which once encapsulated and so ENCAPS (Encapsulation overhead - 20 bytes for IPv4 and 40 for IPv6) bytes longer, have lengths less than or equal to MPTU are encapsulated without any extra processing. No PMTUD problems exist for these packets. For any packet longer than this, assuming the ITR has not yet probed the PMTU to its ETR, the ITR performs some special processing. The packet itself is split into two sections and two packets are sent to the ETR as part of the ITR's attempt to probe the MTU to this ETR. One packet uses UDP encapsulation to convey a nonce, some flags and most of the traffic packet - with the ITR's address in the outer header's source address. This long packet is exactly the same length as the original packet would be once encapsulated. If this exceeds the PMTU to the ETR, then the ITR will be sent a PTB. Assuming this is received, the ITR will determine a new MTU to send in the PTB to the sending host. This process will repeat until the sending host's packets, once encapsulated, no longer exceed the MTU of the path to the ETR. IPTM does not rely on these PTBs. The ETR is instructed, in a shorter packet to report to the ITR whether the long packet arrives or not - and the ETR repeats this report for a while until it is acknowledged. The long packet is accompanied by one or more copies of this shorter packet, which contains a matching nonce, flags and the remainder of the traffic packet. The shorter packet has the sending host's address in the outer header, so ISP BR source address filtering is still enforced. The effect is that as the sending host (or multiple sending hosts whose packets must be tunneled to the one ETR) tries longer and longer packets, the ITR narrows its "zone of uncertainty" (cue Hammond organ, with reverb and ghostly sounds . . .) about the true MTU to this ETR. If the traffic packets necessitate it, the ITR will exactly determine the MTU, and so be able to stop probing it for a while and send PTBs to sending hosts which generate packets which, once encapsulated, would be longer than this reliably determined MTU. Further elaborations are required for the ITR to adapt to changing conditions and discover longer or shorter MTUs. Without some kind of PMTUD system, CES architectures cannot use encapsulation. These techniques will require further design work and extensive testing, but are more secure and less expensive than the only other obvious alternative - using the ITR's address in outer headers and having the ITR maintain a large cache of details about recently sent "long" packets, in order that it can securely accept PTBs if they are too long. Whittle Expires September 8, 2010 [Page 47] Internet-Draft Ivip Architecture March 2010 8. Architectural Elements 8.1. ITRs 8.1.1. Types of ITR and their addresses The ITR function can be implemented in a traditional hardware-based router, in a COTS (Commercial Off The Shelf) server, or as a piece of software in a sending host. The functions are much the same, but an ITR in a sending host does not advertise anything in a routing system - it simply handles outgoing packets which are addressed to any MAB. If an ITR is built with software and a COTS server, it doesn't need to be a "router" in most ordinary respects. For instance it doesn't need multiple interfaces. It may have a single Gigabit Ethernet link and advertise MABs in the local routing system, forwarding its encapsulated packets to a router to be forwarded like any other packet. An ITR could be built into a DSL, HFC cable, fibre or WiMax / 3G router. However, it is probably best to do this only when the ITR function is on a reliable, fast, inexpensive link. Most wireless links are not like this and it would be better to let SPI packets flow out of the link, and be handled by ITRs in the ISP network, which have fast reliable paths to local query servers. An ordinary ITR (not in a sending host, and not a DITR in the DFZ) is a device within an ISP or end-user network which attracts packets addressed to SPI addresses. It may do this by advertising every MAB - so the only packets forwarded to it, other than those addressed to the DITR itself, are those addressed to SPI addresses. Alternatively, if the ITR is a true router (hardware or software) it my advertise the entire address space and so be forwarded all packets not addressed to prefixes advertised by local routers. Then, it would encapsulate packets which are addressed to SPI addresses and forward all other packets according to its ordinary router functions. The ITR's address - the address it uses for tunneling packets from, and which is used for communication with the ETR for PMTUD management - may be on conventional global unicast space or, if in an end-user network, on SPI space. This address is also used for communication with local query servers (QSCs or QSRs) and for receiving PTB messages. Here is a description of what happens when a sending host in an ISP network, such as a QSC or QSR, on the ISP's conventional address space, sends a packet to a host in an end-user network on an SPI address - in this case an ITR or ITFH. The packet will go to an ITR Whittle Expires September 8, 2010 [Page 48] Internet-Draft Ivip Architecture March 2010 in the ISP network (if the QSC or QSR doesn't have an ITFH installed) and then will be tunneled to the ETR for this end-user network. This ETR sends the packet to the SPI-addressed host, in this case an ITR or ITFH. When MHF is used, there is no PMTUD management, no interaction with ETRs and no trace of the ITR's address in any outgoing packets. However, the ITR still needs an address for communicating with local query servers. As just noted, this can be on conventional "core" space or "edge" (SPI) space. 8.1.2. DITRs - Default ITRs in the DFZ DITRs are "Default ITRs in the DFZ". This first use of "Default" is different from the use of "Default" in "Default-Free Zone". (This term looks nonsensical when expanded fully: "Default ITRs in the Default-Free Zone".) The initial "Default" means that this ITR acts as one of (typically) many other such ITRs, all of them outside ISP and end-user networks. These DITRs advertise MABs from many places in the DFZ and so form multiple destinations which are the "default" - what happens to the packet if nothing else happens, meaning the packet does not go into any other ITR before reaching the DFZ. In principle, a DITR could advertise every MAB, or be an otherwise normal DFZ router and encapsulate every packet which is forwarded to it which is addressed to an SPI address. However, there is a burden of work looking up mapping, encapsulating packets and on occasions handling the PMTUD management functions to ETRs, which involves sending PTBs to sending hosts. It is unlikely that anyone running a DFZ router would want their device to do more work, unless they are paid for it by the beneficiaries. The ultimate beneficiaries of DITRs are the end-user networks which the packets are addressed to - and these are the customers of the MABOCs who lease the space to them (except where the one end-user network runs a whole MAB for itself, and so is its own MABOC). The most likely arrangement for DITRs is that the MABOCs who lease SPI space to end-user networks will also run DITRs themselves or contract specialised companies to run DITRs all over the Net for them. In this scenario, a DITR would advertise only those MABs of the MABOCs who are paying the operator for this service. MABOCs would charge their SPI-renting end-user network companies for the traffic handled for their networks by DITRs, so DITRs in general would need to sample traffic reliably and generate reports in a form which would enable the MAB companies to bill their customers fairly. Only DITRs need this traffic sampling capability. Other ITRs would Whittle Expires September 8, 2010 [Page 49] Internet-Draft Ivip Architecture March 2010 have monitoring and management functions, but would not need to collect usage statistics for billing. Theoretically, DITRs could advertise all MABs and so handle packets addressed to every MAB. In practice, I expect DITRs will usually only handle packets addressed to specific MABs. Other ITRs, including those in sending hosts, will handle packets addressed to any MAB. Consequently, these non DITR ITRs all need a reliable method of downloading the latest set of MABs. They will do this as part of discovering and communicating with their one or more local query servers. The one or more QSRs they rely on will determine the current set of MABs by the DNS-based mechanism described in the Ivip- drtm ID. Changes to this set will also need to be propagated to all QSCs and ITRs in the local system, by a mechanism which is yet to be designed. DITRs may be implemented in hardware based routers, or in COTS servers. They are always located on conventional global unicast addresses - never on SPI addresses. DITRs are likely to be busy, so it makes sense to locate them in major datacenters or Internet exchanges, close to one or more full database query servers. DITRs advertise MABs to their neighbouring BGP routers, and have a default route to either one of these routers, or have the full set of DFZ routes with links to multiple neighbouring routers. So (unless they are implemented behind a suitable BGP router) DITRs are BGP routers and may or may not be "DFZ" routers, depending on how they forward their outgoing packets. 8.1.3. Modified Header Forwarding - MHF-only ITRs Ivip for IPv4 and for IPv6 separately may or may not begin with encapsulation. If it does, then all ITRs and ETRs will also be capable of transitioning in the future to using MHF. The MHF techniques are discussed in a later section, but involve much less processing than encapsulation. With MHF, there is no need for PMTUD management. 8.1.4. Encapsulation and PMTUD management When the ITR function is implemented in software - either inside a sending host, or in a COTS server, it will be relatively straightforward to write C code or the like to implement the functions of analysing the packet's destination address, deciding whether to encapsulate it or not, deciding which ETR address to encapsulate it to, and encapsulating it. Once encapsulated, the new packet is presented to the internal packet handling functions and forwarded normally. Whittle Expires September 8, 2010 [Page 50] Internet-Draft Ivip Architecture March 2010 This packet-handling code also needs to consider the length of the packet, with reference to a small set of variables it maintains for the ETR the packet will tunneled to. So the packet's destination address would firstly be used to find an ETR address. Generally, this would be found by reference to the ITR's cached mappings, but for initial packets in a new communication flow, the packet must be held for a few milliseconds or tens of milliseconds while the ITR retrieves the mapping information from its one or more local query servers. Once the destination ETR address is known, the length of the packet is considered. If it is less than some constant, it can be encapsulated and sent without any further processing. If it is longer than this constant, then the ITR needs to perform PMTUD management functions. In this case, the ITR establishes, or has already established, some variables for this ETR. These include an upper and a lower estimate of the MTU to this ETR. If these are different, then there is a "zone of uncertainty" about the MTU. If they are equal, then the ITR has already reliably established the MTU. If the packet length, plus the encapsulation overhead, exceeds the range of possible MTU values the ITR has previously determined for the path to this ETR, then the ITR will send part of the packet back to the sending host in an ICMP PTB message. If the encapsulated length would be less than the lower limit in the "zone of uncertainty" then the packet can be encapsulated without further processing. If the encapsulated length falls within the "zone of uncertainty", then the ITR emits two packets - a long one and a short one - and communicates with the ETR in a way which will usually raise the lower limit of this zone, or lower the upper limit. In the former case, the ITR is able to determine that the encapsulated length did not exceed the MTU and that the ETR received it correctly. The traffic packet's contents are mainly contained in the long packet, which has the same length as the traffic packet would have had if encapsulated. The remainder of the traffic packet is conveyed in a short packet, of which perhaps a few will be sent. This is non-trivial process, which involves the ETR in some work - but it only occurs for packets whose encapsulated length falls within the "zone of uncertainty". Except for rare error conditions, each such operation reduces the size of the "zone of uncertainty" - and typically the zone will be reduced to zero. Once this occurs, at least for the next 10 minutes or so, the ITR need not perform any such probing of the MTU. Every encapsulated packet which is to be sent to this ETR will be either shorter than the MTU, in which case it is encapsulated without any further work - or is longer, in which case a PTB is sent back to the sending host, with an MTU value such that the host will generate Whittle Expires September 8, 2010 [Page 51] Internet-Draft Ivip Architecture March 2010 packets to this destination host of a length which, when encapsulated, will equal this reliably determined MTU. This encapsulation and some kind of PMTUD management is required for any CES architecture which uses encapsulation. All other CES architectures use encapsulation exclusively. There is at least one other approach to PMTUD management which is probably more expensive to perform as securely as this one. The fact that this and other processes are explained in some detail in this Ivip ID and not in the IDs of other proposals does not mean that the other proposals, once developed to the point of proper operation, would be simpler than Ivip. The encapsulation itself is straightforward. The sending host's address is used for the outer source address and the ETR's is used for the outer destination address. For IPv4 packets, the Diffserv, TTL and other flags are copied to the outer header. For IPv6, Traffic Class and Hop Limit bits are also copied. 8.1.5. Mapping lookup and caching Apart from PMTUD management, looking up the mapping for an incoming packet is the most complex task that ITRs need to perform. This task is the same for encapsulation in both IPv4 and IPv4 and for the IPv4 approach to MHF: ETR Address Formatting (EAF). For the IPv6 MHF technique - Prefix Label Forwarding (PLF) - the mapping lookup is similar, but only part of the ETR's address is actually needed for writing 19 or 20 bits into the header. When encapsulation is used, for IPv4 or IPv6, the result of the mapping lookup is an IP address of the ETR, which will become the destination address of the outer header. The result of EAF is similar, and ETR address where the two least significant bits are zero. This will be written into the modified IPv4 header. The result of the PLF mapping will be a 19 or 20 bit value is written into the modified IPv6 header and which identifies one of 2^19 or one of 2^20 contiguous DFZ advertised prefixes, each of which is advertised by a different ISP site. These 20 bits do not uniquely identify an ETR if there are more than one at each ISP site, but they are sufficient for the packet to be forwarded across the DFZ to the nearest BR of that site, where a second mapping lookup may be performed on the destination address to determine which of multiple ETRs at that site the packet should be forwarded to. This following may appear somewhat complex, but it is a description of different approaches to handling ITR to ETR tunneling for both the IPv4 and IPv6 Internets. Ideally, encapsulation won't be necessary Whittle Expires September 8, 2010 [Page 52] Internet-Draft Ivip Architecture March 2010 to at all. At worst, it will be necessary until DFZ and other routers are upgraded to handle EAF or PLF modified header packets. The mapping lookup is driven entirely by the packet's destination address. Ivip does not attempt to send packets of differing types, service class or differing source address to different ETRs. (Nor do the other CES architectures.) After a packet arrives, and has been classified as being addressed to an SPI address (meaning it matches one of the MAB prefixes) the next step is to find out whether the ITR has any mapping cached for the packet's destination address. For IPv4 the full destination address is used. For IPv6, only the most significant 64 bits are used, since SPI space is divided on /64 boundaries. Busy ITRs may have tens or perhaps hundreds of thousands of mappings already cached. An ITR function in a sending host may have only a handful or a few thousand for a busy web-server. A carefully designed algorithm will be needed to find any existing mapping, or to determine that the destination address does not match any cached mapping. In the former case, the mapping consists of a starting address and ending address for the micronet which the destination address falls within - and a single ETR address. This ETR address (or set of PLF bits) is then applied to the packet - by writing it to the outer header when encapsulating, or by writing into the modified header for EAF or PLF. (PLF only uses 19 bits of the ETR address - just enough to distinguish between the 2^19 contiguous prefixes which are reserved for this system. The resulting packet is then ready to be forwarded like any other - according to its outer header, or according to the bits just written into its modified header. If no cached mapping is found, the ITR buffers the packet and sends a map query to a local query server - a QSC or a QSR. This includes a nonce which is used to secure the reply, and any later map update messages the query server sends if the mapping changes during the time the ITR caches it. The local query server sets the caching time on the mapping. This time may be locally configured and could be set differently for different replies by various algorithms in the query server to optimise its interactions with the ITRs, and to limit the number of mappings the ITR caches. (Further work: It may be desirable for each ITR to be able to communicate to its query server(s) the state of its cache and how close to any limits it is running, so the map replies can have their caching times adjusted downwards.) Whittle Expires September 8, 2010 [Page 53] Internet-Draft Ivip Architecture March 2010 In the future, caching times will be discussed fully in the Ivip-drtm ID. The ITR flushes from its cache any mappings whose cache times have expired. The cache includes the starting and ending address of the micronet, the ETR address and the nonce which was sent in the query which returned this mapping. The ITR can also be sent a Cache Update message to the effect of flushing the cached mapping for a given micronet - by a QSC or QSR which previously sent mapping for this micronet in a Map Reply message. This is needed if the micronet has been deleted from the system, such as due to the end-user network changing the way their UAB is divided into micronets. If the ITR receives a packet with a destination address which matches this micronet, then there will be no cached mapping and the ITR will request mapping - and so gain the mapping for whatever micronet this address is now within. At any time when a mapping is cached, the ITR may receive a Cache Update message from a QSR or QSC which previously sent it mapping for this micronet. The Cache Update message, like the Map Request query and the Map Reply message, will be a UDP packet. The Ivip-drtm ID has more information on these messages and their acknowledgement. The Cache Update will be secured by the nonce sent by this ITR in its original Map Request query which resulted in the QSC or QSR sending a Map Reply (also secured by that nonce) which specified the start and end address of this micronet, and its mapping (the ETR address). The most common update will be that this micronet is now mapped to a different ETR address. Another type of update is that this micronet include it being mapped to no ETR (an ETR address of zero) - in which case the ITR will drop subsequent matching packets. As noted above, the final kind of Cache Update is a command to flush the mapping cached for this micronet. This could be encoded via special flags, but it may be simpler, for instance with IPv4, to define a particular ETR address such as 0.0.0.1 as meaning the mapping should be flushed. None of these Cache Updates reset the caching time. So ITR's cached mappings will time out as usual, no matter how many Cache Updates have arrived to alter the ETR address stored in this mapping. If the Cache Update message from the query server reset the caching timeout process, then continued Cache Updates would keep a mapping in the ITR's cache for excessive periods - including if the ITR was not handling any packets for this micronet. In this way, ITRs receive all the updated mapping they need, within a fraction of a second of the changed mapping being received by the nearby QSA. Whittle Expires September 8, 2010 [Page 54] Internet-Draft Ivip Architecture March 2010 8.1.6. ITFH - ITR Function in Host An ITR function in a sending host performs either encapsulation and PMTUD management or MHF as described above. This function is only for packets generated in the host. ITFH should only be used on hosts which have fast, reliable, connections to two or more local query servers. If there are delays, or packet losses, then the extra management traffic between the ITR function and the local query servers may not function well enough to ensure there are no significant delays to traffic packets. In many settings, the software and hardware required to implement an ITR in the sending host will have zero incremental cost. RAM and CPU capacity is now extremely inexpensive. Hosts - such as desktop PCs and servers used in hosting farms and cloud systems - come bristling with multicore CPUs and gigabytes of RAM for the price of a good shirt. The host could be on a conventional global unicast address (PI or PA) or on an SPI address. If it is thought desirable to enable ITFHs in hosts behind NAT, then at least two additional measures would need to be taken. Firstly, if encapsulation was used, the PMTUD exchange with ETRs would need to work through the NAT - which it probably would. Secondly, the ITFH would need to set up and maintain a two- way tunnel to two or more local QSCs. I do not suggest that QSRs should have to maintain sessions with such ITFHs. TCP with a keepalive might do, but SCTP would probably be much better. Then, instead of UDP mapping queries, replies and updates, the same messages would be sent over SCTP. It is not out of the question to link all ITRs, QSCs and QSRs with SCTP, rather than use UDP packets, since the SCTP will ensure reliable delivery of messages, and so reduce the complexity of the code for sending receiving and acknowledging messages. 8.1.7. ITRs auto-discovering local query servers There is further work to do to enable ITRs to automatically discover the addresses of one or more local query servers - whichever two or more QSCs or QSRs the ITR or ITFH is supposed to send its Map Request queries to. This is not absolutely necessary, but would greatly ease the deployment of ITRs in ISP and end-user networks. The more ITRs there are, the less work each one has to do and so the greater the chance that they can be implemented with little cost in a COTS server, rather than an expensive hardware-based router. This principle applies especially to ITFHs. Likewise, it would be desirable for QSCs to be able to automatically discover the upstream QSCs or QSRs they should send their Map Request Whittle Expires September 8, 2010 [Page 55] Internet-Draft Ivip Architecture March 2010 queries to. 8.2. ETRs 8.2.1. In servers or dedicated routers The ETR function can be performed in a dedicated router or in a server with appropriate software. Whether the ETR function is performed in a server with one or more Ethernet ports, or a router with multiple ports of various kinds, depends on how the traffic packets are to be forwarded to the one or more end-user networks being served by this ETR. The methods of forwarding do not need to be part of the Ivip RFCs - just how ETRs handle the incoming packets, and for encapsulation, how they communicate with the ITR for PMTUD management purposes. In the TTR mobility system, the TTRs perform ETR functions. The link to each end-user network is a separate two-way tunnel, established by the Mobile Node (MN) to the TTR. 8.2.2. ETRs in ISP networks An ETR in an ISP network can, in principle, handle packets for many end-user networks - all from a single global unicast address. This has a scaling benefit for IPv4 by supporting a potentially large number of end-user networks, with potentially large numbers of SPI addresses, while requiring only a one of the ISP's IP addresses. (For IPv6, inefficiency of address use is not a concern.) 8.2.3. ETRs at the end-user network site A multihomed end-user network with two links to ISPs might have two ETRs - one for each link. Each ETR will have a stable conventional (non-SPI) global unicast address to receive encapsulated packets on. So each ISP needs to devote at least one of its addresses, or more likely four, for each such ETR. This saves the ISP from having to run an ETR for this customer - all the ISP provides is connectivity and this small amount of stable address space. There could be one physical ETR, with two links to the two ISPs, receiving encapsulated packets as above on the two addresses provided by the two links. This device would be a router of some kind, even if implemented on a server, since it would also be deciding which link to send outgoing packets on. Whittle Expires September 8, 2010 [Page 56] Internet-Draft Ivip Architecture March 2010 8.2.4. MHF ETR functionality - EAF and PLF If Ivip is introduced with encapsulation, its ITR and ETR functions will contain Modified Header Forwarding functionality ready for a future migration from encapsulation to MHF exclusively. The IPv4 MHF technique - ETR Address Forwarding (EAF) - is very similar to the encapsulation arrangement, so the same ETR could do both, from the same address. However, with EAF, the ETR address is specified with the most significant 30 bits, giving a granularity of 4 IP addresses. (But see previous discussion about how this could probably be redesigned to involve a new header type which would allow 31 or 32 bits to be used.) To avoid having to change ETR addresses when encapsulation is turned off, only one ETR should be located in each /30 prefix. The IPv6 approach to MHF - Prefix Label Forwarding (PLF) - is conceptually different from the encapsulation approach in which the packet is tunneled to an ETR at a single IPv6 address. The ITR uses the mapping to write 19 or 20 bits into the IPv6 header. Upgraded routers in the DFZ forward the packet to ISP BRs (Border Routers, facing other ISPs and transit networks) advertising one of 2^19 or 2^20 separate prefixes. While the mapping still specifies an exact 128 bit IP address for the ETR, before MHF can be turned on, all ETRs must be given addresses within the special set of DFZ-advertised prefixes which the MHF system can forward these packets to. On arrival at the BR, the packet itself contains no information of further use - it does not contain the ETR address, just 19 or 20 bits of the address bits which differentiate this contiguous set of prefixes. If there is only one ETR for each such prefix, then the BRs (or perhaps single BR) needs only to forward the packet to the ETR. Alternatively, the ETR function could be performed within the one or more BRs. However, if this prefix has multiple ETRs, then the BR needs to behave like an ITR and perform a second mapping lookup, using the destination address of the packet, to decide how to forward (or perhaps tunnel) the packet to the correct ETR. There are various techniques for doing this, including the ISP using the PLF bits again, interpreted according to its own arrangements by its internal routers, to forward the packet to some internal prefix (perhaps in ULA space) which leads to the correct ETR. I have not yet explored the various ways an ISP could use to get PLF-tunneled packets to the correct ETR, or how techniques and ETR placement arrangements for encapsulation can be made compatible with the PLF arrangements. With both EAF for IPv4 and PLF for IPv6, the work an ETR performs on each tunnelled packet is trivially simple: restore the altered bits Whittle Expires September 8, 2010 [Page 57] Internet-Draft Ivip Architecture March 2010 so the IP header has its standard form again, and forward the packet to the destination network. The ETR does not communicate with the ITR or with any other part of the Ivip system, since the ITRs and ETRs have no Ivip-specific PMTUD problems to solve. If there resulting packet is too long for the next hop, the existing IP stack of the server or router in which the ETR function is performed will implement conventional RFC 1191 PMTUD and generate a PTB to the sending host. 8.2.5. ETR functionality for encapsulation With encapsulation, ETRs receive IP-in-IP packets on a stable global unicast address. The ETR recognises all such packets and decapsulate them. If the outer header source address matches that of the inner packet, then the ETR forwards the packet to the end-user network. If the ETR handles multiple end-user networks, then it will have appropriate configuration or router functionality to forward the packet to the correct end-user network. For PMTUD management, some more complex functionality is required. When the ITR uses special techniques to send a traffic packet, in two parts, as a probe of PMTU to this ETR, it sends a long packet and one short one (or multiple copies of the short one) to the ETR's address. However, these are not IP-in-IP encapsulated. They are both UDP packets - the long one with the ITR's address as the source address, and the shorter one(s) with the sending host's address as the source address. If only the short packet arrives, then the long one was lost - probably due to it being longer than the PMTU from the ITR to this ETR. The ETR informs the ITR of this non-reception, and receives an acknowledgement of this. If both the long and short packets arrive, the ETR reconstructs the full traffic packet, forwards it to the end- user network, and informs the ITR that it has been received correctly. This involves significant complexity in the ETR, but does not involve storing state for more than a few seconds. Once the traffic packet has been decapsulated, if the forwarding step leads to the packet being deemed too long for the next-hop MTU, then the conventional IP stack will generate a PTB to the sending host and RFC 1191 PMTUD will proceed just as it would if there had been no ITR to ETR encapsulation. 8.3. QSRs - Resolving Query Servers Please see the Ivip-drtm ID for a description of QSRs. Whittle Expires September 8, 2010 [Page 58] Internet-Draft Ivip Architecture March 2010 8.4. QSCs - caching query servers A caching query server (QSC) is a relatively simple function, typically implemented as software in a server. The software for ITRs, QSCs, QSRs and QSAs would share some common components. A QSC receives and responds to Map Request queries from ITRs or other QSCs in the same manner as a QSR. The QSC sends Map Request queries to a QSC or QSR in just the same way as was described above for an ITR - and likewise receives Map Reply and Cache Update messages the upstream QSC or QSR as just described. There could be zero, one, two or in principle any number of QSCs between an ITR and the one or more QSDs. All these devices are typically in the same ISP network - or in an end-user network whose ITRs and QSCs use the QSCs and QSRs in the ISP network. So communication between them is very fast, reliable and inexpensive. Typically, there will be little or no packet loss, but the protocols will need to cope with any losses in a robust manner. If a querier sends out a Map Request query and does not get a reply within some quite short time, such as 100ms, then it should try sending the query (with a different nonce) to an alternative upstream query server. Further work: ITRs auto discovering query servers in general - and QSCs autodiscovering other QSCs and QSRs. Manual configuration of the tree-like structures of these devices should also be possible. If the mapping needs of one ITR were completely uncorrelated with the mapping needs of other ITRs served by the same QSR, then there would be little or no benefit in deploying intermediate QSCs. However, there is likely to be sufficient commonality between the mapping needs of tens or hundreds of ITRs and ITFHs to make QSCs a good investment in expanding the capacity of a single QSR to support more ITRs. If 20 ITRs send their queries to QSC1 and another 20 to QSC2, then the queries, replies and map update exchanges which must be performed by the one QSR which both QSC1 and QSC2 query will be significantly reduced. This is because it will sometimes or frequently be the case that QSC1 will already be caching the mapping which is needed to answer a query from one of its 20 ITRs. Without the QSCs, every ITR query would need to be handled by the QSR - and its querier cache (where the QSR retains records of the mappings it sent in Map Replies to various queriers, along with the caching time variables and the nonce which it was sent in the initial Map Request) would be correspondingly larger. Furthermore, if more than one of QSC1's ITRs is caching mapping for a micronet for which the QSR receives a Cache Update, then the QSR only needs to send a single mapping update to QSC1, rather than sending one to each such ITR. Whittle Expires September 8, 2010 [Page 59] Internet-Draft Ivip Architecture March 2010 There is further work to do planning these protocols. The caching times do not affect Ivip's ability to get the mapping updates to all ITRs in real-time. Longer caching times will reduce the need for the querier, such as an ITR, to make another map request if it is still sending packets to the micronet. Longer caching times also increase the number of mapping updates which need to be sent - and perhaps the time is so long that the querier no longer needs the mapping. Shorter caching times reduce the number of cached items, but increase the load of mapping queries and responses. There needs to be coordination between the caching times of Map Replies sent out by a QSA and those sent out by dependent QSRs and QSCs. Also, each querier needs to periodically check that any upstream query server it is caching any mapping from is still alive and has not been rebooted. If the upstream server has died or been rebooted, there is a danger that the cached mapping in the querier should have been changed or flushed due to a Cache Update message which the upstream server would have sent if it had not died or been rebooted. This is for further work. While the exact details are TBD, it is clear that it will be possible to define relatively straightforward protocols by which ITRs, optional QSCs, QSRs and QSAs can be combined to efficiently support the mapping needs of many ITRs per QSR. 8.5. MHF - Modified Header Forwarding 8.5.1. EAF - ETR Address Forwarding for IPv4 Please see [I-D.whittle-ivip-etr-addr-forw] and the discussion above in the ITR section. To-do - rationalise the various mentions of MHF and especially EAF in this ID. EAF will not accept fragmented packets or fragmentable packets longer than some globally agreed constant, somewhat below 1500 bytes. By the time Ivip is introduced, it will have been over 20 years since RFC 1191 PMTUD was introduced. There's no need for fragments or fragmentable packets - and IPv6 does fine without them. EAF requires upgraded routers between ITRs and ETRs. This does not necessarily include every DFZ router, but it is reasonable to approximate the requirement to this. For instance, if a DFZ router never handles packets for networks which contain either ITRs or ETRs, then it does not need to handle EAF formatted packets. EAF ETR addresses contain only the 30 most significant bits. (But see previous notes on how with a new protocol number a new header could Whittle Expires September 8, 2010 [Page 60] Internet-Draft Ivip Architecture March 2010 carry 31 r probably 32 bits of ETR address.) To avoid the need to change ETRs' addresses when encapsulation is transitioned to EAF, ETRs should not be placed closer than 4 IP addresses apart. Perhaps they should be placed on the 01 address of these four. Since ITRs will commonly be placed deep within ISP and end-user networks, and ETRs may be deep within ISP networks (such as at an end-user site, at the end of the link from the ISP) any router between the DFZ and these devices also needs to handle EAF packets. It will be straightforward to build this capability into new routers, and into firmware updates for many existing routers. The upgrade only concerns the FIB. All that is altered is that the FIB forwards the packet according to the 30 (32?) bits ETR address bits in the header, rather than using the destination address. There is no change to BGP functions, the RIB or how the RIB writes to the FIB. If it takes a few years before Ivip or the like is introduced, it is possible that by then, many or almost all of the installed DFZ routers will be able to do this with a firmware update. With a year or two's notice, upgrading all the DFZ routers, and likewise many internal routers, would enable Ivip to be introduced in its final mode of operation - without encapsulation overhead or its PMTUD problems. This means all ITRs can be a lot simpler - and that ETRs can be trivially simple. Reducing the complexity of ITRs is perhaps the biggest challenge in designing a CES architecture, since we want ITRs to be cheap and plentiful, including them being easy to add to the stacks of sending hosts. Starting with EAF would also avoid the need for devising a transition mechanism from encapsulation. 8.5.2. PLF - Prefix Label Forwarding, for IPv6 The current state of PLF design is described in [PLF for IPv6]. Please see this for more details, including why it is totally different from MPLS and how it could be extended to provide a similar 2^19 or 2^20 destination forwarding system within each ISP (or end- user) network. While EAF is pretty much a functional replacement of IPv4's encapsulation system, PLF is rather different in that it only takes the packet to a BR of one of 2^19 or 2^20 DFZ-advertised prefixes. This would be a regular, contiguous, set of prefixes used only by ISPs - for this and for potentially other purposes. If Ivip for IPv6 began with encapsulation, then it would make sense for the ETRs to be already located in these special prefixes. Whittle Expires September 8, 2010 [Page 61] Internet-Draft Ivip Architecture March 2010 Otherwise, they would need to be moved there before EAF could be turned on. EAF may require a second lookup at the BR of the ISP's network - if there are more than one ETRs for that prefix. One way of forwarding the packet from the BR to the correct ETR would be to use these PLF bits for a similar system within the ISP's network, with 2^19 or 2^20 internal prefixes. How the ISP uses these bits is a private matter. This could be a very powerful way of directing traffic inside a large provider network. This would give rise to ePLF - the one system for the DFZ - and iPLF, as used inside an individual ISP network. Rapid adoption of IPv6 is still somewhere beyond the immediately foreseeable future. So there's no hurry about deploying a scalable routing solution for IPv6. I think the most likely scenario for widespread adoption of IPv6 is one or more large 3G systems using it to give each phone (or whatever) its own global unicast address. This in itself will not cause a scaling problem, since these will be large systems with few new prefixes to add to the DFZ. However, there would then be a strong need for mobility - and the TTR approach has advantages over traditional MIP techniques, as discussed below. Perhaps by the time Ivip is deployed for IPv6, all the IPv6 DFZ routers will be upgradable to PLF with firmware updates - so scalable routing could be done without encapsulation. PLF involves small changes to the FIB and to the RIB. It does not involve any new BGP functionality. 8.6. TTR Mobility TTR Mobility is fully described, with diagrams, in [TTR Mobility]. This architecture will work equally well for IPv4 and IPv6. The MN can be on any kind of address, including behind multiple layers of NAT, on DHCP addresses and on addresses provided by conventional Mobile IP protocols. The MN can even be on an SPI address which is within another MN's micronet. No stack or application changes are required and the hosts communicate normally with all other hosts, including of course others using TTR mobility. There is no home- agent and paths to correspondent hosts are generally optimal. Mapping changes are not required when the MN gains a new address. They are not actually required at all, but are desirable if the MN moves to a part of the network which is far from its current TTR. This may be a distance of 1000km or more. Then, it should establish a tunnel to a nearby TTR so the TTR company can change the mapping of its micronet to this new TTR. With Ivip's real-time control of mapping, this means the MN could close the tunnel to the old TTR within five or so seconds of the mapping change being sent. Changing Whittle Expires September 8, 2010 [Page 62] Internet-Draft Ivip Architecture March 2010 the mapping does not cause any glitch in connectivity, since the MN gets packets from both TTRs during the changeover. The MN needs some additional tunneling software - which is controlled by the TTR company. This could be added alongside existing stacks, or integrated into the stack. Ideally the MN to TTR interface would be standardised in RFCs, but this is not strictly necessary, since the MN only needs to interoperate with TTRs of the TTR company chosen by the MN's owner. RFC-standardised MN and TTR functionality would be desirable, by allowing easy choice between TTR companies without the need to install software. However, there is a lot of scope for innovation in this area, and it might be difficult to adequately develop a full range of desirable protocols soon enough for the expected rapid uptake of mass-market Mobility. I think this approach to mobility, for IPv4 and at some stage for IPv6, is so attractive that there would be a business case for a company setting up its own Ivip-like system just for this purpose - irrespective of the need for a scalable routing solution. Such a system would need to use encapsulation. Multiple such systems could exist at the same time - and a MN in one system A would be able to communicate directly with a MN in another system B via the following paths (->) or tunnels (==>): MN-A ==> TTR-A -> (via DFZ) -> ITR-B ==> TTR-B ==> MN-B. Any such systems should be designed to upgraded in the future to comply with future RFCs for an Ivip-like system, including initial or long-term adoption of Modified Header Forwarding rather than encapsulation. Whittle Expires September 8, 2010 [Page 63] Internet-Draft Ivip Architecture March 2010 9. Security Considerations Security analysis can only be done in the years to come, once the protocols are designed in some detail. Ivip ITRs and ETRs are much simpler than those of LISP. Ivip ETRs easily enforce ISP BR source address filtering. For LISP ETRs to enforce this would be at least administratively complex and very expensive for large numbers of filtered prefixes - and it may be impossible to do while allowing for ITRs in the local ISP network tunneling to this ETR. Whittle Expires September 8, 2010 [Page 64] Internet-Draft Ivip Architecture March 2010 10. IANA Considerations [To do.] Whittle Expires September 8, 2010 [Page 65] Internet-Draft Ivip Architecture March 2010 11. Informative References [C-E-Sep-Elim] Jen, D., Zhang, L., Lan, L., and B. Zhang, "Towards a Future Internet Architecture: Arguments for Separating Edges from Transit Core", September 2008, . [Constraints-Voluntary] Whittle, R., "List of constraints on a successful scalable routing solution which result from the need for widespread voluntary adoption", April 2009, . [Critique of draft-jen-mapping-00] Whittle, R., "draft-jen-mapping does not apply to the TTR Mobility architecture", January 2010, . [DFZ-unfrag-1470] Whittle, R., "Google sends 1470 byte unfragmentable packets", August 2008, . [Deering-1996] Deering, S., "The Map & Encap Scheme for scalable IPv4 routing with portable site prefixes", March 1996, . [Host-Responsibilities] Whittle, R., "Objections to burdening hosts with more Routing and Addressing responsibilities", December 2009, < http://www.firstpr.com.au/ip/ivip/RRG-2009/ host-responsibilities/>. [I-D.adan-idr-tidr] Adan, J., "Tunneled Inter-domain Routing (TIDR)", draft-adan-idr-tidr-01 (work in progress), December 2006. [I-D.ietf-lisp] Farinacci, D., Fuller, V., Meyer, D., and D. Lewis, "Locator/ID Separation Protocol (LISP)", draft-ietf-lisp-06 (work in progress), January 2010. [I-D.irtf-rrg-recommendation] Li, T., "Recommendation for a Routing Architecture", draft-irtf-rrg-recommendation-05 (work in progress), February 2010. Whittle Expires September 8, 2010 [Page 66] Internet-Draft Ivip Architecture March 2010 [I-D.jen-mapping] Jen, D. and L. Zhang, "Understand Mapping", draft-jen-mapping-00 (work in progress), October 2009. [I-D.lear-lisp-nerd] Lear, E., "NERD: A Not-so-novel EID to RLOC Database", draft-lear-lisp-nerd-06 (work in progress), December 2009. [I-D.lewis-lisp-interworking] Lewis, D., "Interworking LISP with IPv4 and IPv6", draft-lewis-lisp-interworking-00 (work in progress), December 2007. [I-D.meyer-lisp-cons] Brim, S., "LISP-CONS: A Content distribution Overlay Network Service for LISP", draft-meyer-lisp-cons-04 (work in progress), April 2008. [I-D.rja-ilnp-intro] Atkinson, R., "ILNP Concept of Operations", draft-rja-ilnp-intro-02 (work in progress), December 2008. [I-D.whittle-ivip-drtm] Whittle, R., "DRTM - Distributed Real Time Mapping for Ivip and LISP", draft-whittle-ivip-drtm-01 (work in progress), March 2010. [I-D.whittle-ivip-etr-addr-forw] Whittle, R., "Ivip4 ETR Address Forwarding", draft-whittle-ivip-etr-addr-forw-00 (work in progress), January 2010. [I-D.whittle-ivip-fpr] Whittle, R., "Fast Payload Replication mapping distribution for Ivip", draft-whittle-ivip-fpr-01 (work in progress), March 2010. [I-D.whittle-ivip-glossary] Whittle, R., "Glossary of some Ivip and scalable routing terms", draft-whittle-ivip-glossary-01 (work in progress), March 2010. [Ivip Summary and Analysis] Whittle, R., "Ivip Conceptual Summary and Analysis", December 2008, . [Ivip-2007-06-15] Whittle Expires September 8, 2010 [Page 67] Internet-Draft Ivip Architecture March 2010 Whittle, R., "ViP: Anycast ITRs in the DFZ & mobile tunnels", June 2007, . [LISP-ALT-Critique] Whittle, R., ""How can the ALT structure scale to 10^8, 10^9 or 10^10 EIDs with minimal delay times and robustness against single points of failure?"", December 2009, . [Namespace] Whittle, R., "The meaning of the term *namespace* in addressing, computer networking etc.", April 2009, . [PLF for IPv6] Whittle, R., "Prefix Label Forwarding (PLF) - Modified Header Forwarding for IPv6", August 2008, . [PMTUD-Frag] Whittle, R., "IPTM - Ivip's approach to solving the problems with encapsulation overhead, MTU, fragmentation and Path MTU Discovery", April 2008, . [TTR Mobility] Whittle, R. and S. Russert, "TTR Mobility Extensions for Core-Edge Separation Solutions to the Internets Routing Scaling Problem", August 2008, . [Vogt-2009] Vogt, C., "Simplifying Internet Applications Development With A Name-Based Sockets Interface", December 2009, . [loc-id-sep-vs-ces] Whittle, R., "Loc/ID Separation is different from Core- Edge Separation", January 2010, . Whittle Expires September 8, 2010 [Page 68] Internet-Draft Ivip Architecture March 2010 Appendix A. Acknowledgements Thanks to the following people for their help and encouragement: Juan Jo Aden, Noel Chiappa, Olivier Bonaventure, Brian Carpenter, Dino Farinacci, Vince Fuller, Joel M. Halpern, Geoff Huston, Ved Kafle, Eliot Lear, Simon Leinen, Tony Li, Jeroen Massar, Dave Meyer, Chris Morrow, Dave Oran, Robert Raszuk, Jason Schiller, John Scudder, K. Sriram, Markus Stenberg, Letong Sun, Christian Vogt, Kilian Weniger and Xiaoming Xu. This is not to imply that these people support Ivip. I especially thank Steve Russert, formerly of Boeing, for collaborating on the TTR Mobility paper for MobiArch '08. The original draft wasn't accepted and by the time we revised it to the point of being happy with it, the paper was 2.5 times as long as the conference page limit. Whittle Expires September 8, 2010 [Page 69] Internet-Draft Ivip Architecture March 2010 Author's Address Robin Whittle First Principles Email: rw@firstpr.com.au URI: http://www.firstpr.com.au/ip/ivip/ Whittle Expires September 8, 2010 [Page 70]