Network Working Group R. Whittle Internet-Draft First Principles Intended status: Experimental January 13, 2010 Expires: July 17, 2010 Ivip (Internet Vastly Improved Plumbing) Architecture draft-whittle-ivip-arch-03.txt Abstract Ivip (Internet Vastly Improved Plumbing) is a core-edge separation solution to the routing scaling problem, for both IPv4 and IPv6. It provides portable address space which is suitable for multihoming and inbound traffic engineering (TE) to end-user networks of all types and sizes. The subset of the global unicast address space which is used in this fashion is called SPI (Scalable Provider Independent) space. End-user networks divide their SPI space into "micronets", each with a common mapping to a particular ETR (Egress Tunnel Router) address. ITRs (Ingress Tunnel Routers) receive packets which are addressed to SPI addresses and, after looking up the mapping at a local query server, tunnel traffic packets to the ETR which delivers them to the end-user network. Ivip pushes mapping changes in real- time to local full-database query servers - such as in each ISP. These answer ITRs' mapping queries and push subsequent changes to the mapping to any ITRs which would still be caching the mapping of an earlier reply. The mapping for a micronet consists of a single ETR address. ITRs make no decisions between multiple ETR addresses. End-user networks would typically contract a separate company to change the mapping of their micronet, in response to the reachability of their ETRs and TE and portability between ISPs. Ivip includes two extensions for ITR-to-ETR tunneling without encapsulation and the resulting Path MTU Discovery problems - one for IPv4 and the other for IPv6. Both involve modifying the IP header and require most DFZ routers to be upgraded. Ivip is a good basis for the TTR (Translating Tunnel Router) approach to mobility, in which mobile hosts retain an SPI micronet of one IPv4 address (or IPv6 /64) no matter what physical addresses they are using, including behind NAT. TTR mobility for both IPv4 and IPv6 involves generally optimal paths and works with unmodified correspondent hosts. Status of this Memo This Internet-Draft is submitted to IETF in full conformance with the provisions of BCP 78 and BCP 79. Internet-Drafts are working documents of the Internet Engineering Task Force (IETF), its areas, and its working groups. Note that Whittle Expires July 17, 2010 [Page 1] Internet-Draft Ivip Architecture January 2010 other groups may also distribute working documents as Internet- Drafts. Internet-Drafts are draft documents valid for a maximum of six months and may be updated, replaced, or obsoleted by other documents at any time. It is inappropriate to use Internet-Drafts as reference material or to cite them other than as "work in progress." The list of current Internet-Drafts can be accessed at http://www.ietf.org/ietf/1id-abstracts.txt. The list of Internet-Draft Shadow Directories can be accessed at http://www.ietf.org/shadow.html. This Internet-Draft will expire on July 17, 2010. Copyright Notice Copyright (c) 2010 IETF Trust and the persons identified as the document authors. All rights reserved. This document is subject to BCP 78 and the IETF Trust's Legal Provisions Relating to IETF Documents (http://trustee.ietf.org/license-info) in effect on the date of publication of this document. Please review these documents carefully, as they describe your rights and restrictions with respect to this document. Code Components extracted from this document must include Simplified BSD License text as described in Section 4.e of the Trust Legal Provisions and are provided without warranty as described in the BSD License. Whittle Expires July 17, 2010 [Page 2] Internet-Draft Ivip Architecture January 2010 Table of Contents 1. Introduction . . . . . . . . . . . . . . . . . . . . . . . . . 5 2. Goals . . . . . . . . . . . . . . . . . . . . . . . . . . . . 9 2.1. IPv4 and IPv6 . . . . . . . . . . . . . . . . . . . . . . 9 2.2. Portability, multihoming and TE for billions of end-user networks . . . . . . . . . . . . . . . . . . . . 9 2.3. Modular separation of the control of mapping from the core-edge separation architecture itself . . . . . . . . . 11 2.4. Simple ITRs and ETRs with little or no communication between them . . . . . . . . . . . . . . . . . . . . . . . 11 2.5. Maximise the flexibility with which ITRs and ETRs can be located . . . . . . . . . . . . . . . . . . . . . . . . 12 2.6. Mobility . . . . . . . . . . . . . . . . . . . . . . . . . 13 2.7. Elimination of encapsulation and PMTUD problems . . . . . 14 2.8. No requirement for new host functionality . . . . . . . . 15 2.9. Full benefits to all adopters . . . . . . . . . . . . . . 15 2.10. Business incentives to deploy new infrastructure . . . . . 16 2.11. Maintenance of existing levels of security and robustness . . . . . . . . . . . . . . . . . . . . . . . . 17 3. Non-goals . . . . . . . . . . . . . . . . . . . . . . . . . . 19 3.1. Complete core-edge separation is not required . . . . . . 19 3.2. Mapping changes need not be free of financial cost . . . . 20 3.3. No attempt to cope with partially reachable ETRs . . . . . 21 3.4. No attempt to avoid all the mapping data being stored at any one location . . . . . . . . . . . . . . . . . . . 23 3.5. It may not be possible to completely eliminate unfair burdens . . . . . . . . . . . . . . . . . . . . . . . . . 24 3.6. No attempt to mix IPv4 and IPv6 . . . . . . . . . . . . . 25 3.7. Not Locator - Identifier Separation . . . . . . . . . . . 25 4. Architectural Choices . . . . . . . . . . . . . . . . . . . . 27 4.1. Core-edge separation rather than elimination . . . . . . . 27 4.1.1. Core-Edge Elimination (CEE) architectures . . . . . . 27 4.1.2. Core-Edge Seperation (CES) architectures . . . . . . . 29 4.2. Local query servers . . . . . . . . . . . . . . . . . . . 31 4.3. Real-time mapping distribution . . . . . . . . . . . . . . 32 4.4. SPI address management . . . . . . . . . . . . . . . . . . 32 4.5. IP in IP encapsulation . . . . . . . . . . . . . . . . . . 35 4.6. MHF initially or in the long term to avoid encapsulation and PMTUD problems . . . . . . . . . . . . . 35 4.7. Outer header address is that of the sending host . . . . . 35 4.8. IPTM (ITR Probes Tunnel MTU) PMTUD management . . . . . . 36 5. Architectural Elements . . . . . . . . . . . . . . . . . . . . 39 5.1. ITRs . . . . . . . . . . . . . . . . . . . . . . . . . . . 39 5.1.1. Types of ITR and their addresses . . . . . . . . . . . 39 5.1.2. DITRs - Default ITRs in the DFZ . . . . . . . . . . . 40 5.1.3. Modified Header Forwarding - MHF-only ITRs . . . . . . 41 5.1.4. Encapsulation and PMTUD management . . . . . . . . . . 41 Whittle Expires July 17, 2010 [Page 3] Internet-Draft Ivip Architecture January 2010 5.1.5. Mapping lookup and caching . . . . . . . . . . . . . . 43 5.1.6. ITFH - ITR Function in Host . . . . . . . . . . . . . 45 5.1.7. ITRs auto-discovering local query servers . . . . . . 46 5.1.8. ITRs regulating their advertisements . . . . . . . . . 46 5.2. ETRs . . . . . . . . . . . . . . . . . . . . . . . . . . . 47 5.2.1. In servers or dedicated routers . . . . . . . . . . . 47 5.2.2. ETRs in ISP networks . . . . . . . . . . . . . . . . . 47 5.2.3. ETRs at the end-user network site . . . . . . . . . . 47 5.2.4. MHF ETR functionality - EAF and PLF . . . . . . . . . 48 5.2.5. ETR functionality for encapsulation . . . . . . . . . 49 5.3. QSDs - full database query servers . . . . . . . . . . . . 50 5.3.1. Placement in ISP and end-user networks . . . . . . . . 50 5.3.2. QSD initialization and reception of mapping updates . 51 5.3.3. Responding to queries . . . . . . . . . . . . . . . . 53 5.3.4. Sending mapping updates to ITRs and QSCs . . . . . . . 54 5.4. QSCs - caching query servers . . . . . . . . . . . . . . . 55 5.5. FMS - Fast-Push Mapping Distribution System . . . . . . . 56 5.6. MHF - Modified Header Forwarding . . . . . . . . . . . . . 63 5.6.1. EAF - ETR Address Forwarding for IPv4 . . . . . . . . 63 5.6.2. PLF - Prefix Label Forwarding, for IPv6 . . . . . . . 64 5.7. TTR Mobility . . . . . . . . . . . . . . . . . . . . . . . 65 6. Security Considerations . . . . . . . . . . . . . . . . . . . 67 7. IANA Considerations . . . . . . . . . . . . . . . . . . . . . 68 8. Informative References . . . . . . . . . . . . . . . . . . . . 69 Appendix A. Acknowledgements . . . . . . . . . . . . . . . . . . 72 Author's Address . . . . . . . . . . . . . . . . . . . . . . . . . 73 Whittle Expires July 17, 2010 [Page 4] Internet-Draft Ivip Architecture January 2010 1. Introduction Version 03 of this Ivip-arch ID is a freshly written document which is shorter than the original from 2007. Some terminology has been changed and the presentation is optimised for people who are involved in the RRG. Please see [I-D.whittle-ivip-glossary] for definitions of some terms and acronyms. Please refer to the RRG mailing list and http://www.firstpr.com.au/ip/ivip/ for the latest developments. The Ivip (pr. "Eye-vip") project began in June 2007 and in early 2010 is one of the three Core-Edge Separation (CES) architectures being considered by the RRG (IRTF Routing Research Group) [I-D.irtf-rrg-recommendation] - the others being LISP [I-D.ietf-lisp] and TIDR [I-D.adan-idr-tidr]. The routing scaling problem can be summarised as there being practical limits and unfair cost-burdens to providing portability, multihoming and inbound traffic engineering (TE) for the potentially large number of end-user networks which want or need this. The same problem exists in principle for IPv4 and IPv6, but only IPv4 has a problem at present. The burden placed on the interdomain routing system (often referred to loosely as the Default-Free Zone - DFZ) by the advertised prefixes of ISPs is generally thought not to be a problem. So the challenge is to find a way of providing address space and new methods of routing so that the portability, multihoming and TE needs of potentially millions or billions of end-user networks can be served in a "scalable" manner: efficiently, robustly and without unfair burdens falling on anyone, such as those who operate the DFZ routers. The unfair, unsustainable, burden is caused by the number of separately advertised PI prefixes of end-user networks today - and the rate at which these prefixes have their point of advertisement changed. (Also, if an end-user network changes the type of advertisement frequently, such as with more or less ASNs, this too is a burden.) The most important part of the burden is on the DFZ's "BGP control plane". This is partly the inter-router BGP traffic and the overall behaviour of routers - particularly any difficulty which the excessive number of prefixes causes in the system converging to good enough best-paths in the event of an outage. It is also the burden of CPU effort and storage in the RIB of each router. This includes the effort of writing changes to the FIB when RIB information changes. Also, FIBs may have their ability to handle packets temporarily disabled while new information is written. Whittle Expires July 17, 2010 [Page 5] Internet-Draft Ivip Architecture January 2010 The actual number of prefixes each DFZ router has to handle is a major part of the problem, though the total RIB burden also depends on how many neighbours each router has. The number of prefixes in the FIB is a serious burden too, but it is widely believed that this is not the most important problem. Any solution which only helps with the FIB is not really a solution to the problem. In order to provide portability etc. to millions or perhaps billions of end-user networks in a scalable manner, it follows that the DFZ routers must not have to consider the prefixes of each individual network at all in their RIB or FIB. Consequently, most approaches to solving the problem concentrate on providing a new type of address, or a subset of existing addresses, which is suitable and attractive for providing end-user networks with portability, multihoming and TE, but which places only very slight burden on the DFZ compared to the burden each PI prefix places today. Support for mobility has not generally been considered part of the routing scaling problem. However, with the proliferation of cellphones, VoIP, other IP applications it is reasonable to assume that in the future, most hosts will be mobile devices, generally running on limited battery power and relying on wireless links which are frequently slow, unreliable and/or expensive. Mobility is arguably an extreme form of portability and/or multihoming. To embark on a major architectural enhancement for scalable routing, in a manner which did not support billions of mobile devices, would make little sense. So while mobility is not a formal part of the routing scaling problem, Ivip and some other proposed architectures seek to provide mobility on a massive scale. In these scenarios, each mobile device is generally considered to be a separate end-user network. An entire corporation's network, or that of a large university, is also an "end-user network". So in the following discussion, this term could mean a wide variety of things - far beyond the small subset of end-user networks whish are currently able to gain and advertise PI space. Apart from Core-Edge Separation architectures, there are three other classes of architectural enhancement for solving the Internet's routing and addressing scaling problem. Firstly, improving BGP so the DFZ can cope with vastly greater numbers of end-user prefixes - which is impossible. Secondly, replacing BGP with some other arrangement so the interdomain routing system can cope with this increase in the number of end-user network prefixes. It may be technically possible to devise such a scheme - but there appear to be insurmountable difficulties in introducing it to the operating IPv4 or IPv6 Internets. Thirdly, Core-Edge Whittle Expires July 17, 2010 [Page 6] Internet-Draft Ivip Architecture January 2010 Elimination (CEE) architectures promise scalable routing and a simplified (compared to CES architectures), arguably more elegant, network - with hosts gaining new routing and addressing responsibilities and with the DFZ working much as it does today, but only for ISP prefixes. The reasons for choosing Core-Edge Separation over Elimination for Ivip are described in detail in Section 4. In summary, the CES approach is the only one which meets the constraints imposed by the need for widespread voluntary adoption. Furthermore, and contrary to some widely held beliefs, I argue that network-centric approach of CES architectures produces a better performing Internet than one in which hosts do all the routing and addressing work. This is particularly so considering that in the future, the majority of hosts will be operating over typically slow, unreliable and potentially costly wireless last-mile links. Ivip is based on some unique architectural choices, including: ITRs (Ingress Tunnel Routers) receiving mapping changes in real-time; local full-database query servers; migration to Modified Header Forwarding (MHF) to avoid encapsulation and its PMTUD (Path MTU Discovery) difficulties; and (when encapsulation is used) the use of the sending host's address as the outer header's source address, so that ETRs can easily enforce ISP BR (Border Router) source address filtering on decapsulated packets. The following description assumes that Ivip will be introduced with encapsulation, with long-term migration to MHF. However, it is possible that by the time the introduction date is set that most DFZ routers will have firmware based FIBs, and so could be easily upgraded to support MHF. In that case, ITRs and ETRs could be much simpler, since they would not need to handle encapsulation or PMTUD management. Below, Ivip is generally assumed to be introduced as a single system for the purposes of solving the routing scaling problem. However, multiple independent systems along the lines of Ivip (with encapsulation) could also be introduced without need for standardisation for the purpose of supporting commercial TTR Mobility services. I publicly disclose and discuss all Ivip developments as rapidly as possible in order to gain support and constructive critiques - and in the hope that any novel ideas will remain free from patent encumbrances. This ID is intended for readers who are broadly familiar with the routing scaling problem and RRG discussions and who have, ideally, Whittle Expires July 17, 2010 [Page 7] Internet-Draft Ivip Architecture January 2010 familiarised themselves with LISP. This ID provides not only a general description of Ivip, but the rationale for architectural choices which distinguish Ivip from other approaches. Some aspects of Ivip's architecture are discussed in greater detail in separate documents: The fast push mapping distribution system to the local query servers could be implemented in various ways, such as flooding mapping data through a meshed network of servers. The current plan involves a global network of servers in a structure resembling crosslinked multicast trees. Please refer to [I-D.whittle-ivip-db-fast-push] for further details. The TTR approach to mobility is described in [TTR Mobility]. The IPv4 approach to Modified Header Forwarding (MHF) is described in detail in [I-D.whittle-ivip-etr-addr-forw]. The IPv6 approach is described in [PLF for IPv6] and the best summary of its operation can be found at the end of the ~10k word Ivip Conceptual Summary and Analysis: [Ivip Summary and Analysis] . Ivip's approach to Path MTU Discovery, when ITRs tunnel using encapsulation, is discussed in [PMTUD-Frag]. Whittle Expires July 17, 2010 [Page 8] Internet-Draft Ivip Architecture January 2010 2. Goals 2.1. IPv4 and IPv6 Ivip is intended to solve the routing scaling problem (as described in the introduction), for IPv4 and IPv6, for very large numbers of end-user networks. Much of Ivip is identical in principle for both Internets. However the mapping information for IPv6 is lengthier and there are other differences, such as in Path MTU Discovery (PMTUD) when encapsulation is used, and in the IPv4 and IPv6 approaches to MHF which remove the need for encapsulation. 2.2. Portability, multihoming and TE for billions of end-user networks Ivip is intended to provide scalable address space for billions of end-user networks - for both IPv4 and IPv6. The new kind of address space - SPI (Scalable Provider Independent) space - is suitable for end-user networks to use in a portable fashion, meaning they can keep this space when choosing another ISP for Internet connectivity. Portability of the end-user network address space which is used to identify hosts, routers and networks is an absolute requirement of scalable routing. Even if a network could reliably and inexpensively renumber all its hosts and routers, and change all its configuration files which contained such addresses, it would never be able to reliably and securely alter all the other places where these identifying addresses reside in other networks. These includes the use of these addresses in referrals, existing communication sessions, config files of VPNs and hard-coded (however questionably) into firmware and software. Assuming the end-user network has two or more ISPs, SPI space will also support multihoming and inbound traffic engineering. In the following, "TE" refers to "inbound traffic engineering" - the ability to steer incoming traffic streams between two or more ISPs. Ivip's approach to TE differs from that of other core-edge separation schemes. It is potentially finer-grained, more flexible and more able to respond to rapid changes in traffic patterns. Ivip is intended to provide SPI space for billions of end-user networks. With about 3.7 billion unicast IP addresses, it seems unlikely that billions of end-user networks could exist within IPv4, but it would be technically possible if most of the space was reserved for such networks, if most of them used a single IP address for their Whittle Expires July 17, 2010 [Page 9] Internet-Draft Ivip Architecture January 2010 micronets, and if most of the unicast address space was used for SPI space. An upper estimate of the number of non-mobile networks which would want or need SPI space is in the 10^7 range. This is on the basis of a population of 10^10, with there being only one organisation per 10^3 people which needs portability, multihoming and/or TE enough to establish a physical link to a second ISP and pay two ISPs rather than one for Internet access. This is an order-of-magnitude estimate. Perhaps with inexpensive wireless links to another ISP, increasing numbers of end-user networks would want to multihome instead of relying on their single fibre, DSL or HFC cable connection. Already, many homes and offices can connect via two or more of these types of cabled connections. Nonetheless, even if it was convenient and inexpensive, it is hard to imagine more than 10^7 non-mobile end-user networks being sufficiently concerned about multihoming or portability to use SPI space, rather than the small amount of PA space they currently receive from their single ISP. The goal of scalable routing is to scalably provide portability, multihoming and TE to all networks which want or need it. However, it is reasonable to assume that most home and SOHO networks, and some smaller factory and office networks, will remain happy with the reliability of their single-provider service, and will not concerned about portability when choosing another ISP. On this basis, scalable routing architectures which are intended to scale to more than about 10^7 end-user networks will be doing so primarily for mobile end-user networks, typically consisting of a single hand-held device which operates from a single IPv4 or IPv6 address or from a single IPv6 /64 prefix. A small number of end-user networks will have multiple sites or some other reason to split their SPI space into multiple micronets, but in any realistic scenario involving billions of such networks, the great majority of such networks will be a single site or device, with little or no need for TE or greater address space than a single IPv4 address or an IPv6 /64. Therefore, it is reasonable to expect that most of these billions of networks will require only a single micronet of SPI addresses. So, for these scenarios of billions of end-user networks, the total number of separately mapped micronets of SPI address space will be only marginally greater than the number of end-user networks. Whittle Expires July 17, 2010 [Page 10] Internet-Draft Ivip Architecture January 2010 2.3. Modular separation of the control of mapping from the core-edge separation architecture itself Ivip's real-time mapping system means that the tunneling behaviour of all ITRs can be controlled directly. The mapping consists of a single ETR address, so unlike with other core-edge separation architectures, Ivip ITRs do not need to make any choices between multiple ETRs for the purposes of multihoming service restoration or TE. Control of the tunneling behaviour of Ivip ITRs rests entirely outside the Ivip system. It is the responsibility of end-user networks to control this mapping at all times - and they are likely to delegate this responsibility to a company they hire for this purpose. Exactly how end-user networks make their decisions about mapping - and how, for instance, a Multihoming Monitoring (MM) company might detect ETR failure, and alter mapping accordingly - is entirely separate from Ivip's mapping system, ITRs and ETRs. It would be a private, flexible, arrangement between an end-user network and the MM company it hires to continually probe the network's reachability via its two or more ETRs. This means the frequency and type of probing, and the decision-making algorithms, can be completely open-ended and subject to development and customisation - without any constraints or need for changes in the RFCs which define Ivip. This modular separation of the detection and decision-making functions from the core-edge separation is good engineering practice and ensures that the Ivip subsystem can be used flexibly, including for purposes not yet anticipated. Other CES techniques monolithically integrate the following functions into the core-edge separation architecture itself - primarily by specifying exactly how all ITRs must behave: multihoming failure detection, decisions about how to choose between ETRs to restore service and how to implement TE. This would add greatly to the complexity of the system itself, make it harder to introduce new methods of testing reachability etc. and restrict all end-user networks to relying on the necessarily restricted set of functions which can reasonably be built into all ITRs. 2.4. Simple ITRs and ETRs with little or no communication between them With encapsulation, the only time ITRs engage in two-way communication is when probing the Path MTU to the ETR, by using a special pair of packets which carry a larger traffic packet than has previously been successfully received by the ETR from this ITR. Whittle Expires July 17, 2010 [Page 11] Internet-Draft Ivip Architecture January 2010 Apart from this ITRs do not communicate with anything but their local query servers. ETRs do not communicate with any other part of the Ivip system than the ITR for this PMTU management function. If MHF is used rather than encapsulation, there is no need for ITRs to communicate with ETRs - so ITRs only communicate with query servers and ETRs do not communicate at all. Consequently ETRs and ITRs can be simple functions in existing routers or in standalone servers. The ITR function can also be implemented in the sending host (ITFH), though this is not advisable if the sending host is on a slow, unreliable, link such as a wireless link. ETRs must be on conventional global unicast addresses - not SPI addresses. ITRs can be both kinds of address. Ivip may include an option for an ITR or ITFH to set up a two-way persistent tunnel to its one or more local query servers, which would allow an ITR function to be behind one or more layers of NAT. This "tunnel" could be as simple as TCP from the ITR, or ITFH, to each query server, with keepalive packets. It is important to make ITRs as simple as possible, in order that they may be inexpensive and therefore, if desired, more numerous - so as to reduce the load on each one. ETRs are simpler than ITRs, since they simply decapsulate packets with a comparison between outer and inner source addresses and do not look up or cache mapping information. Ivip with encapsulation uses simple IP-in-IP encapsulation. There is no special header and no other data piggybacked onto traffic packets. This minimises encapsulation overhead and reduces the complexity of both ITRs and ETRs. Other core-edge separation architectures use their own headers to carry extra information with each traffic packet, with that header behind a UDP header. These other architectures also require ITRs to determine reachability to multiple ETRs. 2.5. Maximise the flexibility with which ITRs and ETRs can be located Ivip ITRs can be located in the sending host, in the sending-host's end-user network (which may be an ISP network or an end-user network using either SPI or conventional PI space) or in the ISP network which the host's end-user network connects to the Net through. If there is no such ITR, the packet will enter the DFZ and be forwarded to the nearest (in BGP terms) DITR (Default ITR in the DFZ, previously known as OITRD for Open ITR in the DFZ). ETRs can be located in ISP networks with a link to the end-user network. ETRs can also be located at the end-user network end of a Whittle Expires July 17, 2010 [Page 12] Internet-Draft Ivip Architecture January 2010 link from an ISP, and so be physically located at the end-use site. In both cases, their address must be a conventional global unicast address (usually from one of the ISP's prefixes) - not an SPI address or behind NAT. 2.6. Mobility One of Ivip's goals is to support massive mobility, since this is surely a major facet of the future of Internet communications. It would make no sense to introduce one set of architectural changes to solve the routing scaling problem as it appears today, and then have to devise and introduce a second set to provide for billions of mobile devices. Ivip is a good basis for the TTR approach to mobility, and would be attractive to deploy for this reason alone. It is frequently assumed that in order for a core-edge separation scheme to support mobility, the Mobile Node (MN) must be its own ETR. LISP-MN makes this assumption. So does draft-jen-mapping-00 [I-D.jen-mapping] - a critique of which is [Critique of draft-jen-mapping-00]. TTR mobility does not involve mapping changes every time the MN gains a new physical address, since it continues to use the same one or more TTRs as its one or more ETRs. Mapping changes are needed only to use a new TTR, which is desirable after the MN moves a large distance, such as 1000km or more. Although the TTR approach to mobility could be used with other core- edge separation schemes, Ivip is a better basis for TTR mobility than other core-edge separation schemes such as LISP, APT [I-D.jen-apt], TRRP [TRRP] or TIDR. None of these other proposals provide a method of ITRs gaining updated mapping within a few seconds, as Ivip does. With Ivip's real-time mapping system, the Mobile Node (MN) can begin using a new, nearby, TTR within seconds and, most importantly, within a few seconds no ITR will be tunneling packets to the previous, and now more distant, TTR. Therefore the MN can promptly end the tunnel to the previous TTR and use the new TTR exclusively. Without this real-time mapping, the MN would need to retain tunnels to one or more previous TTRs for as long as the mapping system takes to ensure no ITRs are tunneling packets to them. This could take 10 to 30 minutes or more for the non-Ivip core-edge separation schemes. TTR Mobility is not required to solve today's routing scaling problem. It may be regarded as separate to Ivip, because it could be used with other core-edge separation schemes. However, it is best to consider TTR Mobility as a natural extension of the basic Ivip Whittle Expires July 17, 2010 [Page 13] Internet-Draft Ivip Architecture January 2010 architecture, which does not place any constraints on the basic architecture other than that its mapping system will need to scale to billions of (mostly mobile, handheld device) end-user networks. 2.7. Elimination of encapsulation and PMTUD problems When ITRs use encapsulation to tunnel traffic packets to ITRs, there are serious problems with Path MTU Discovery (PMTUD) for the sending host. If the packet with its encapsulation header is too long for the next hop link of some router between the ITR and ETR, then there needs to be a mechanism by which the sending host receives a valid ICMP Packet Too Big message, with an MTU value which will result in an encapsulated packet of the correct length. It is challenging to solve this problem securely and without unreasonable amounts of state in the ITR. Ivip's solution - ITR Probes Path MTU [PMTUD-Frag] - involves extra complexity and state in ITRs and to a lesser extent in ETRs. This, and the transmission overhead of the encapsulation header (particularly heavy with IPv6 VoIP packets) provides strong motivation to either avoid encapsulation entirely, or to introduce Ivip with encapsulation, but in the long-term transition to an alternative system which lacks these problems. Ivip has two techniques, known collectively as Modified Header Forwarding (MHF) which replace encapsulation as the ITR to ETR tunneling technique. They are: 1. ETR Address Forwarding (EAF) - for IPv4. [I-D.whittle-ivip-etr-addr-forw] 2. Prefix Label Forwarding (PLF) - for IPv6. [PLF for IPv6]. If Ivip is introduced with encapsulation, all ITRs and ETRs will be capable of supporting MHF. At some date in the future, the DFZ routers will be upgraded to support this, probably without any significant cost. Ideally, it would be possible to establish Ivip from the outset without encapsulation. This would save having to develop the more complex ITR and ETR functions required by encapsulation - especially the PMTUD functionality. It would also eliminate the need to design a transition arrangement. It may be many years before Ivip or something like it is scheduled for deployment. I have not been able to reliably determine what Whittle Expires July 17, 2010 [Page 14] Internet-Draft Ivip Architecture January 2010 proportion of current DFZ routers have firmware-based FIBs. Any such router could be upgraded with a firmware update in order to support MHF. As the years pass, the chance that most or essentially all DFZ routers could be upgraded in this way, for very little cost, increases. Initial deployment with MHF is a goal, with the alternative goal being eventual transition to MHF. 2.8. No requirement for new host functionality It is a primary goal not to require any new host functionality - in stacks or applications. However, as an option, the ITR function can be integrated into sending hosts when this is desired. Mobile hosts using the TTR Mobility approach will have a little extra functionality, which could be implemented in the stack or perhaps outside it, as a separate piece of software. The IP stack itself and all applications remain unchanged and communicate with all other hosts, mobile or not, using by current protocols and addressing. One reason for avoiding the need for new host functionality is to enable the system to be widely enough adopted to solve the routing scaling problem, given the constraints imposed by the need for voluntary adoption. [Constraints-Voluntary] Another more fundamental reason is to ensure there is no extra burden on hosts, which would be particularly a problem for hosts which are on slow, expensive and unreliable links. This includes hosts on 3G wireless links - and in the future it is reasonable to expect this to be true of the majority of hosts. While many people are attracted to the idea of hosts doing more, and leaving the network to be simple, there are objections to this. I intend to write these up as an ID, but for now they are on a web-page and in RRG discussions. [Host-Responsibilities] In summary, it is highly undesirable for a new architecture to require all hosts to do more routing and addressing management than they currently do: just DNS lookups. The delays which are inherent any such arrangement are highly undesirable and the way these delays are worsened by one or both hosts being on high latency, unreliable, wireless links is particularly objectionable. Also, it is desirable not to enforce extra complexity or communication requirements on all hosts, since many of them will be constrained by battery power limitations. 2.9. Full benefits to all adopters Ivip provides the full benefits of portability, multihoming and TE to all end-user networks which adopt its SPI space. Whittle Expires July 17, 2010 [Page 15] Internet-Draft Ivip Architecture January 2010 In order to do this, packets from hosts in networks which lack ITRs must be forwarded to an ITR and tunneled to the correct ETR. This is achieved by placing a number of ITRs in the DFZ. These are known as DITRs (Default ITRS in the DFZ) and were previously known as OITRDs (Open ITRs in the DFZ). When Ivip was first announced [Ivip-2007-06-15] these were named (erroneously): "Anycast ITRs in the DFZ". By placing DITRs widely around the Net, path lengths from any sending host to the ETR are minimised. LISP Proxy Tunnel Routers (PTRs) perform the same function. [I-D.lewis-lisp-interworking] For a scalable routing solution to be widely enough adopted, it must provide compelling benefits to all adaptors, including the earliest. Without DITRs, PTRs or their equivalent, only a small fraction of packets being sent to an end-user network would use the new system. Yet the goal is for all adaptors to use the new form of addressing entirely, and so not to have to use the existing unscaleable BGP and PI space approach to portability, multihoming and TE. 2.10. Business incentives to deploy new infrastructure Some scalable routing proposals involve no additions to the network - just the adoption of new functionality in the end-user networks which use it. These are generally "Core-Edge Elimination" architectures. [C-E-Sep-Elim] No such proposal meets the constraints imposed by the need for widespread voluntary adoption. Firstly, most or all of them involve changes to host stacks and applications, which is impractical in the absence of compelling motivation for the authors of this software to make such major changes. Secondly all such proposals only provide portability, multihoming and TE benefits for packets sent from other networks which have adopted the scheme. Therefore, only if all networks adopted it would any one network be able to abandon its current routing and addressing arrangements. The benefits of scalable routing in a global sense, and for each adopter, the abandonment of unscaleable alternative routing and addressing arrangements, are only achieved after full (or almost full) adoption by all networks. Yet there is insufficient direct incentive for early adopters for even a fraction of networks to adopt it. A Core-Edge Separation architecture with DITRs, PTRs or some equivalent functionality provides full benefits to all adopters, and so is capable of being widely enough adopted to solve the routing scaling problem. Scalable routing benefits accrue in direct proportion to the number of adopting networks. The problem can be Whittle Expires July 17, 2010 [Page 16] Internet-Draft Ivip Architecture January 2010 substantially solved by widespread adoption. Complete adoption is desirable, but not at all required. Core-Edge Separation schemes typically do not require any changes to hosts - to stacks or applications. They do however involve the creation of at least two items of global infrastructure, before any end-user network can use the system. The first is some kind of mapping system, by which ITRs can decide which ITR to tunnel a packet to, when they receive a packet addressed to an SPI address ("EID" in LISP terminology). The second is one or more sets of DITRs, PTRs or their equivalent. Ivip's technical structure lends itself to business models in which those who construct and run these two types of infrastructure can do so on a potentially profitable basis, by charging end-user networks according to the use they make of the mapping system and of the DITRs. 2.11. Maintenance of existing levels of security and robustness All scalable routing schemes complexify the Internet - so it is unlikely that the goals of not degrading security and robustness to any degree can be fully realized. Only once Ivip is fully designed and carefully analysed can there be a realistic estimation of the security and robustness problems it will entail. However, it remains a goal to minimise and ideally to eliminate any such degradation. Ivip's approach to handling the PMTUD problems inherent in encapsulation is intended to be secure against attacks - such as from spoofed ICMP Packet Too Big messages. Ivip is the only core-edge separation scheme to provide an inexpensive method of ETRs enforcing the source address filtering ISPs may impose on packets arriving at their Border Routers (BRs). Such filtering is imposed to prevent outside attackers spoofing the address of any host inside the ISP's network - and includes dropping packets with private (RFC 1918) source addresses. This is achieved by the simple arrangement of the ITR using the sending host's address as the outer header source address in all the encapsulated packets in the tunnel to the ETR. ETRs simply compare the inner source address with the outer, and drop any decapsulated packets where the two differ. Whittle Expires July 17, 2010 [Page 17] Internet-Draft Ivip Architecture January 2010 This is only for encapsulation. When MHF is used, there is no need for ETRs to perform any such task, since the original packet is sent across the DFZ, with the sending host's source address in the IP header - so BR filtering occurs normally and the ETR never receives a packet which violates these filtering rules. (With encapsulation, when the ITR occasionally probes the PMTU to an ETR, it sends an additional packet with the source address being that of the ITR, but this does not alter the ETR's ability to enforce BR source address filtering.) Global query server CES architectures such as LISP-ALT involve multiple devices, often in distant networks and other countries, handling mapping queries which include the ITR's and the destination host's address. While the ITR doesn't necessarily imply that the query came from a given sending host, nonetheless this information, if it fell into the hands of an attacker, would represent a serious security breach. To prevent this, every ALT router would need to be secured at least as well as DFZ routers. This is not necessarily impossible, but Ivip's global mechanism, the fast-push system of Replicators, does not carry queries or anything else which could reveal sending host or receiving network activity. Authentication of mapping changes at full-database query servers means that an attacker who gains control of one or more Replicators will be unable to alter the mapping of ITRs. Whittle Expires July 17, 2010 [Page 18] Internet-Draft Ivip Architecture January 2010 3. Non-goals 3.1. Complete core-edge separation is not required At least one core-edge separation scheme - APT (which is no longer being developed) - had the goal of completely separating the global unicast address space into two subsets: core and edge. In this scenario, only ISPs would have core addresses and all end-user networks (or perhaps all end-user networks which needed portability, multihoming and TE) had edge addresses. Then, in theory, it would be possible to prevent any host in an edge network from sending packets to the core - which was supposed to provide some security benefits. Ivip does not rely for its benefits (improvements to routing scalability, or the benefits for end-user networks) on complete adoption of SPI (edge) space by all end-user networks, or by the subset of them which want or need portability, multihoming and TE. Ideally, for scalability, the only prefixes advertised in the DFZ would be those of ISPs (including those used to serve many end-user networks with PA space) and the relatively small number of prefixes which encompass the SPI space. "Relatively small" is in comparison to the very large number of micronets these prefixes contain and to the likewise very large numbers of end-user networks which are using this SPI space. The full benefits for end-user networks which adopt SPI space - portability, multihoming and TE - do not depend at all on how many other end-user networks adopt SPI space. The benefit of routing scalability depends on how many end-user networks which need or want portability, multihoming and TE actually do adopt SPI, rather than the two undesirable alternatives of either not getting these benefits, or getting them by the unscaleable method of advertising conventional PI prefixes in the DFZ. In order to maximise routing scalability, the more end-user networks which adopt SPI space, the better. But there is no need or intention to have them all adopt it. A satisfactory outcome for scalable routing would be for some or many of the end-user networks which currently advertise PI prefixes in the DFZ to continue doing so - and for the great majority of all other end-user networks which want or need portability, multihoming and TE to use SPI space instead. Whittle Expires July 17, 2010 [Page 19] Internet-Draft Ivip Architecture January 2010 3.2. Mapping changes need not be free of financial cost It appears that the designers of other core-edge separation architectures have a goal of mapping changes being free of financial cost. This is not a goal of Ivip. Ivip is the only core-edge architecture to contemplate or require mapping changes to be paid for - by the end-user network whose micronet of SPI space the mapping applies to. All other proposals avoid financial costs such as this. In the case of the global query server systems - LISP-CONS, LISP-ALT and TRRP there is no need for payment, since changing the mapping has no direct impact beyond the authoritative query server(s) in which the mapping is changed. (Unless there are provisions for sending mapping changes to particular ITRs which might need it, which may be a part of LISP.) APT, like Ivip, involves pushing all mapping changes to a large number of local full-database query servers, via a flooding technique. In APT, these local full database query servers reside in each ISP and are called Default Mappers. There is no mention of charging for mapping changes, but this raises questions about what disincentives there might be to discourage an end-user network from making large numbers of mapping changes, and so burdening the mapping distribution system and all APT-using ISPs with these changes. Ivip's arrangement for charging end-users for each mapping change, and for each change to the way their SPI space is divided into micronets, is intended to solve a large part of the problem of funding the fast-push mapping distribution system. It is not a complete solution, since it could be argued that there is no incentive for a given ISP's full database query server to keep on receiving and storing mappings for some end-user networks which, for some reason, that ISP's ITRs never handle packets to those end-user networks. Still, this partial solution enables market forces to set the cost of mapping changes - so the end-user networks effectively pay for the cost of delivering their mapping changes most or all of the way to the full database query servers. The cost of changes should be low enough to be a trivial issue in the rare events of multihoming service restoration and portability to another ISP. The cost should also be low enough to make reasonably frequent changes for TE attractive, when it allows significantly better utilization of multiple links to ISPs. It should also be low enough to present no problems for TTR Mobility, whenever mapping Whittle Expires July 17, 2010 [Page 20] Internet-Draft Ivip Architecture January 2010 changes due to the MN moving more than about 1000km. Although mapping changes needs to be transmitted reliably to all full database query servers, within a few seconds, via a specially constructed set of servers, each change is reasonably compact, and they are sent en-masse, multiple changes per packet. So the fast- push network will be much more efficient in terms of bandwidth and CPU resources than the nearest current equivalent - changes to DFZ advertisements percolating slowly, neighbour to neighbour, through expensive DFZ routers. 3.3. No attempt to cope with partially reachable ETRs Ivip's use of a single ETR address in the mapping is different from the use of multiple ETR addresses in the mapping information of all other core-edge separation schemes. This gives rise to a potential benefit of those other schemes which is not a goal of Ivip. Ivip ITRs all over the Net tunnel packets which are addressed to any particular micronet to a single ETR at any one time. (This is ignoring perhaps a second or less when the mapping is changed, and some ITRs receive the cache update from their query server earlier than others.) It is up to the multihoming end-user network to ensure that the mapping changes in a manner which maximises the connectivity of its network during a multihoming service restoration event. For instance, an end-user network has two ISPs ISP-A and ISP-B, and can map its one or more micronets to either ETR-A or ETR-B. Whether the ETRs are in the ISP or at the end-user site is not important. ETR-A's connection to the rest of the Net is via ISP-A and likewise ETR-B's is via ISP-B. In this example, only one micronet is considered, but the same principles apply with multiple micronets. When both ISPs and ETRs are working well - that is to say both ETRs are reachable from any router in the DFZ and both ETRs are connected to the end-user network - the end-user network may have the mapping set to ETR-A. If an external monitoring company (contracted by the end-user network) detects that the end-user network is no longer reachable via ETR-A, then it will issue a mapping change so that the micronet is mapped to ETR-B instead. As long as ETR-B is connected to the end-user network and is reachable from any router in the DFZ, then this is a perfectly good outcome: full connectivity is restored within a few seconds of the mapping change being issued. However, if ETR-B is unreachable from some subset of the DFZ routers (and therefore from a subset of sending hosts in end-user and ISP networks) AND this subset of DFZ routers can reach ETR-A (and assuming ETR-A is still connected to the end-user network), then Ivip Whittle Expires July 17, 2010 [Page 21] Internet-Draft Ivip Architecture January 2010 cannot ensure complete connectivity, since neither ETR is reachable from all other networks, and since Ivip can only steer packets to one ETR or another. (Actually, practical connectivity only concerns the fraction of DFZ routers and other networks which are currently sending packets to this end-user network - but the ideal is that the end-user network is always reachable from all other networks.) Other core-edge separation techniques have a potential advantage in this scenario, since it is possible that all the ITRs which are currently sending packets may be able to discern the reachability of the two ETRs and adapt their tunneling by choosing an ETR which is reachable. In this circumstance, the non-Ivip core-edge separation scheme would be able to restore full connectivity when Ivip could not. However, this set of circumstances - both ETRs being partially reachability and the patterns of reachability being complementary - is likely to be a highly transient state, since the DFZ routers will rapidly adapt their best-paths to restore full connectivity to both ISPs and their ETRs. Also, it cannot be assured or assumed that the non-Ivip ITRs would choose the reachable ETR fast enough to take advantage of this difficult situation. It is possible that a non-Ivip ITR may be able to detect non- reachability of a particular ETR when the Ivip approach would not. In that case, the non-Ivip approach would be superior due to this particular ITR changing its mapping and so retaining connectivity. With Ivip, end-user networks will be able to choose between many Multihoming Monitoring (MM) companies and each company would have a range of options for how frequent the reachability probing occurs, how many servers in the DFZ are used to probe the path via each ETR and how decisions should be made if there appears to be a reachability problem. A MM company with probing servers scattered widely around the Net should be able to detect most reachability problems experienced by in any part of the DFZ, but it can't necessarily detect every one. How the MM company decides which outages to respond to, with a mapping change, is a matter for the company and the end-user network to decide. Ivip's external, user-supplied, detection of reachability problems and creation of mapping changes can be the subject of ongoing innovation and choice, with the intention that it be more effective at restoring full connectivity than the individual, isolated, efforts of non-Ivip ITRs - which have a difficult task reliably and inexpensively testing reachability to various ETRs. This is particularly the case if tens or hundreds of thousands of ITRs are tunneling to one ETR. Such ITRs may not actually probe reachability Whittle Expires July 17, 2010 [Page 22] Internet-Draft Ivip Architecture January 2010 of ETRs with ping or the like, but rely on ICMP messages due to traffic packets not reaching the ETR. A difficulty with this is that ICMP messages may be lost, and are not necessarily always generated if there is an outage. Furthermore, it is costly for ITRs to be able to securely distinguish genuine ICMP messages from spoofed packets. 3.4. No attempt to avoid all the mapping data being stored at any one location The global query system core-edge separation schemes (LISP-CONS, LISP-ALT and TRRP) seem to be predicated on the belief that it is either impossible, or at least extremely undesirable, to require the full set of mapping data to be sent to or stored at any one location. The attraction of this position is that these architectures are not prevented from handling very large numbers (billions) of EID prefixes (SPI micronets in Ivip terminology) due to any constraints of storage, CPU power or transmission bandwidth. Since it is clearly feasible to transmit and store numbers of mappings such as 10^6 to 10^8 to single servers or routers, these architectures are presumably intended to scale to billions of EIDs. Of these three, only LISP-ALT is being actively developed. After two years development, questions about its ability to scale to these numbers have not yet been resolved. [LISP-ALT-Critique] In early 2010, version 07 of draft-lear-lisp-nerd claims that NERD (each ITR downloads the full mapping database) will scale to at least 10^8 EIDs. There is no work yet which shows how LISP-ALT could scale to such numbers. Ivip and APT both incur significant costs and technical challenges moving the mapping changes to large numbers of full-database query servers and storing a copy of the complete mapping database (as it is constantly updated) in each such server. Ivip does not have a goal of avoiding this transmission and storage, of as many as 10^10 mappings, for two reasons: Firstly, by the time such adoption occurs, it will not be too difficult to store this data in RAM. (Consumer hard drives today already provide sufficient storage.) Secondly, the costs of transmission and storage are judged to be a worthwhile price for the benefits of local full-database query servers: ITRs getting mapping reliably and without significant delays - and so not leading to any lost or significantly delayed packets, which are a problem with the global query server approaches. So this central goal of some other core-edge separation schemes is not a goal of Ivip. Whittle Expires July 17, 2010 [Page 23] Internet-Draft Ivip Architecture January 2010 3.5. It may not be possible to completely eliminate unfair burdens Another motivation for the global query server core-edge separation schemes seems to be avoiding all "unfair burdens". Unfair burdens are central to the current routing scaling problem - that any end- user network with PI space can advertise it, as one or more prefixes, in the DFZ and so burden all DFZ routers with the task of maintaining a best path to each such prefix. This is a massive unfair burden problem, since it falls on all ISPs and some large end-user networks. These organisations who suffer the cost, including the need to buy more expensive routers, have no reasonable way of preventing these actions by end-user networks, or of collecting payment from them to offset the very real burdens which are imposed. Local query server systems (APT and Ivip) do have an "unfair burden" problem in that every network which operates full database query servers needs to pay for bandwidth and storage for every mapping change in the entire system. Yet only a subset of those mapping changes will ever be used in that network, since hosts in that that network are only sending packets to a subset of SPI micronets (EID prefixes in APT terminology). It is possible to imagine an ISP in Nova Scotia analysing the mapping changes and determining that 10% of them come from some range of SPI addresses, or some country, which the ISP never sends packets to. More generally, for almost any ISP in the world, the great majority of mapping changes are not needed - even if there is no discernable pattern. Ivip does not have a goal of completely avoiding this particular kind of "unfair burden", for several reasons. Firstly, the bandwidth used by the incoming mapping changes and the RAM or hard drive space required to store them is expected not to present significant technical or cost difficulties, in the context of hardware development and general traffic volumes. A fully deployed system with up to 10^10 IPv6 micronets would involve considerable storage. Each mapping would consist of 64 bits for the starting /64 prefix of the micronet, up to 64 bits for the length of the micronet, also in units of /64s - and then 128 bits for the ETR address. This is 32 bytes per mapping - a total of 320 gigabytes. By the time such levels of adoption arise, there should be no difficulty fitting this in RAM on a COTS (Commercial Off The Shelf) server. Secondly, to the extent that this unfair burden is a problem for ISPs, it is a far better alternative to solve the scalable routing problem in this manner than to attempt to solve it with a global query server architecture. Such architectures inevitably delay Whittle Expires July 17, 2010 [Page 24] Internet-Draft Ivip Architecture January 2010 and/or drop a significant proportion of initial packets and so would be difficult or impossible to have adopted widely - and would degrade application responsiveness if they were widely used. Thirdly, there is scope for market-based solutions to this unfair burden - such as the originators of mapping changes paying all ISPs to accept and use their changes. Global query server architectures attempt to avoid this unfair burden problem, but create other problems of the same kind - such as difficulties financing the expensive and crucial global query server system in an environment where queries cannot be charged for, and in which mapping changes likewise incur no fee. Ivip's goal is to minimise unfair burdens and to eliminate them where possible. It appears impossible to devise a scalable routing system which completely avoids unfair burdens, without creating serious technical difficulties or barriers to voluntary adoption, such as the need to rewrite host stacks and applications. 3.6. No attempt to mix IPv4 and IPv6 Ivip for IPv4 is intended to be a free-standing system completely independent of Ivip for IPv6. An IPv4 ITR could be implemented in the same server or router as an IPv6 ITR - just as ITR, ETR and query server functions could be performed in the one device. Likewise, the mapping distribution systems for IPv4 and IPv6 are intended to be separate and independent - but there's nothing to prevent one server being a Replicator for both systems. 3.7. Not Locator - Identifier Separation There is considerable terminological inexactitude regarding the use of the term "Loc/ID Separation". True Locator - Identifier separation involves hosts handling packets using two addresses of different types, usually called Locator and Identifier, which therefore are in different namespaces. If both types of address are numeric and a Locator and an Identifier were numerically identical they would refer to different things because this numeric value has different meanings in each namespace. (Further discussion of the meaning of "namespace" is at: [Namespace] ). HIP and ILNP [I-D.rja-ilnp-intro] are examples of Locator / Identifier Separation. LISP (Locator/Identifier Separation Protocol), Ivip, APT, TRRP and TIDR are not. An architecture which uses FQDNs as Identifiers and IP addresses Whittle Expires July 17, 2010 [Page 25] Internet-Draft Ivip Architecture January 2010 (always PI, to ensure scalability) as Locators is also an example of true Loc/ID separation - for instance Name-Based Sockets [Vogt-2009]. LISP, Ivip and other Core-Edge separation architectures do not present hosts with separate Locator and Identifier addresses. The host sees only IP addresses, which perform both functions simultaneously - just as they do without core-edge separation. ITRs are the only devices which treat packets differently if their destination address is in the "edge" subset of the global unicast address range. The full arguments about why Core-Edge Separation cannot correctly be construed as "Locator / Identifier Separation" are at: [loc-id-sep-vs-ces] Whittle Expires July 17, 2010 [Page 26] Internet-Draft Ivip Architecture January 2010 4. Architectural Choices 4.1. Core-edge separation rather than elimination 4.1.1. Core-Edge Elimination (CEE) architectures Core-edge elimination (CEE) involves hosts dealing with two kinds of address: Identifiers and Locators. Other terminology may be used, such as "host identifier" or "logical address" instead of "Identifier" and "physical address" instead of "Locator". The simplest adaptation of existing protocols is to retain IP addresses as Locator addresses and develop a separate namespace for the Identifier addresses. Each host retains its one or more Identifiers, no matter which one or more Locator addresses it is using. The Locator addresses are global unicast addresses which are supplied by ISPs as PI space. The simplest form of multihomed end-user network would gain a PI prefix from each of its ISPs and each of its hosts would use one address from each prefix as a Locator address. Each such prefix is part of a larger (in terms of number of addresses - shorter in terms of prefix length) prefix the ISP advertises in the DFZ. The ISP can split one such advertised prefix into many smaller (longer) prefixes for multiple end-user networks. This solves the routing scaling problem because the total number of large (short) prefixes advertised by all ISPs is scalable, whereas - if not for the core-edge elimination scheme - the number of prefixes advertised in the DFZ by multihoming end-user networks would be an unacceptable burden on all DFZ routers and on the entire DFZ BGP control plane. Applications connect to other hosts solely in terms of their Identifier addresses. It is the task of each host's stack to adapt to changes in other hosts' Locators, and to inform other hosts which need to know about this host's changed Locators. The Identifier addresses may be numeric or have some other form, and there is typically a DNS mapping from FQDNs to one or more Identifier addresses, just as there are to IP addresses today Some key points about core-edge elimination architectures include: 1. Identifiers are from a completely different namespace than Locators. If both are numeric, and a Locator is numerically equal to an Identifier, there can be no confusion about the separate entities each refers to, since the Identifier is interpreted in a different namespace from that used for Locators. Therefore, if IP addresses are used as Locators, IP Whittle Expires July 17, 2010 [Page 27] Internet-Draft Ivip Architecture January 2010 addresses cannot be used as Identifiers. 2. Host stacks are responsible for choosing which of a correspondent host's Locators to send a packet to. This work is not done by network elements, such as routers. (However some CEE architectures may have routers alter part or all of the outgoing destination address, or perhaps source address, to exert-network centric control over traffic flows.) 3. While there is typically a global, decentralised mapping system by which hosts can use another host's Identifier (perhaps in combination with one of its Locators) to look up that host's complete set of one or more Locators, the network itself remains simple and hosts take on more responsibilities than they have with existing IP protocols. This is regarded as a virtue by many people, and represents an extension of TCP/IPs "dumb network, smart end-points" approach, especially when compared to the telephone network. 4. Since applications need to work with a different kind of address element than an IP address for establishing and maintaining communications with other hosts, the host stack, its API and applications themselves need to be substantially rewritten in order to be able to work with a CEE architecture. 5. While it may be possible to slowly introduce such an architecture, the benefits of portability, multihoming and TE only apply to packets sent between hosts using the new system. 6. CEE architectures are subject to the critique that the extra management packets which hosts must send and receive as part of the new system is likely to create extra costs, delays and/or unreliability compared to current IP techniques. 7. This critique can be extended to argue that mobile hosts, due to their typically slow, not-necessarily reliable and potentially costly wireless links are especially impacted by these new responsibilities. 8. Core-edge elimination schemes typically do not apply to IPv4 and so are based on IPv6 or on entirely new arrangements. Points 4 and 5 constitute insurmountable barriers to the adoption of CEE architectures, since adoption must be very widespread, within a period of years, rather than decades, and since adoption must occur on a voluntary basis. [Constraints-Voluntary] Point 6 is an argument that while CEE architectures are theoretically Whittle Expires July 17, 2010 [Page 28] Internet-Draft Ivip Architecture January 2010 elegant and simple, the facts of delay and loss of packets across global query server systems such as DNS, or whatever mapping system is used to securely determine a host's full set of Locators will contribute to delays in sending application packets. Also, if the two hosts have to exchange management packets with each other, for authentication purposes, before any application packets can be sent, then this will slow down the establishment of communications - especially if the hosts are far apart, on high latency links or if packets are lost. Point 7 implies that in order to create a network which performs best, given the vagaries of slow and unreliable last-mile links, all hosts should not have to perform these additional Routing and Addressing management functions - that such functions be handled by better-connected devices, such as routers in ISPs' data-centers. [Host-Responsibilities] The only existing routing scaling problem is in the IPv4 Internet. In early 2010 the IPv4 DFZ has about 300k prefixes with a doubling time of about 4.5 years. The IPv6 DFZ has about 855 prefixes - 1/350th the IPv4 number. Even if IPv6 prefix numbers had a doubling time of 1.0 years, it would be mid 2018 before the number reached current IPv4 levels - which are not yet unworkable. IPv6 adoption rates have consistently disappointed IETF expectations. Despite the run-out of unallocated IPv4 space, there is no sign yet that large numbers of existing users can have their Internet needs adequately served via IPv6 addresses alone. For the reasons described in points 4 to 8, Ivip instead adopts a core-edge separation approach. 4.1.2. Core-Edge Seperation (CES) architectures Ivip uses a Core-Edge Separation (CES) Architecture. CES does not involve the creation of new namespaces and does not require any changes to host stacks or applications. A subset of the global unicast address space is converted to a new type of address which, in Ivip, is known as Scalable PI (SPI) space. This subset will consist of a growing number of prefixes, each of which is advertised in the DFZ. Within each such prefix, the SPI space can be divided up amongst many (thousands to potentially millions) of separate end-user networks. If a network gains more than one basic unit of address space - an IPv4 address or an IPv6 /64 prefix - it can divide this space into multiple separately mapped "micronets". As more and more space is converted for use as SPI space, this "edge" Whittle Expires July 17, 2010 [Page 29] Internet-Draft Ivip Architecture January 2010 space grows to become a significant fraction of the total global unicast space. There must always be some conventional, non-SPI, space, since ETRs must be located on such addresses. There are many uses of space within ISPs which do not need to be on SPI space - including the large numbers of IPv4 addresses, or in the future IPv6 /64s, which are used for individual home and SOHO customers. Each such customer gets what is effectively a small (long) prefix of PI space, which is suitable for their purposes because they do not want or need portability, multihoming or TE. As noted in the non-goals section, Ivip does not require or aim for complete conversion of all end-user networks to SPI space. Many will be happy with existing PI arrangements, and some larger existing end- user networks with their own (unscaleable) PA prefixes will probably retain their current arrangements. Nonetheless, SPI space is intended to be attractive to all end-user networks, including the largest corporations, universities and government departments. CES involves the progressive repurposing of existing address space. It does not involve the creation of any separate namespaces. If "separation" in "Locator/Identifier separation" means separate namespaces, then this term only properly applies to core-edge elimination schemes - which do exactly this. CES can be introduced gradually, and with DITRs (or their LISP equivalent - PTRs) the benefits of portability, multihoming and TE can be supported for all packets sent to the adopting end-user network. Therefore 100% of traffic receives these benefits, in contrast to CEE architectures where only the subset of traffic originating from other upgraded networks has these benefits. Assuming a CES architecture does not significantly reduce performance, robustness or security - and if it provides significant and immediate benefits to all adopters - then it meets the constraints due to the need for widespread voluntary adoption. [Constraints-Voluntary] All CES architectures I am aware of do not require hosts to perform additional work to manage routing and addressing. So no CES architecture is subject to the critique which applied to CEE architectures, particularly with reference to mobile hosts. [Host-Responsibilities] The historical roots of Core-Edge Separation architectures can be found in the mid-1990s - Steve Deering's "Map & Encap" for IPv4 [Deering-1996], Robert Hinden's "New Scheme for Internet Routing and Addressing (ENCAPS) for IPNG" (RFC 1955) and the 1992 crocker-ip- encaps-01.txt. Whittle Expires July 17, 2010 [Page 30] Internet-Draft Ivip Architecture January 2010 4.2. Local query servers Probably the greatest challenge for a core-edge separation scheme is how to ensure ITRs can securely, reliably and rapidly obtain the mapping they need in order to be able to decide which ETR to tunnel a packet to. There are three basic approaches to this problem: 1. The complete global set of mapping changes is sent to each ITR, which maintains an up-to-date copy of the full mapping database. 2. Local full-database query servers are located in ISP networks and potentially in end-user networks in which ITRs are based. The complete global set of mapping changes is sent to each such query server, which maintains an up-to-date copy of the full mapping database. ITRs query one or more of these and so obtain mapping quickly and reliably. 3. No site or device stores a complete copy of the global mapping database. Instead, there is a global network by which ITRs can send query to the authoritative query server for the particular micronet of addresses which match the destination address of the packet the ITR needs to tunnel. The only architecture to propose option 1 was LISP-NERD. This is widely regarded as scaling poorly with large numbers of end-user networks. LISP-NERD was to be retired, but a new version 07 ID appeared in early January 2010. [I-D.lear-lisp-nerd] Ivip and APT use option 2. In Ivip, the local full database query servers are called QSDs. In APT, they are called Default Mappers, and also handle the encapsulation of some packets. All other core-edge separation schemes to date use option 3. The most prominent examples are LISP-CONS [I-D.meyer-lisp-cons], LISP-ALT and TRRP. The global query server network approach has obvious advantages in terms of there being no hardware-imposed limit to the number of query servers or end-user networks which can be supported. Furthermore, changes to mapping impose no direct burden on any other devices - whereas for option 1 or 2, information must be sent to potentially hundreds of thousands of devices all around the world. However, global query server systems pose apparently insoluble problems of delay and potential unreliability - due the delays and Whittle Expires July 17, 2010 [Page 31] Internet-Draft Ivip Architecture January 2010 risk of packet losses which are inherent in their global nature. Furthermore it seems to be impossible to make these systems scale to the very large numbers of EIDs required for ubiquitous mobile adoption. [LISP-ALT-Critique] Local full-database query servers are the clear choice for Ivip. Even the largest imaginable mapping databases and the highest rates of change due to TTR mobility should not present insurmountable problems in terms of hardware, software and communications bandwidth, by the time such large numbers (10^9 to 10^10) of end-user networks are using the system. 4.3. Real-time mapping distribution If the mapping is distributed slowly to the QSDs - or for any other reason, can't be sent to all ITRs which need it within a few seconds - then ITRs need to make their own decisions about which ETR to use. Therefore, the mapping needs to include multiple ETR addresses, and ITRs need to be much more complex to probe reachability (and/or somehow detect non-reachability with encapsulated traffic packets) and make the best choice between multiple ETRs. By getting mapping changes to ITRs in real-time - within a few seconds at most - Ivip achieves several major benefits. Firstly, the mapping information can be more compact, since only a single ETR address is needed. Secondly, ITRs can be much less complex, and do not need to do any reachability testing. Thirdly, the real time control of all ITRs which is given to end-user networks modularly separates the reachability, multihoming service restoration and TE decision making systems from the core-edge separation scheme itself. Mapping needs to be delivered in a secure and robust manner, which presents some challenges in an essentially real-time global system. There may be multiple ways of achieving this, but the currently planned system for pushing all mapping changes to QSDs looks perfectly feasible. The second stage of delivering mapping to ITRs is that QSDs are able to securely send mapping updates to all their client ITRs which need it. This would occur a small fraction of a second after the QSD received a mapping update from the global fast-push system. 4.4. SPI address management Traditional IP techniques divide address space into binary boundary prefixes. Ivip uses traditional prefixes for the largest unit of SPI space - the "Mapped Address Block" (MAB). The smaller divisions of this do not use prefixes or binary boundaries. The units of dividing Whittle Expires July 17, 2010 [Page 32] Internet-Draft Ivip Architecture January 2010 SPI space are IPv4 addresses and IPv6/64s. A MAB is a prefix of address space which is devoted to use as SPI space. The single MAB is advertised in the DFZ, by all the DITRs which are located around the Net, attracting packets addressed to any address in the MAB. (It would also be possible to load share the MAB between multiple DITRs, each advertising a segment of it, but in general complete MABs will be advertised.) For instance, an IPv4 MAB may be 11.22.0.0/16. A MAB might have previously been conventional PI space of an end-user network, and may now be used exclusively by this end-user network. In this case, it will presumably be used to serve the needs of many sites within this network, so achieving routing scaling by removing the need to advertise each such smaller (longer) prefix in the DFZ. Most MABs will be operated by specialised companies - perhaps ISPs but not necessarily a company which actually provides Internet connectivity. The MAB operating company typically acquires rights to multiple prefixes of global unicast space, advertises each of them in a global system of DITRs and then rents out smaller portions of the MABs, on a yearly basis, to a large number of end-user networks. Each end-user network rents a section of the MAB - a User Address Block (UAB). One end-user network might rent multiple non-contiguous UABs in the one MAB, and multiple UABs in multiple MABs. For simplicity, the following discussion assume they rent a single UAB, such as: 11.22.33.84 to 11.22.33.95 inclusive. This is an 18 IP address UAB. UABs could be as small as a single IPv4 address or IPv6 /64 or could be very large, including as large as the MAB itself. The end-user network which rents this UAB is responsible for generating mapping changes to suit its needs - and for multihoming would typically hire a Multihoming Monitoring (MM) company and give them the credentials required to control the mapping. The end-user network can split their UAB up as they wish into typically smaller sections, known as "micronets". (Bill Herrin first used this term in TRRP.) A micronet is a contiguous set of any positive number of IPv4 addresses or IPv6 /64s which fit within the one UAB. This 18 IP address UAB could be used as a single 18 IP address micronet, or it could be split in any way - such as into as many as 18 single IP address micronets. Each micronet is covered by a single Ivip mapping - it is mapped to a single ETR address. MABs and micronets are important to ITRs and most of the mapping Whittle Expires July 17, 2010 [Page 33] Internet-Draft Ivip Architecture January 2010 system. UABs are not needed for these, but are an administrative construct of SPI space which an end-user network is authorised to change the mapping for. The company which runs the MAB would provide a method by which the end-user network, or some other company it authorises, can change the mapping and the division of the UAB into micronets quickly and securely. This would involve the end-user network having complete control, but being able to give a username and password to another party such as the MM (Multihoming Monitoring) company, by the MM company could control the mapping of some or all of the end-user's UAB space. The technical and administrative arrangements for this are described in [I-D.whittle-ivip-db-fast-push]. For each mapping change and each change to the division of the UAB into micronets, the end-user network would incur a fee from the MAB company. The MAB company would charge fees for renting the UAB space, and for the load placed on the DITRs which cover this MAB. The MAB company may run its own DITRs, or may contract this out to another company which specialises in this service. As long as there are significant numbers of hosts in networks without ITRs, it will be an important part of the MAB company's service to locate DITRs in all corners of the Net, to ensure good load sharing between them and to minimise the total path length from the sending host to whichever ETR the end-user network chooses to map their Micronet to. This flexible integer-based approach to dividing SPI space is intended to maximise the efficiency with which it is can be used. Since a single physical site, such as a branch office, may be able to operate perfectly well on one or a few IPv4 addresses, or on a single IPv6 /64, a seemingly small UAB of 18 IPv4 addresses could be used to serve the needs of as many branch offices. Each such site could be multihomed with two or more local ISPs. As fresh expanses of IPv4 space disappear, there will be continuing pressure to slice and dice the address space more finely so it can be used by more and more ISPs and end-user networks. However, the convention in the DFZ is not to propagate prefixes longer than /24. This 256 IP address granularity inherent in the current arrangement leads to considerable underutilization of space. With SPI address able to be sliced and diced freely in the smallest possible increments, a much greater utilization can be expected, in a scalable fashion, than is possible with current techniques. Whittle Expires July 17, 2010 [Page 34] Internet-Draft Ivip Architecture January 2010 4.5. IP in IP encapsulation When encapsulation is used, there is a simple IP-in-IP header. There is no need for ITRs to communicate with ETRs, except for the purpose of PMTUD management. So, when the ITR tunnels traffic packets ordinarily (in all cases except for the special Path MTU measurement protocol, which is only used rarely) there is no need for a UDP header to enclose a special header with extra information. Architectures with slow mapping distribution and which therefore require ITRs to choose between multiple ETRs typically require the ITRs and ETRs to communicate - but this is not needed for Ivip. 4.6. MHF initially or in the long term to avoid encapsulation and PMTUD problems Both the IPv4 and IPv6 headers have un-used bits which can be employed to direct the packet from ITR to ETR. This path is primarily across the DFZ but typically includes routers inside ISP and end-user networks. These routers need to be upgraded - and in the long-term this can be done without significant cost, simply by building the new capabilities into new routers and implementing it in firmware updates. 4.7. Outer header address is that of the sending host When encapsulation is used, it seems natural to use the ITR's address as the outer header's source address. This is consistent with traditional tunneling, and ensures the ITR gets any ICMP messages, including especially Packet Too Big (PTB) messages. There are two problems with this conventional approach, which is used by LISP and other CES architectures. Firstly, it is very expensive for the ITR to securely respond to PTB messages. Secondly, this approach means that any ISP BR filtering (dropping) of incoming packets according to their source address will not affect the packets at the BRs and must be replicated in the ETR. For more than a few such blocked prefixes, this is extremely expensive too - and we want ETRs to be as simple as possible. The answer is to have the ITR use the sending host's source address in the outer header of the encapsulated packet. All ITRs will therefore generate packets with identical inner and outer source addresses. ISP BR filtering will drop the packets with source addresses matching any prefix inside the ISP's network and the ETR will never need to handle such packets. The ETR needs to enforce this in the case where an attacker sends a packet to the ETR, with an inner packet having a banned source Whittle Expires July 17, 2010 [Page 35] Internet-Draft Ivip Architecture January 2010 address and the outer header having a source address which is allowable. This enforcement is achieved by the ETR performing simple logic on each decapsulated packet: If its source address does not match the outer header's source address, the packet is dropped. This arrangement of the outer source address being that of the sending host requires a novel approach to Path MTU Discovery management. 4.8. IPTM (ITR Probes Tunnel MTU) PMTUD management As long as encapsulation is used, there needs to be a method of informing sending hosts, via traditional RFC 1191 techniques of what length packet to send, so that once encapsulated, these packets may reach, but not exceed the MTU of the path between the ITR and ETR. This is true of any core-edge separation architecture which uses encapsulation. It is a complex topic and there is a solution, but it requires considerable thought and significant complexity in all ITR and ETR. PMTUD management occurs naturally via RFC 1191 mechanisms for DF=1 traffic packets if the router with the too-small MTU is between the sending host and the ITR, or between the ETR and the destination host. Without encapsulation - with MHF - packet lengths are not increased in the ITR to ETR "tunnel", and the modified routers in this path will convert a too-long packet back to its original IP header format, before passing it to the ICMP PTB algorithm. The difficult task is to make PMTUD work for the path between the ITR and ETR, where the original packet is encapsulated. I intend to write up IPTM in an ID. For now, the fullest description is on a web page. [PMTUD-Frag] Here is an overview of the process, which is much the same for IPv4 and IPv6. This system involves restrictions on the length of IPv4 DF=0 (fragmentable) packets which are accepted by this system. It is reasonable to expect applications not to generate such packets, which place a serious burden on the network of they are too long. Google servers have been observed sending 1470 byte DF=0 packets. [DFZ-unfrag-1470] Such companies could presumably be persuaded to refrain from sending DF=0 packets altogether by the time a scalable routing solution is deployed. In the long-term, with EAF in place of encapsulation for IPv4, fragmentable packets addressed to SPI addresses will be dropped by all ITRs. A simple approach to PMTUD management would be to choose some packet length, marginally below 1500 bytes and require all ITRs to accept only packets which are the encapsulation overhead number of bytes Whittle Expires July 17, 2010 [Page 36] Internet-Draft Ivip Architecture January 2010 shorter than this. Longer packets would cause the ITR to generate a PTB and the sending host would send a suitably shortened packet instead. This would be simple and perform reasonably well in today's DFZ, where the Path MTU can reasonably be assumed to be 1460 bytes or more. However, such a scheme would fail to take advantage of jumboframe sized MTUs whenever they appear in the DFZ. ITR to ETR MTUs of around 9k bytes are likely to become more and more prevalent as more routers adopt Gigabit Ethernet interfaces, which handle these large packets. The encapsulated packet has the sending host's source address. If such a packet reached a router with a next hop MTU which was longer than the packet, the router would transmit a PTB to the sending host. However, the sending host should ignore it, since the destination address in the enclosed packet headers will be that of the ETR, not of the destination host - and the rest of the enclosed headers will not match the packet it sent. Also, the MTU figure in the PTB is higher than the figure the sending host needs to adhere to. So the challenge is for the ITR to generate RFC 1191 PTBs when necessary, in an inexpensive and secure manner, whilst adapting to potentially higher or lower MTUs to the ETR due to routing path changes - while making full use of jumboframe paths if and when they exist. Security in this case means being immune to spoofed PTBs - a single one of which could greatly reduce the MTU for all traffic from the ITR to a given ETR for at least ten minutes. A careful decision will be made to assign a value such as 1200 bytes to a globally agreed constant MPMTU (Minimum Path MTU). Once set, this value must remain agreed to indefinitely. A BCP would require all DFZ routers, and all routers between the DFZ and any ITR or ETR (and of course the links between these) to handle packets of this length. Any packets, which once encapsulated and so ENCAPS (Encapsulation overhead - 20 bytes for IPv4 and 40 for IPv6) bytes longer, have lengths less than or equal to MPTU are encapsulated without any extra processing. No PMTUD problems exist for these packets. For any packet longer than this, assuming the ITR has not yet probed the PMTU to its ETR, the ITR performs some special processing. The packet itself is split into two sections and two packets are sent to the ETR as part of the ITR's attempt to probe the MTU to this ETR. One packet uses UDP encapsulation to convey a nonce, some flags and most of the traffic packet - with the ITR's address in the outer header's source address. This long packet is exactly the same length Whittle Expires July 17, 2010 [Page 37] Internet-Draft Ivip Architecture January 2010 as the original packet would be once encapsulated. If this exceeds the PMTU to the ETR, then the ITR will be sent a PTB. Assuming this is received, the ITR will determine a new MTU to send in the PTB to the sending host. This process will repeat until the sending host's packets, once encapsulated, no longer exceed the MTU of the path to the ETR. IPTM does not rely on these PTBs. The ETR is instructed, in a shorter packet to report to the ITR whether the long packet arrives or not - and the ETR repeats this report for a while until it is acknowledged. The long packet is accompanied by one or more copies of this shorter packet, which contains a matching nonce, flags and the remainder of the traffic packet. The shorter packet has the sending host's address in the outer header, so ISP BR source address filtering is still enforced. The effect is that as the sending host (or multiple sending hosts whose packets must be tunneled to the one ETR) tries longer and longer packets, the ITR narrows its "zone of uncertainty" (cue Hammond organ, with reverb and ghostly sounds . . .) about the true MTU to this ETR. If the traffic packets necessitate it, the ITR will exactly determine the MTU, and so be able to stop probing it for a while and send PTBs to sending hosts which generate packets which, once encapsulated, would be longer than this reliably determined MTU. Further elaborations are required for the ITR to adapt to changing conditions and discover longer or shorter MTUs. Without some kind of PMTUD system, core-edge separation architectures cannot use encapsulation. These techniques will require further design work and extensive testing, but are more secure and less expensive than the only other obvious alternative - using the ITR's address in outer headers and having the ITR maintain a large cache of details about recently sent "long" packets, in order that it can securely accept PTBs if they are too long. Whittle Expires July 17, 2010 [Page 38] Internet-Draft Ivip Architecture January 2010 5. Architectural Elements 5.1. ITRs 5.1.1. Types of ITR and their addresses The ITR function can be implemented in a traditional hardware-based router, in a COTS (Commercial Off The Shelf) server, or as a piece of software in a sending host. The functions are much the same, but an ITR in a sending host does not advertise anything in a routing system - it simply handles outgoing packets which are addressed to any MAB. If an ITR is built with software and a COTS server, it doesn't need to be a "router" in most ordinary respects. For instance it doesn't need multiple interfaces. It may have a single Gigabit Ethernet link and advertise MABs in the local routing system, forwarding its encapsulated packets to a router to be forwarded like any other packet. An ITR could be built into a DSL, HFC cable, fibre or WiMax / 3G router. However, it is probably best to do this only when the ITR function is on a reliable, fast, inexpensive link. Most wireless links are not like this and it would be better to let SPI packets flow out of the link, and be handled by ITRs in the ISP network, which have fast reliable paths to local query servers. An ordinary ITR (not in a sending host, and not a DITR in the DFZ) is a device within an ISP or end-user network which attracts packets addressed to SPI addresses. It may do this by advertising every MAB - so the only packets forwarded to it, other than those addressed to the DITR itself, are those addressed to SPI addresses. Alternatively, if the ITR is a true router (hardware or software) it my advertise the entire address space and so be forwarded all packets not addressed to prefixes advertised by local routers. Then, it would encapsulate packets which are addressed to SPI addresses and forward all other packets according to its ordinary router functions. The ITR's address - the address it uses for tunneling packets from, and which is used for communication with the ETR for PMTUD management - may be on conventional global unicast space or, if in an end-user network, on SPI space. This address is also used for communication with local query servers and for receiving PTB messages. Here is a description of what happens when a sending host in an ISP network, such as a QSD, on the ISP's conventional address space, sends a packet to a host in an end-user network on an SPI address - in this case an ITR or ITFH. The packet will go to an ITR in the ISP network (if the QSD doesn't have an ITFH installed already) and then Whittle Expires July 17, 2010 [Page 39] Internet-Draft Ivip Architecture January 2010 will be tunneled to the ETR for this end-user network. This ETR sends the packet to the SPI-addressed host, in this case an ITR or ITFH. When MHF is used, there is no PMTUD management, no interaction with ETRs and no trace of the ITR's address in any outgoing packets. However, the ITR still needs an address for communicating with local query servers. 5.1.2. DITRs - Default ITRs in the DFZ DITRs are "Default ITRs in the DFZ". This first use of "Default" is different from the use of "Default" in "Default Free Zone". This term looks nonsensical when expanded fully. The initial "Default" means that this ITR acts as one of (typically) many other such ITRs, all of them outside ISP and end-user networks. These DITRs advertise MABs from many places in the DFZ and so form multiple destinations which are the "default" - what happens to the packet if nothing else happens (it does not go into any other ITR before reaching the DFZ. In principle, a DITR could advertise every MAB, or be an otherwise normal DFZ router and encapsulate every packet which is forwarded to it which is addressed to an SPI address. However, there is a burden of work looking up mapping, encapsulating packets and on occasions handling the PMTUD management functions to ETRs, which involves sending PTBs to sending hosts. It is unlikely that anyone running a DFZ router would want their device to do more work, unless they are paid for it by the beneficiaries. The beneficiaries of DITRs are the end-user networks which the packets are addressed to. The most likely arrangement for DITRs is that the MAB companies who charge end-user networks rent for their SPI space will also run DITRs themselves or contract specialised companies to run DITRs all over the Net for them. In this scenario, a DITR would advertise only those MABs of the MAB companies who are paying the operator for this service. MAB companies would charge their SPI-renting end-user network companies for the traffic handled for their networks by DITRs, so DITRs in general would need to sample traffic reliably and generate reports in a form which would enable the MAB companies to bill their customers fairly. Only DITRs need this traffic sampling capability. Other ITRs would have monitoring and management functions, but would not need to collect usage statistics for billing. Theoretically, DITRs could advertise all MABs and so handle packets addressed to every MAB. In practice, I expect DITRs will usually Whittle Expires July 17, 2010 [Page 40] Internet-Draft Ivip Architecture January 2010 only handle packets addressed to specific MABs. Other ITRs, including those in sending hosts, will handle packets addressed to any MAB. Consequently, these non DITR ITRs all need a reliable method of downloading the latest set of MABs. They will do this as part of discovering and communicating with their one or more local query servers. DITRs may be implemented in hardware based routers, or in COTS servers. They are always located on conventional global unicast addresses - never on SPI addresses. DITRs are likely to be busy, so it makes sense to locate them in major datacenters, close to one or more full database query servers. 5.1.3. Modified Header Forwarding - MHF-only ITRs Ivip for IPv4 and for IPv6 separately may or may not begin with encapsulation. If it does, then all ITRs and ETRs will also be capable of transitioning in the future to using MHF. The MHF techniques are discussed in a later section, but involve much less processing than encapsulation. With MHF, there is no need for PMTUD management. 5.1.4. Encapsulation and PMTUD management When the ITR function is implemented in software - either inside a sending host, or in a COTS server, it will be relatively straightforward to write C code or the like to implement the functions of analysing the packet's destination address, deciding whether to encapsulate it or not, deciding which ETR address to encapsulate it to, and encapsulating it. Once encapsulated, the new packet is presented to the internal packet handling functions and forwarded normally. This packet-handling code also needs to consider the length of the packet, with reference to a small set of variables it maintains for the ETR the packet will tunneled to. So the packet's destination address would firstly be used to find an ETR address. Generally, this would be found by reference to the ITR's cached mappings, but for initial packets in a new communication flow, the packet must be held for a few milliseconds or tens of milliseconds while the ITR retrieves the mapping information from its one or more local query servers. Once the destination ETR address is known, the length of the packet is considered. If it is less than some constant, it can be encapsulated and sent without any further processing. If it is longer than this constant, then the ITR needs to perform PMTUD Whittle Expires July 17, 2010 [Page 41] Internet-Draft Ivip Architecture January 2010 management functions. In this case, the ITR establishes, or has already established, some variables for this ETR. These include an upper and a lower estimate of the MTU to this ETR. If these are different, then there is a "zone of uncertainty" about the MTU. If they are equal, then the ITR has already reliably established the MTU. If the packet length, plus the encapsulation overhead, exceeds the range of possible MTU values the ITR has previously determined for the path to this ETR, then the ITR will send part of the packet back to the sending host in an ICMP PTB message. If the encapsulated length would be less than the lower limit in the "zone of uncertainty" then the packet can be encapsulated without further processing. If the encapsulated length falls within the "zone of uncertainty", then the ITR emits two packets - a long one and a short one - and communicates with the ETR in a way which will usually raise the lower limit of this zone, or lower the upper limit. In the former case, the ITR is able to determine that the encapsulated length did not exceed the MTU and that the ETR received it correctly. The traffic packet's contents are mainly contained in the long packet, which has the same length as the traffic packet would have had if encapsulated. The remainder of the traffic packet is conveyed in a short packet, of which perhaps a few will be sent. This is non-trivial process, which involves the ETR in some work - but it only occurs for packets whose encapsulated length falls within the "zone of uncertainty". Except for rare error conditions, each such operation reduces the size of the "zone of uncertainty" - and typically the zone will be reduced to zero. Once this occurs, at least for the next 10 minutes or so, the ITR need not perform any such probing of the MTU. Every encapsulated packet which is to be sent to this ETR will be either shorter than the MTU, in which case it is encapsulated without any further work - or is longer, in which case a PTB is sent back to the sending host, with an MTU value such that the host will generate packets to this destination host of a length which, when encapsulated, will equal this reliably determined MTU. This encapsulation and some kind of PMTUD management is required for any CES architecture which uses encapsulation. All other CES architectures use encapsulation exclusively. There is at least one other approach to PMTUD management which is probably more expensive to perform as securely as this one. The fact that this and other processes are explained in some detail in this Ivip ID and not in the IDs of other proposals does not mean that the other proposals, once developed to the point of proper operation, would be simpler than Ivip. The encapsulation itself is straightforward. The sending host's Whittle Expires July 17, 2010 [Page 42] Internet-Draft Ivip Architecture January 2010 address is used for the outer source address and the ETR's is used for the outer destination address. For IPv4 packets, the Diffserv, TTL and other flags are copied to the outer header. For IPv6, Traffic Class and Hop Limit bits are also copied. 5.1.5. Mapping lookup and caching Apart from PMTUD management, looking up the mapping for an incoming packet is the most complex task that ITRs need to perform. This task is the same for encapsulation in both IPv4 and IPv4 and for the IPv4 approach to MHF: ETR Address Formatting (EAF). For the IPv6 MHF technique - Prefix Label Forwarding (PLF) - the mapping lookup is similar, but only part of the ETR's address is actually needed for writing 19 or 20 bits into the header. When encapsulation is used, for IPv4 or IPv6, the result of the mapping lookup is an IP address of the ETR, which will become the destination address of the outer header. The result of EAF is similar, and ETR address where the two least significant bits are zero. This will be written into the modified IPv4 header. The result of the PLF mapping will be a 19 or 20 bit value is written into the modified IPv6 header and which identifies one of 2^19 or one of 2^20 contiguous DFZ advertised prefixes, each of which is advertised by a different ISP site. These 20 bits do not uniquely identify an ETR if there are more than one at each ISP site, but they are sufficient for the packet to be forwarded across the DFZ to the nearest BR of that site, where a second mapping lookup may be performed on the destination address to determine which of multiple ETRs at that site the packet should be forwarded to. Please refer to the description and example of PLF operation at the end of Ivip- summary.pdf. [Ivip Summary and Analysis] This following may appear somewhat complex, but it is a description of different approaches to handling ITR to ETR tunneling for both the IPv4 and IPv6 Internets. Ideally, encapsulation won't be necessary to at all. At worst, it will be necessary until DFZ and other routers are upgraded to handle EAF or PLF modified header packets. The mapping lookup is driven entirely by the packet's destination address. Ivip does not attempt to send packets of differing types, service class or differing source address to different ETRs. (Nor do the other core-edge separation schemes.) After a packet arrives, and has been classified as being addressed to an SPI address (meaning it matches one of the MAB prefixes) the next step is to find out whether the ITR has any mapping cached for the packet's destination address. For IPv4 the full destination address Whittle Expires July 17, 2010 [Page 43] Internet-Draft Ivip Architecture January 2010 is used. For IPv6, only the most significant 64 bits are used, since SPI space is divided on /64 boundaries. Busy ITRs may have tens or perhaps hundreds of thousands of mappings already cached. An ITR function in a sending host may have only a handful or a few hundred. A carefully designed algorithm will be needed to find any existing mapping, or to determine that the destination address does not match any cached mapping. In the former case, the mapping consists of a starting address and ending address for the micronet which the destination address falls within - and a single ETR address. This ETR address (or set of PLF bits) is then applied to the packet - by writing it to the outer header when encapsulating, or by writing into the modified header for EAF or PLF. (PLF only uses 19 bits of the ETR address - just enough to distinguish between the 2^19 contiguous prefixes which are reserved for this system. The resulting packet is then ready to be forwarded like any other - according to its outer header, or according to the bits just written into its modified header. If no cached mapping is found, the ITR buffers the packet and sends a map query to a local query server. This includes a nonce which is used to secure the reply, and any later map update messages the query server sends if the mapping changes during the time the ITR caches it. The local query server sets the caching time on the mapping. This time is not transmitted as part of the fast-push mapping system - it is locally configured and could be set differently for different replies by various algorithms in the query server to optimise its interactions with the ITRs, and to limit the number of mappings the ITR caches. (Further work: It may be desirable for each ITR to be able to communicate to its query server(s) the state of its cache and how close to any limits it is running, so the map replies can have their caching times adjusted downwards.) The ITR flushes from its cache any mappings whose cache times have expired. The cache includes the starting and ending address of the micronet, the ETR address and the nonce which was sent in the query which returned this mapping. At any time when a mapping is cached, the ITR may receive a Map Update message from its local Query Server. This will be secured by the nonce of the original query. The most common update will be that this micronet is now mapped to a different ETR address. Another type of update is that this micronet include it being mapped to no ETR (an ETR address of zero) - in which case the ITR will drop subsequent matching packets. If the end-user network splits a micronet into two Whittle Expires July 17, 2010 [Page 44] Internet-Draft Ivip Architecture January 2010 or more smaller micronets, the original micronet is removed from the cache and the new micronets are written. Alternatively, a micronet might be joined with adjacent micronets to make a new, larger micronet, with the same or a different ETR address. In this case, there will be an update for every already cached micronet affected by the change - and the previously cached micronet will remain, with the new ETR address. None of these updates reset the caching time. So the mappings, however modified, will time out as usual. If the mapping update message from the query server reset the caching timeout process, then continued updates would keep a mapping in the ITR's cache for excessive periods - including if the ITR was not handling any packets for this micronet. In this way, ITRs receive all the updated mapping they need, within a fraction of a second of the changed mapping being received by the local query server. This may be directly from the QSD, or from the QSD via one or more QSCs. 5.1.6. ITFH - ITR Function in Host An ITR function in a sending host performs either encapsulation and PMTUD management or MHF as described above. This function is only for packets generated in the host. ITFH should only be used on hosts which have fast, reliable, connections to two or more local query servers. If there are delays, or packet losses, then the extra management traffic between the ITR function and the local query servers may not function well enough to ensure there are no significant delays to traffic packets. In many settings, the software and hardware required to implement an ITR in the sending host will have zero incremental cost. RAM and CPU capacity is now extremely inexpensive. Hosts, such as desktop PCs and servers used in hosting farms and cloud systems come bristling with multicore CPUs and gigabytes of RAM for the price of a good shirt. The host could be on a conventional global unicast address (PI or PA) or on an SPI address. If it is thought desirable to enable ITFHs in hosts behind NAT, then at least two additional measures would need to be taken. Firstly, if encapsulation was used, the PMTUD exchange with ETRs would need to work through the NAT - which it probably would. Secondly, the ITFH would need to set up and maintain a two- way tunnel to two or more local query servers. TCP with a keepalive would be sufficient. Then, instead of UDP mapping queries, replies and updates, the same messages would be sent over TCP. It is not out Whittle Expires July 17, 2010 [Page 45] Internet-Draft Ivip Architecture January 2010 of the question to link all ITRs, QSCs and QSDs with TCP, rather than use UDP packets, since the TCP will ensure reliable delivery of messages, and so reduce the complexity of the code for sending and receiving messages. 5.1.7. ITRs auto-discovering local query servers There is further work to do to enable ITRs to automatically discover the addresses of one or more local query servers. This is not absolutely necessary, but would greatly ease the deployment of ITRs in ISP and end-user networks. The more ITRs there are, the less work each one has to do and so the greater the chance that they can be implemented with little cost in a COTS server, rather than an expensive hardware-based router. This principle applies especially to ITFHs. 5.1.8. ITRs regulating their advertisements Any ITR - including DITRs and ITFHs - should only process SPI packets and advertise MAB prefixes if it is able to obtain mapping, directly or indirectly, from a full database query server which is fully up- to-date. An ITR which loses touch with its query servers, or which is informed in some way TBD that the query servers can no longer provide up-to- date mapping, should stop processing SPI packets immediately, and simply forward these packets like any other router. The packets will be handled by some ITR upstream, in the local network, the ISP network or by a DITR. To continue to process packets based on cached and potentially out of date mapping risks sending packets to the wrong ETRs. It is better to let the packets be handled by other ITRs which are working properly. There is a potential difficulty if a local full database query server is online, but has recognised that its mapping for one or more MABs has corrupt or stale data. One option would be for this server to pass on queries for these MABs to other full database query servers - yet the nearest one with up-to-date mapping may be some distance away in another ISP, and there would need to be commercial arrangements for this. Another option would be for ITRs to be able to recognise that they can't get up-to-date mapping for one or more MABs, and to stop advertising these MABs and processing packets addressed to them. Either approach is probably preferable to a full database query server going offline just because one part of its database is corrupt, due to some transient failure in the fast-push mapping distribution. If the problem can be corrected promptly, within a few tens of seconds, then it is probably OK to let the dependent ITRs function normally, with the outdated mapping information in the QSD, Whittle Expires July 17, 2010 [Page 46] Internet-Draft Ivip Architecture January 2010 provided they all get any updates which were delayed by the outage. However, if the mapping information is corrupt, it should not be used to reply to mapping queries. To do so would risk traffic packets being tunneled to ETRs or to any address at all - which is unacceptable from a security point of view. DITRs need to be closely connected to robust QSDs and should have a backup connection to some other, potentially more distant, QSDs. It would be highly undesirable for the DITR to be turning its advertisements of MABs on and off due to some problem with its query server(s), since this would unreasonably burden other DFZ routers. 5.2. ETRs 5.2.1. In servers or dedicated routers The ETR function can be performed in a dedicated router or in a server with appropriate software. Whether the ETR function is performed in a server with one or more Ethernet ports, or a router with multiple ports of various kinds, depends on how the traffic packets are to be forwarded to the one or more end-user networks being served by this ETR. The methods of forwarding do not need to be part of the Ivip RFCs - just how ETRs handle the incoming packets, and for encapsulation, how they communicate with the ITR for PMTUD management purposes. In the TTR mobility system, the TTRs perform ETR functions. The link to each end-user network is a separate two-way tunnel, established by the Mobile Node (MN) to the TTR. 5.2.2. ETRs in ISP networks An ETR in an ISP network can, in principle, handle packets for many end-user networks - all from a single global unicast address. This has a scaling benefit for IPv4 by supporting a potentially large number of end-user networks, with potentially large numbers of SPI addresses, while requiring only a one of the ISP's IP addresses. (For IPv6, inefficiency of address use is not a concern.) 5.2.3. ETRs at the end-user network site A multihomed end-user network with two links to ISPs might have two ETRs - one for each link. Each ETR will have a stable conventional (non-SPI) global unicast address to receive encapsulated packets on. So each ISP needs to devote at least one of its addresses, or more likely four, for each such ETR. This saves the ISP from having to run an ETR for this customer - all the ISP provides is connectivity Whittle Expires July 17, 2010 [Page 47] Internet-Draft Ivip Architecture January 2010 and this small amount of stable address space. There could be one physical ETR, with two links to the two ISPs, receiving encapsulated packets as above on the two addresses provided by the two links. This device would be a router of some kind, even if implemented on a server, since it would also be deciding which link to send outgoing packets on. 5.2.4. MHF ETR functionality - EAF and PLF If Ivip is introduced with encapsulation, its ITR and ETR functions will contain Modified Header Forwarding functionality ready for a future migration from encapsulation to MHF exclusively. The IPv4 MHF technique - ETR Address Forwarding (EAF) - is very similar to the encapsulation arrangement, so the same ETR could do both, from the same address. However, with EAF, the ETR address is specified with the most significant 30 bits, giving a granularity of 4 IP addresses. To avoid having to change ETR addresses when encapsulation is turned off, only one ETR should be located in each /30 prefix. The IPv6 approach to MHF - Prefix Label Forwarding (PLF) - is conceptually different from the encapsulation approach in which the packet is tunneled to an ETR at a single IPv6 address. The ITR uses the mapping to write 19 or 20 bits into the IPv6 header. Upgraded routers in the DFZ forward the packet to ISP BRs advertising one of 2^19 or 2^20 separate prefixes. While the mapping still specifies an exact 128 bit IP address for the ETR, before MHF can be turned on, all ETRs must be given addresses within the special set of DFZ- advertised prefixes which the MHF system can forward these packets to. On arrival at the BR, the packet itself contains no information of further use - it does not contain the ETR address, just 19 or 20 bits of the address bits which differentiate this contiguous set of prefixes. If there is only one ETR for each such prefix, then the BRs (or perhaps single BR) needs only to forward the packet to the ETR. Alternatively, the ETR function could be performed within the one or more BRs. However, if this prefix has multiple ETRs, then the BR needs to behave like an ITR and perform a second mapping lookup, using the destination address of the packet, to decide how to forward (or perhaps tunnel) the packet to the correct ETR. There are various techniques for doing this, including the ISP using the PLF bits again, interpreted according to its own arrangements by its internal routers, to forward the packet to some internal prefix (perhaps in ULA space) which leads to the correct ETR. I have not yet explored the various ways an ISP could use to get PLF-tunneled packets to the Whittle Expires July 17, 2010 [Page 48] Internet-Draft Ivip Architecture January 2010 correct ETR, or how techniques and ETR placement arrangements for encapsulation can be made compatible with the PLF arrangements. Please refer to the description and example of PLF operation at the end of Ivip-summary.pdf. [Ivip Summary and Analysis] With both EAF for IPv4 and PLF for IPv6, the work an ETR performs on each tunnelled packet is trivially simple: restore the altered bits so the IP header has its standard form again, and forward the packet to the destination network. The ETR does not communicate with the ITR or with any other part of the Ivip system, since the ITRs and ETRs have no Ivip-specific PMTUD problems to solve. If there resulting packet is too long for the next hop, the existing IP stack of the server or router in which the ETR function is performed will implement conventional RFC 1191 PMTUD and generate a PTB to the sending host. 5.2.5. ETR functionality for encapsulation With encapsulation, ETRs receive IP-in-IP packets on a stable global unicast address. The ETR recognises all such packets and decapsulate them. If the outer header source address matches that of the inner packet, then the ETR forwards the packet to the end-user network. If the ETR handles multiple end-user networks, then it will have appropriate configuration or router functionality to forward the packet to the correct end-user network. For PMTUD management, some more complex functionality is required. When the ITR uses special techniques to send a traffic packet, in two parts, as a probe of PMTU to this ETR, it sends a long packet and one short one (or multiple copies of the short one) to the ETR's address. However, these are not IP-in-IP encapsulated. They are both UDP packets - the long one with the ITR's address as the source address, and the shorter one(s) with the sending host's address as the source address. If only the short packet arrives, then the long one was lost - probably due to it being longer than the PMTU from the ITR to this ETR. The ETR informs the ITR of this non-reception, and receives an acknowledgement of this. If both the long and short packets arrive, the ETR reconstructs the full traffic packet, forwards it to the end- user network, and informs the ITR that it has been received correctly. This involve significant complexity in the ETR, but does not involve storing state for more than a few seconds. Once the traffic packet has been decapsulated, if the forwarding step leads to the packet being deemed too long for the next-hop MTU, then the conventional IP stack will generate a PTB to the sending host and Whittle Expires July 17, 2010 [Page 49] Internet-Draft Ivip Architecture January 2010 RFC 1191 PMTUD will proceed just as it would if there had been no ITR to ETR encapsulation. 5.3. QSDs - full database query servers 5.3.1. Placement in ISP and end-user networks ISP networks and end-user networks do not absolutely need ITRs - since packets sent by hosts in these networks could always be forwarded to the DFZ and handled there by DITRs. However, as a service to customers, to ensure the packets their hosts send to SPI addresses take the shortest path to the correct ETR, and to avoid depending on DITRs, ISPs will generally want to install ITRs in their network. This means they need to have at least one full database query server (QSD), for these ITRs to obtain mapping from. To rely on QSDs outside their network is probably not reliable or robust enough, since if an ITR can't get mapping, it won't be able to handle new flows of user traffic to SPI addresses. Full database query servers could be implemented in a conventional router, but it makes most sense to implement them in COTS servers. (Currently we think of conventional routers being a very expensive place to run complex software. However, with cheap multicore 64 bit CPUs and gigabytes of RAM, future router models might have plug-in PCBs to add blade servers or the like to run such complex software in the same chassis.) Query servers do not handle traffic packets. They must be located on stable global unicast addresses. Generally speaking, if the QSD is in an ISP network or in an end-user network which does not use SPI space, but which has ITRs, it will be on conventional (not SPI) space. QSDs in end-user networks which use SPI space will be located on SPI addresses. This means that the Replicators which send the mapping feeds to these QSDs will produce the feed packets addressed to the QSD's SPI address. Therefore, the feed packets will either need to be tunneled by a nearby ITR, or the Replicator itself could have an ITR function built in (ITFH). If an end-user network (using PA, PI or SPI space) runs ITRs (including ITFHs), then it could have these ITRs send their queries to the QSDs of its one or more ISPs. This would be the best arrangement for smaller networks, since running one, or ideally two QSDs in its own network involves having two or four feeds of mapping data from Replicators outside its network. ISPs are not necessarily going to charge for this mapping feed traffic if the Replicators are within their network, since they are already accepting mapping feeds from outside their networks for their Whittle Expires July 17, 2010 [Page 50] Internet-Draft Ivip Architecture January 2010 own QSDs, and it does not add to their upstream traffic load to replicate the mapping feeds so several can be sent to each substantial end-user network. However, once the Ivip system is handling large numbers of end-user networks, such as with a billion micronets, running QSDs is a non-trivial task. For instance, the initialisation operation described below could involve large quantities of bandwidth. Every ISP which has ITRs, including ITFHs, needs at least one full database query server (QSD). For robustness, each should have two or perhaps three or more. Each QSD should have at least two feeds of mapping information from two upstream Replicators, ideally via different upstream links. Ideally, each QSD in the ISP network should have its mapping feeds from different Replicators. A Replicator could be implemented in the same server as a QSD - and this would be fine in the earlier years of deployment. Later, when the QSD has to handle more mapping updates and requests, it would probably make sense to put the Replicator function on another server. 5.3.2. QSD initialization and reception of mapping updates Booting up a QSD involves it downloading snapshots of the mapping database from RUAS servers, whilst storing all mapping updates which were generated after the snapshot. Then, while still storing incoming mapping updates, the stored updates are applied in chronological order to bring the QSDs copy of the mapping database fully up to date and ready to receive updates as they arrive. This initialisation process results in the QSD having a complete list of all the MABs in the Ivip system. (Further work: plan how a running QSD finds out about additions and deletions from this list of MABs, including the commencement and cessation of mapping updates for these MABs.) Normally, each QSD will receive two (or perhaps more) feeds of mapping updates from two not-too-distant Replicators in topologically different locations. Each feed, in a given second, consists of a number of UDP packets, each with updates for one or more MABs. In the current design of the fast-push mapping distribution system, the feed consists of UDP packets secured with Datagram Transport Layer Security RFC 4347. If a packet is missing from the stream from one Replicator, a packet with the same updates will most likely be received from the other Replicator. (Each Replicator uses the same system of two redundant feeds to generate the mapping feed it sends out to other Replicators and to QSDs.) If one or more packets are lost from both streams, then the QSD will retrieve the missing updates from a server operated by the RUAS which Whittle Expires July 17, 2010 [Page 51] Internet-Draft Ivip Architecture January 2010 generated these updates. In this way, even with temporary loss of mapping feed packets, the QSD is able to keep its local copy of the mapping database up-to date with all the mapping updates. The time delay from an end-user network making a mapping change (or the change being made by some company authorised to do so by the end-user network) to the update being integrated into QSDs all over the Net, will be in the order of 4 seconds. The QSD also needs diagnostic and management functions, such as to be able to report problems with missing packets. It may also be configured to choose feeds from alternative Replicators if one or more of its current Replicators is not working properly. There is considerable complexity in these functions, but the software, once written and tested, can run on COTS servers. Mapping data is most likely to be stored in a format which is also suitable for responding to mapping queries. Otherwise the mapping data itself needs to be updated, and then some other data structure suitable for query resolution needs to be replicated. The QSD will store the full mapping database in RAM. For IPv4, the mapping data itself is very simple - just 12 bytes in its most compact form: 32 bits for the micronet's starting address, 32 bits for its length or end (though micronets would rarely exceed 2^10 IPv4 addresses). If the IPv4 Internet ever supports 2 billion micronets, then this is 24 gigabytes, which can be done with DDR3 server RAM today. The storage format may not be this compact, but by the time such large numbers of end-user networks adopt SPI space, COTS servers will have more RAM. The difficulties of writing the code for, and running both QSDs and Replicators are justified by the avoidance of initial packet delays and the complexity of global query server systems such as LISP-ALT. IPv6 mapping is more voluminous - 64 bits for the micronet's starting address, 64 bits for its length or end and 128 bits for the ETR address. This is 32 bytes. The worst case storage requirement, not counting any compaction or expansion due to how it is stored in an easily traversable form for answering queries, and making updates, is 10 billion micronets - 320 gigabytes. This fits on a consumer hard drive today, and will surely fit in COTS server RAM by the time such massive adoption of mobile IPv6 devices occurs. In addition to applying the updates to the local copy of the database, any updates which match items in the "querier cache" are used to send mapping updates (using a protocol described below) to the matching querier. Whittle Expires July 17, 2010 [Page 52] Internet-Draft Ivip Architecture January 2010 5.3.3. Responding to queries The simplest model of ITRs and QSDs is that all ITRs send their queries to a single QSD. In practice each ITR may have two or more QSDs and may send a query to one if it does not get a response within a few tens of milliseconds from the first. ITRs send mapping queries when they receive a packet addressed to an SPI address which they do not have any cached mapping for. While waiting for the map reply, the ITR buffers the one or more packets which need this mapping. As soon as the mapping is received, the ITR tunnels these packets to the ETR specified in the mapping. In practice, as is discussed below, it will often be desirable to place one or more levels of caching query server (QSC) between the ITRs and the QSD. So a QSD may receive mapping queries from both ITRs or QSCs. Queries from ITRs use exactly the same protocol as those from QSDs. The mapping query is a UDP packet sent to a well-known port on the QSD. It contains a nonce which is specific to this query. (Any repeat of the same query will use a different nonce.) The packet also contains the destination address of the packet which the ITR needs mapping for. The ITR (or QSC) only knows that this address is within one of the MABs, and that it does not match any micronet in its cache. The QSD looks up this address in its database and returns the ETR address. A zero address is an instruction to the ITR to drop the packet. The map reply packet contains the nonce from the query, and four items of information: the start address of the micronet, its end or length, the ETR address and a TTL caching time, in seconds. If the querier is an ITR, it adds this mapping to its cache and is then able to handle any packets addressed to this micronet. The QSD sets the caching time, according to local policy and perhaps according to some functions which in some way help optimise the operation of this QSD, any QSCs and their dependent ITRs. The QSD makes a record of the response in its querier cache. It does not need to store the address which was queried. It stores the address of the querier, the nonce in the query, the time the response was sent, the caching time (or the time at which the caching time will expire) as well as the micronet starting point and length and the ETR address. A background process deletes records whose caching time expires. Whittle Expires July 17, 2010 [Page 53] Internet-Draft Ivip Architecture January 2010 5.3.4. Sending mapping updates to ITRs and QSCs The following process is potentially expensive, due to the potentially large number of records in the querier cache. This may set some upper limits on the capacity of a single QSD to support ITRs directly - in terms of many ITRs, and or the traffic patterns of these ITRs, leading to a large and slow to index querier cache in the QSD. The use of QSCs between the ITRs and the QSD should significantly reduce the number of entries in the querier cache, and so enable a QSD to serve more ITRs. The querier cache needs to be indexable on the addresses of its micronets. Micronets do not overlap, so the cache would have an index of the entries in ascending order by address - which will enable reasonably rapid searches. Once a mapping update from the upstream Replicators has been applied to the QSDs mapping database, a check is performed to test whether it covers any address space in the querier cache. If so, then the QSD will generate a mapping update to that querier. The simplest example is when a micronet has its ETR address changed. The resulting mapping update includes the nonce from the original query, and a nonce or sequence number to distinguish this mapping update packet from any others. The mapping update is sent as a UDP packet to the querier. (TCP sessions between ITRs, QSCs and QSDs might be a good idea rather than UDP packets.) The QSD expects an acknowledgement - also a UDP packet with the nonce and sequence number. It will resend the mapping update packet until it gets an acknowledgement - or too many retries generates an error event. This may signal that the querier is no longer alive. So within a few tens of milliseconds of the QSD receiving a mapping update from the fast- push mapping system, it relays a suitable one or more mapping updates to any queriers which are caching a mapping which is affected by the just-received update. When a micronet changes its ETR address, the mapping update is simple - it contains the micronet's start and end (or length) and the new ETR address. If the querier is an ITR, it updates its cache and so tunnels any matching packets to the new ETR address. Mapping changes could be more complex - since existing micronets could be divided into smaller ones or merged with adjacent ones. Some moderately complex logic will deal with the various possibilities subject to the constraints that no matter what mapping updates are received, the querier will not be given mapping for any addresses beyond the range of the micronet in the initial map reply message. This may involve multiple mapping updates which require the Whittle Expires July 17, 2010 [Page 54] Internet-Draft Ivip Architecture January 2010 querier to remove the existing micronet from its cache, and replace it with two or more smaller micronets. Any altered cache items in the querier, including new ones when the initial micronet has been divided, will have exactly the same caching timeout as the original. So no matter what mapping changes arrive, the resulting cache items will expire in the querier at the same time as the original cached mapping would expire. If mapping updates extended the cache time, then there may be no limit to how long the querier needs to cache some micronets, even though it may not need this mapping any more. From this it can easily be imagined that ITRs will be able to quickly and reliably receive the mapping they need from a local QSD and that the QSD will update the mapping cache of all ITRs which are caching mappings for which it receives updates. 5.4. QSCs - caching query servers A caching query server (QSC) is a relatively simple function, typically implemented as software in a server. The software for ITRs, QSCs and QSDs would share some common components. A QSC receives and responds to queries from ITRs or other QSCs in the same manner as a QSD. The QSC sends queries to a QSC or QSD in just the same way as was described above for an ITR - and likewise receives map reply messages and map update messages from the upstream QSC and QSD as just described. There could be zero, one, two or in principle any number of QSCs between an ITR and the one or more QSDs. All these devices are typically in the same ISP network - or in an end-user network with close connections to the QSD(s) in the ISP network. So communication between them is very fast, reliable and inexpensive. Typically, there will be little or no packet loss, but the protocols will need to cope with any losses in a robust manner. If a querier sends out a query and does not get a reply within some quite short time, such as 100ms, then it should try sending the query (with a different nonce) to an alternative upstream query server. Further work: ITRs auto discovering query servers in general - and QSCs autodiscovering other QSCs and QSDs. Manual configuration of the tree-like structures of these devices should also be possible. If the mapping needs of one ITR were completely uncorrelated with the mapping needs of other ITRs served by the same QSD, then there would be little or no benefit in deploying intermediate QSCs. However, there is likely to be sufficient commonality between the mapping needs of tens or hundreds of ITRs and ITFHs to make QSCs a good Whittle Expires July 17, 2010 [Page 55] Internet-Draft Ivip Architecture January 2010 investment in expanding the capacity of a single QSD to support more ITRs. If 20 ITRs send their queries to QSC1 and another 20 to QSC2, then the queries, replies and map update exchanges which must be performed by the one QSD which both QSC1 and QSC2 query will be significantly reduced. This is because it will frequently be the case that QSC1 will already be caching the mapping which is needed to answer a query from one of its 20 ITRs. Without the QSCs, every ITR query would need to be handled by the QSD - and its querier cache would be correspondingly larger. Furthermore, if more than one of QSC1's ITRs is caching mapping for a micronet for which the QSD gets a mapping update, then the QSD only needs to send a single mapping update to QSC1, rather than sending one to each such ITR. There is further work to do planning these protocols. The caching times do not affect Ivip's ability to get the mapping updates to all ITRs in real-time. Longer caching times will reduce the need for the querier, such as an ITR, to make another map request if it is still sending packets to the micronet. Longer caching times also increase the number of mapping updates which need to be sent - and perhaps the time is so long that the querier no longer needs the mapping. Shorter caching times reduce the number of cached items, but increase the load of mapping queries and responses. While the exact details are TBD, it is clear that it will be possible to define relatively straightforward protocols by which ITRs, QSDs and optionally QSCs can be combined to efficiently support the mapping needs of many ITRs per QSD. Once the code for the ITR and QSD is written, writing the QSC code will be relatively easy. Since QSCs don't need a mapping feed from Replicators, they can be numerous and located close to ITRs - especially if there are a large number of sending hosts with ITFH ITR functions built-in. A QSC close to a bunch of ITRs is also attractive if this is not directly close to the QSD - for instance in an end-user network which uses one or more QSDs in the ISP network. 5.5. FMS - Fast-Push Mapping Distribution System Ivip's most important architectural feature is that end-user networks are able to control all ITRs in real time - the address of the ETR to which packets which are addressed to their micronets will be tunneled. Previous sections describe how ITRs request and cache mapping from QSDs, and have their caches updated by the QSD if there is a change to this mapping within the caching time. While there is considerable complexity in the code required for ITRs, QSDs and QSCs, the protocols used between them are not particularly complex, and the communication takes place over physically and topologically short Whittle Expires July 17, 2010 [Page 56] Internet-Draft Ivip Architecture January 2010 distances, within ISP and end-user networks. So the delays and potential packet losses of the global Internet are not a concern for these elements of Ivip. The most adventurous aspect of Ivip is the Fast-Push Mapping Distribution System (FMS), which enables mapping change commands originating from end-user networks or some other organisation acting on their behalf to be transmitted securely and robustly to all QSDs in the Net. There may be hundreds of thousands of these and the system should scale well to QSD numbers in the 10^7 range. The following material broadly describes the requirements of this system and one possible way of implementing it. Perhaps there is a simple flooding protocol with a large mesh of servers which would work well. The following cross-linked multicast- like set of trees can be viewed as a flooding system, with each Replicator flooding the packet to 20 other Replicators as soon as it gets the first packet from one of its upstream Replicators. More detailed material is in [I-D.whittle-ivip-db-fast-push]. Figure 1 depicts the structure of part of the FMS. Whittle Expires July 17, 2010 [Page 57] Internet-Draft Ivip Architecture January 2010 \ | / } Update information from end-users - directly \ V / } or indirectly - to one of a dozen or so RUASes. \|/ RUAS-X ->--------------[snapshot & missing packet HTTP server 1] /|\ \--[snapshot & missing packet HTTP server 2] / | \ / V \ Streams of packets containing identical real-time | mapping updates to the 8 Launch servers. | \ \ | / / Each of the 8 Launch servers gets a \ \ V / / stream from each RUAS. \ \ | / / <>[Launch server N]<> The 8 Launch servers have links with each / / | \ \ other. Each second, each one sends a set / / V \ \ of updated packets to 20 level 1 / / | \ \ Replicators. Each level 1 Replicator | receives two streams, each from a | different Launch server. \ \ / Even with packet losses and link failures, \ / most of the 80 level 1 Replicators receive level 1 \ / a complete set of update packets, each [Replicator] second, which they each replicate to 20 / / | \ \ level 2 Replicators. / / V \ \ / | | | \ In this example, each Replicator consumes | two feeds from the upstream level, and / generates 20 feeds to Replicators in / the level below (numbered one above the \ / current level). So each level involves \ / 10 times the number of Replicators. level 2 \ / [Replicator] These figures might be typical of later / / | \ \ years with billions micronets and 100ks / / V \ \ of QSDs to drive. In the first five or / | | | \ ten years, with fewer updates, the / | | | \ amplification ratio of each level could / | | | \ be much higher, with fewer levels. / | | | \ | | | Replicators are well-connected COTS | | servers at peering points and ISP data | | centers, though the 8000 Level 3 and [Levels 3 and 4] 80,000 Level 4 Replicators may be in [Replicators ] ISPs and larger end-user networks. \ | \ / \ | \ / Up to 800k QSDs get two or more ideally \ | \ / identical full feeds of updates. QSD QSD Whittle Expires July 17, 2010 [Page 58] Internet-Draft Ivip Architecture January 2010 Figure 1: Multiple levels of Replicators drive hundreds of thousands of QSDs. The FMS needs to robust, secure and fast. Ideally it should take no more than four seconds for mapping changes to reach all QSDs. Once there, all ITRs which need the mapping change will be sent it within a small fraction of a second. The FMS needs to be decentralised so there is no single point of technical, administrative or commercial failure. The main part of it will be run by a number of organizations working together. These organisations, of which there might be five to perhaps twenty, are called Root Update Authorisation Server (RUAS) organizations. The final parts of the FMS - the last one or more levels of Replicators fanning out the mapping update packets to QSDs - may be run by local ISPs rather than these organizations. Each RUAS organisation operates its own RUAS system, which would be a multi-server system with no single point of failure within itself. Each RUAS is responsible for the mapping updates of particular multiple MABs. Within the FMS's Replicator system, which fans out the mapping updates to QSDs, each packet contains multiple mapping updates, and some other regularly inserted information regarding snapshots of each MAB's mapping information. Each RUAS accepts mapping updates, after appropriate authentication, from end-user networks or whichever organisation they contract to control the mapping of their micronets. The RUAS may accept these updates directly or indirectly via other organisations. Typically, mapping updates would be sent via some automated process, such as from a MM (Multihoming Monitoring) company - but manual control such as via a web-based form would also be possible. The most likely scenario for multihomed end-user networks is that their appointed MM company would send mapping changes to the MAB company from whom the end-user network is renting their SPI space. The MAB company's servers would convey the mapping change within a fraction of a second to the RUAS this company has chosen to handle the mapping updates for their MABs. In the current design, RUASes are loosely interlinked and have some form of dynamically changed quota so that in each second, each RUAS has an upper limit on the number of mapping updates it can send. This is to ensure that that the number sent will not overload the Replicators. If an RUAS has a peak of updates to send, more than this limit, then some will need to be delayed for a few seconds. The RUAS organizations collectively run a set of Launch servers - for Whittle Expires July 17, 2010 [Page 59] Internet-Draft Ivip Architecture January 2010 instance 8 in the above diagram. Launch servers are strategically located at major data centers around the world, such as one in China, one in Japan, one in North America etc. There is a pipelined process which ensures that each second, the Launch servers all send out an identical set of updates. This is to ensure that all QSDs get the same set. In the first stage of the pipeline, for one second, the RUAS would collect incoming mapping updates and sort them according to which MAB they concerned. In the second stage, the RUASes would transmit their updates to every Launch server. In the third stage, also lasting a second, the Launch servers compare notes and form a quorum which agrees on exactly which sets of updates will be sent to the FMS. This will require some careful programming, but there may be existing protocols which can help with this. The outcome of the third stage of the pipeline is that all Launch servers will have the same set of updates ready to launch into the FMS in the next second - and all RUASes will know this set while preparing which updates to send in the next second. In the final stage, each of the (for example) 8 Launch servers sends out the updates, as a series of UDP packets secured by DTLS (RFC 4347) to multiple Level 1 Replicators. Each Replicator is a COTS server with suitable software. The sessions with these Launch servers are reasonably stable, but the whole structure of the FMS cross-linked trees of Replicators must be adaptable to gradual change, while in continuous operation. Each Level 1 Replicator will receive at least two sets of update packets each second, from different Launch servers. Ideally these will arrive from different links and will be from topologically diverse Launch servers. Each packet carries fields which specify which second it was sent in, and which number the packet is in the complete set of packets sent that second. While the packets of each feed are different, in terms of their DTLS protection and source addresses, the mapping change payload of similarly numbered packets from different Launch servers will be identical. So, for instance, as long as the Replicator receives a packet 23 for the current second from one of its upstream Launch servers, then it will be able to replicate this packet's information to multiple Replicators in the next level. If a Replicator does not receive any packets of a given number, it makes no attempt to obtain the information from another source. As soon as a Level 1 Replicator receives the first copy of a given numbered packet for this second, assuming it is received completely and that DTLS ensures it is really the packet sent by the Launch server, this Replicator transmits copies of its payload to all its Whittle Expires July 17, 2010 [Page 60] Internet-Draft Ivip Architecture January 2010 downstream Replicators in the next level. Each such packet is handled by DTLS for the session previously established with each Downstream Replicator. In this example, there are 4 levels of Replicators, each receiving two streams and generating 20 streams to Replicators on the next level. So the amplification factor is 10. In this example, 80,000 Level 4 Replicators send update streams to 800,000 QSDs. It may not be strictly necessary for a Level N Replicator to receive packets from Level N-1 Replicators, but this is the easiest way explain the structure. There is no danger of loops since a Replicator will ignore a packet with second and sequence number it has already sent. This resembles a multicast arrangement - and multicast is very subject to packet loss. However, each Replicator has a very good chance of receiving all the information, despite occasional packet losses and the failure or unteachability of any one upstream Replicator. The transmission of data should be nearly instantaneous at each Replicator, so the mapping data should be fanned out all over the world, through four or more layers, in less than a second. At the branch-tips of this cross-linked tree-like structure of Replicators, the packets are received by QSDs. Each QSD should have two or more feeds of updates, ideally coming over different physical links from topologically diverse Replicators. Like Replicators, QSDs receive the update packets securely via DTLS, and discard all but the first of each sequence number each second. The one server could run both QSD and Replicator functions. In principle one server could do this and be an ITR and ETR as well. In the initial stages of deployment, with update rates relatively low compared to the full-scale 10^10 mobile end-user network scenario, it will probably be attractive to run multiple Ivip functions on the one server. The RUAS companies will probably run the first two or so levels of Replicators - which would the mapping updates to major exchange points all over the Net. For levels 3 and beyond, ISPs could be relied upon to run their own Replicators, and to offer feeds to each other, as part of ensuring the mapping feed for all ISPs in their area will be robust. There is significant complexity in the RUAS systems, but their interface to the Launch servers is the only thing which needs to be standardised. The Launch servers need to run a complex algorithm so they all decide, and all know the final decision, on which updates all Launch servers, or perhaps a majority of them, will send in the next second. This is to ensure that all level 1 Replicators are sent the same set of packets. This is primarily to avoid a problem where Whittle Expires July 17, 2010 [Page 61] Internet-Draft Ivip Architecture January 2010 one Launch server only sends its updates to few, or none, of the level 1 Replicators. Such a condition, combined with lost packets and dead links could result in the cross-linking of the system being insufficient to ensure all QSDs get the same set of updates. An RUAS which was for some reason unsuccessful in having some or all of its updates accepted for launch in the next second would add the updates they contain to the list of updates to send in the second which follows. This failure may have been due to an insufficient number of active Launch servers receiving this RUAS's full set of update packets. To form a quorum which agrees on what updates to send, it is not necessary that every Launch server has every update. As long as the majority do, then due to the crosslinked nature of the Replicator system (and assuming some maximum number of dead links and dropped packets), it can be assured that every QSD will get the same set of mapping updates. As noted above, QSDs will sometimes fail to receive the same numbered packet from both of its upstream Replicators. Each RUAS will provide HTTP servers by which any QSD, or indeed any other host, could download specific packets from the last minute or so. Each packet would be named according to its place in that second's sequence and according to the second it was sent in. There would be some arrangement by which QSDs can reliably ascertain from each packet how many packets there are in each second - and failing that, this information could be obtained from the RUASes HTTP servers. At regular intervals, such as every few minutes, the RUAS will include a special type of "snapshot" message in the update stream. This will concern one of its MABs, and provide some kind of hash result or other compact number by which a QSD can check the validity of this MAB's section of the mapping database. This means that QSDs will need to maintain their internal data in a standardised way which is amenable to computing a local hash function to compare with that received in the snapshot message. Each snapshot message will coincide with a compact dump of this MAB's mapping database, at a particular point in time. When a QSD is initialising, or if it looses sync with updates and needs to regenerate its database for one or more MABs, it waits for a snapshot message and records it and all subsequent updates for this MAB. It then downloads the snapshot from one of the RUAS's HTTP servers, unpacks it into RAM and applies all the updates received since the snapshot. This brings this MAB's mapping up-to-date, ready for answering mapping queries and ready to be updated as further updates arrive. In the long-term future, if QSDs ever hold 50 gigabytes or more of Whittle Expires July 17, 2010 [Page 62] Internet-Draft Ivip Architecture January 2010 mapping information (that is around a billion or more IPv6 micronets) it would be good to have a protocol so one QSD could initialize itself by copying live data from another. This is a challenging piece of infrastructure to design and run. However, the code for the Replicator, once written, will serve well for all Replicators. Parts of this code will be re-used in the RUAS and QSD code. There needs to be some kind of monitoring and management system for Replicators, and some localised arrangements for ISPs to collaborate on inter-linking their various levels of Replicators. The FMS is not necessarily expensive in terms of hardware, since the whole RUAS, Launch server, Replicator, QSD and QSC system can be implemented on COTS servers. Replicators store no data and are not involved in complex communication protocols other than the relatively simple functionality described above. Nor is it necessarily expensive in terms of bandwidth for an ISP to receive multiple feeds of mapping updates from Replicators near and far. The exact volume of updates depends on many factors, including how low the price per update is, and how many end-user networks find this price attractive for real-time inbound TE steering of traffic to maximise the utilization of their data links. Market forces will cause RUASes to set the fee per update at a level which encourages deployment of a fast Replicator system, and lowers fees to make updates more attractive for end-user networks - within the limits imposed by the actual cost of running the RUAS and Replicator systems. The use of this system for 10^10 micronets, most of them for mobile devices, should be technically and economically feasible by the time such levels of adoption are achieved. The TTR mobility system does not involve mapping updates when the MN changes its physical address, or moves to another access network. Mapping changes are desirable when the MN's point of connection is far enough away from the TTR to choose a closer TTR, in order to minimise path lengths. This would be infrequent, since MNs do not frequently move such distances of 1000km or more. 5.6. MHF - Modified Header Forwarding 5.6.1. EAF - ETR Address Forwarding for IPv4 Please see [I-D.whittle-ivip-etr-addr-forw] and the discussion above in the ITR section. EAF will not accept fragmented packets or fragmentable packets longer Whittle Expires July 17, 2010 [Page 63] Internet-Draft Ivip Architecture January 2010 than some globally agreed constant, somewhat below 1500 bytes. By the time Ivip is introduced, it will have been over 20 years since RFC 1191 PMTUD was introduced. There's no need for fragments or fragmentable packets - and IPv6 does fine without them. EAF requires upgraded routers between ITRs and ETRs. This does not necessarily include every DFZ router, but it is reasonable to approximate the requirement to this. For instance, if a DFZ router never handles packets for networks which contain either ITRs or ETRs, then it does not need to handle EAF formatted packets. EAF ETR addresses contain only the 30 most significant bits. To avoid the need to change ETRs' addresses when encapsulation is transitioned to EAF, ETRs should not be placed closer than 4 IP addresses apart. Perhaps they should be placed on the 01 address of these four. Since ITRs will commonly be placed deep within ISP and end-user networks, and ETRs may be deep within ISP networks (such as at an end-user site, at the end of the link from the ISP) any router between the DFZ and these devices also needs to handle EAF packets. It will be straightforward to build this capability into new routers, and into firmware updates for many existing routers. The upgrade only concerns the FIB. All that is altered is that the FIB forwards the packet according to the 30 bits ETR address bits in the header, rather than using the destination address. There is no change to BGP functions, the RIB or how the RIB writes to the FIB. Since it is going to take a few years before Ivip or the like is introduced, it is possible that by then, many or almost all of the installed DFZ routers will be able to do this with a firmware update. With a year or two's notice, upgrading all the DFZ routers, and likewise many internal routers, would enable Ivip to be introduced in its final mode of operation - without encapsulation overhead or its PMTUD problems. This means all ITRs can be a lot simpler - and that ETRs can be trivially simple. Reducing the complexity of ITRs is perhaps the biggest challenge in designing a core-edge separation architecture, since we want ITRs to be cheap and plentiful, including them being easy to add to the stacks of sending hosts. Starting with EAF would also avoid the need for devising a transition mechanism from encapsulation. 5.6.2. PLF - Prefix Label Forwarding, for IPv6 The current state of PLF design is described in [PLF for IPv6]. Please see this for more details, including why it is totally different from MPLS and how it could be extended to provide a similar 2^19 or 2^20 destination forwarding system within each ISP (or end- Whittle Expires July 17, 2010 [Page 64] Internet-Draft Ivip Architecture January 2010 user) network. Please refer to the description and example of PLF operation at the end of Ivip-summary.pdf. [Ivip Summary and Analysis] While EAF is pretty much a functional replacement of IPv4's encapsulation system, PLF is rather different in that it only takes the packet to a BR of one of 2^19 or 2^20 DFZ-advertised prefixes. This would be a regular, contiguous, set of prefixes used only by ISPs - for this and for potentially other purposes. If Ivip for IPv6 began with encapsulation, then it would make sense for the ETRs to be already located in these special prefixes. Otherwise, they would need to be moved there before EAF could be turned on. EAF may require a second lookup at the BR of the ISP's network - if there are more than one ETRs for that prefix. One way of forwarding the packet from the BR to the correct ETR would be to use these PLF bits for a similar system within the ISP's network, with 2^19 or 2^20 internal prefixes. How the ISP uses these bits is a private matter. This could be a very powerful way of directing traffic inside a large provider network. This would give rise to ePLF - the one system for the DFZ - and iPLF, as used inside an individual ISP network. Rapid adoption of IPv6 is still somewhere beyond the immediately foreseeable future. So there's no hurry about deploying a scalable routing solution for IPv6. I think the most likely scenario for widespread adoption of IPv6 is one or more large 3G systems using it to give each phone (or whatever) its own global unicast address. This in itself will not cause a scaling problem, since these will be large systems with few new prefixes to add to the DFZ. However, there would then be a strong need for mobility - and the TTR approach has advantages over traditional MIP techniques, including seamless, generally optimal path mobility with the MN on any address, in any access network. Perhaps by the time Ivip is deployed for IPv6, all the IPv6 DFZ routers will be upgradable to PLF with firmware updates - so scalable routing could be done without encapsulation. PLF involves small changes to the FIB and to the RIB. It does not involve any new BGP functionality. 5.7. TTR Mobility TTR Mobility is fully described, with diagrams, in [TTR Mobility]. This architecture will work equally well for IPv4 and IPv6. The MN can be on any kind of address, including behind multiple layers of NAT, on DHCP addresses and on addresses provided by conventional Whittle Expires July 17, 2010 [Page 65] Internet-Draft Ivip Architecture January 2010 Mobile IP protocols. The MN can even be on an SPI address which is within another MN's micronet. No stack or application changes are required and the hosts communicate normally with all other hosts, including of course others using TTR mobility. There is no home- agent and paths to correspondent hosts are generally optimal. Mapping changes are not required when the MN gains a new address. They are not actually required at all, but are desirable if the MN moves to a part of the network which is far from its current TTR. This may be a distance of 1000km or more. Then, it should establish a tunnel to a nearby TTR so the TTR company can change the mapping of its micronet to this new TTR. With Ivip's real-time control of mapping, this means the MN could close the tunnel to the old TTR within five or so seconds of the mapping change being sent. Changing the mapping does not cause any glitch in connectivity, since the MN gets packets from both TTRs during the changeover. The MN needs some additional tunneling software - which is controlled by the TTR company. This could be added alongside existing stacks, or integrated into the stack. Ideally the MN to TTR interface would be standardised in RFCs, but this is not strictly necessary, since the MN only needs to interoperate with TTRs of the TTR company chosen by the MN's owner. RFC-standardised MN and TTR functionality would be desirable, by allowing easy choice between TTR companies without the need to install software. However, there is a lot of scope for innovation in this area, and it might be difficult to adequately develop a full range of desirable protocols soon enough for the expected rapid uptake of mass-market Mobility. I think this approach to mobility, for IPv4 and at some stage for IPv6, is so attractive that there would be a business case for a company setting up its own Ivip-like system just for this purpose - irrespective of the need for a scalable routing solution. Such a system would need to use encapsulation. Multiple such systems could exist at the same time - and a MN in one system A would be able to communicate directly with a MN in another system B via the following paths (->) or tunnels (==>): MN-A ==> TTR-A -> (via DFZ) -> ITR-B ==> TTR-B ==> MN-B. Any such systems should be designed to upgraded in the future to comply with future RFCs for an Ivip-like system, including initial or long-term adoption of Modified Header Forwarding rather than encapsulation. Whittle Expires July 17, 2010 [Page 66] Internet-Draft Ivip Architecture January 2010 6. Security Considerations Security analysis can only be done in the years to come, once the protocols are designed in some detail. Ivip ITRs and ETRs are much simpler than those of LISP. The fast- push mapping system is a tightly structured, easily secured, system for pumping data from one set of sources out to many QSDs. It is unusual, but it is simpler than global query server systems such as LISP-ALT or DNS, and a lot simpler than BGP, with its routing information percolating through the DFZ one router to the next, via complex individual messages. The ALT network carries queries which could be used to infer communication patterns of users, with considerable detail. The global part of Ivip simply pushes mapping data - and so does not handle any such queries. Queries are made, but typically the QSD is in the ISP's network, where security can be well assured - rather than having to trust that every ALT router which handles a mapping query is secure against compromise. Ivip ETRs easily enforce ISP BR source address filtering. For LISP ETRs to enforce this would be administratively complex and very expensive for large numbers of filtered prefixes. Whittle Expires July 17, 2010 [Page 67] Internet-Draft Ivip Architecture January 2010 7. IANA Considerations [To do.] Whittle Expires July 17, 2010 [Page 68] Internet-Draft Ivip Architecture January 2010 8. Informative References [C-E-Sep-Elim] Jen, D., Zhang, L., Lan, L., and B. Zhang, "Towards a Future Internet Architecture: Arguments for Separating Edges from Transit Core", September 2008, . [Constraints-Voluntary] Whittle, R., "List of constraints on a successful scalable routing solution which result from the need for widespread voluntary adoption", April 2009, . [Critique of draft-jen-mapping-00] Whittle, R., "draft-jen-mapping does not apply to the TTR Mobility architecture", January 2010, . [DFZ-unfrag-1470] Whittle, R., "Google sends 1470 byte unfragmentable packets", August 2008, . [Deering-1996] Deering, S., "The Map & Encap Scheme for scalable IPv4 routing with portable site prefixes", March 1996, . [Host-Responsibilities] Whittle, R., "Objections to burdening hosts with more Routing and Addressing responsibilities", December 2009, < http://www.firstpr.com.au/ip/ivip/RRG-2009/ host-responsibilities/>. [I-D.adan-idr-tidr] Adan, J., "Tunneled Inter-domain Routing (TIDR)", draft-adan-idr-tidr-01 (work in progress), December 2006. [I-D.ietf-lisp] Farinacci, D., Fuller, V., Meyer, D., and D. Lewis, "Locator/ID Separation Protocol (LISP)", draft-ietf-lisp-05 (work in progress), September 2009. [I-D.irtf-rrg-recommendation] Li, T., "Recommendation for a Routing Architecture", draft-irtf-rrg-recommendation-03 (work in progress), December 2009. Whittle Expires July 17, 2010 [Page 69] Internet-Draft Ivip Architecture January 2010 [I-D.jen-apt] Jen, D., Meisel, M., Massey, D., Wang, L., Zhang, B., and L. Zhang, "APT: A Practical Transit Mapping Service", draft-jen-apt-01 (work in progress), November 2007. [I-D.jen-mapping] Jen, D. and L. Zhang, "Understand Mapping", draft-jen-mapping-00 (work in progress), October 2009. [I-D.lear-lisp-nerd] Lear, E., "NERD: A Not-so-novel EID to RLOC Database", draft-lear-lisp-nerd-06 (work in progress), December 2009. [I-D.lewis-lisp-interworking] Lewis, D., "Interworking LISP with IPv4 and IPv6", draft-lewis-lisp-interworking-00 (work in progress), December 2007. [I-D.meyer-lisp-cons] Brim, S., "LISP-CONS: A Content distribution Overlay Network Service for LISP", draft-meyer-lisp-cons-04 (work in progress), April 2008. [I-D.rja-ilnp-intro] Atkinson, R., "ILNP Concept of Operations", draft-rja-ilnp-intro-02 (work in progress), December 2008. [I-D.whittle-ivip-db-fast-push] Whittle, R., "Ivip Mapping Database Fast Push", draft-whittle-ivip-db-fast-push-02 (work in progress), January 2010. [I-D.whittle-ivip-etr-addr-forw] Whittle, R., "Ivip4 ETR Address Forwarding", draft-whittle-ivip-etr-addr-forw-00 (work in progress), January 2010. [I-D.whittle-ivip-glossary] Whittle, R., "Glossary of some Ivip and scalable routing terms", draft-whittle-ivip-glossary-00 (work in progress), January 2010. [Ivip Summary and Analysis] Whittle, R., "Ivip Conceptual Summary and Analysis", December 2008, . [Ivip-2007-06-15] Whittle Expires July 17, 2010 [Page 70] Internet-Draft Ivip Architecture January 2010 Whittle, R., "ViP: Anycast ITRs in the DFZ & mobile tunnels", June 2007, . [LISP-ALT-Critique] Whittle, R., ""How can the ALT structure scale to 10^8, 10^9 or 10^10 EIDs with minimal delay times and robustness against single points of failure?"", December 2009, . [Namespace] Whittle, R., "The meaning of the term *namespace* in addressing, computer networking etc.", April 2009, . [PLF for IPv6] Whittle, R., "Prefix Label Forwarding (PLF) - Modified Header Forwarding for IPv6", August 2008, . [PMTUD-Frag] Whittle, R., "IPTM - Ivip's approach to solving the problems with encapsulation overhead, MTU, fragmentation and Path MTU Discovery", April 2008, . [TRRP] Herrin, W., "Tunneling Route Reduction Protocol (TRRP)", August 2007, . [TTR Mobility] Whittle, R. and S. Russert, "TTR Mobility Extensions for Core-Edge Separation Solutions to the Internets Routing Scaling Problem", August 2008, . [Vogt-2009] Vogt, C., "Simplifying Internet Applications Development With A Name-Based Sockets Interface", December 2009, . [loc-id-sep-vs-ces] Whittle, R., "Loc/ID Separation is different from Core- Edge Separation", January 2010, . Whittle Expires July 17, 2010 [Page 71] Internet-Draft Ivip Architecture January 2010 Appendix A. Acknowledgements Thanks to the following people for their help and encouragement: Juan Jo Aden, Noel Chiappa, Olivier Bonaventure, Brian Carpenter, Dino Farinacci, Vince Fuller, Joel M. Halpern, Geoff Huston, Ved Kafle, Eliot Lear, Simon Leinen, Tony Li, Jeroen Massar, Dave Meyer, Chris Morrow, Dave Oran, Robert Raszuk, Jason Schiller, John Scudder, K. Sriram, Markus Stenberg, Letong Sun, Christian Vogt, Kilian Weniger and Xiaoming Xu. This is not to imply that these people support Ivip. I especially thank Steve Russert, formerly of Boeing, for collaborating on the TTR Mobility paper for MobiArch '08. The original draft wasn't accepted and by the time we revised it to the point of being happy with it, the paper was 2.5 times as long as the conference page limit. Whittle Expires July 17, 2010 [Page 72] Internet-Draft Ivip Architecture January 2010 Author's Address Robin Whittle First Principles Email: rw@firstpr.com.au URI: http://www.firstpr.com.au/ip/ivip/ Whittle Expires July 17, 2010 [Page 73]