Internet Engineering Task Force Walter Milliken INTERNET-DRAFT BBN draft-ietf-ipatm-arch-00.txt July 7, 1995 IP Multicasting over ATM: System Architecture Issues Status of this Memo This document is an internet draft. Internet Drafts are working documents of the Internet Engineering Task Force (IETF), its Areas, and its Working Groups. Note that other groups may also distribute working documents as Internet Drafts. Internet Drafts are draft documents valid for a maximum of six months. Internet Drafts may be updated, revised, replaced, or obsoleted by other documents at any time. It is not appropriate to use Internet Drafts as reference material or to cite them other than as "working draft" or "work in progress". Please check the 1id-abstracts.txt listing contained in the internet-drafts shadow directories on nic.ddn.mil, nnsc.nsf.net, nic.nordu.net, ftp.nisc.src.com, or munnari.oz.au to learn the current status of any Internet Draft. 1. Abstract This document summarizes the various issues associated with implementing IP multicast over ATM subnets and wide-area networks. The reader is assumed to have read the Internet Draft "IP over ATM: A Framework Document" [1], which classifies various types of ATM subnet and end-to-end models, and also describes proposals for IP operation over those models. The terminology and various network models discussed here are drawn primarily from that document, and the reader is assumed to be familiar with them. This document describes IP multicast over three ATM subnet models: o an SVC-based LAN ATM (LATM) subnet, o a PVC-based WAN ATM (WATM) subnet without multicast services -- a non-broadcast, multiple-access (NBMA) subnet), and o an SVC-based WAN ATM (WATM) subnet supporting multicast. Milliken INTERNET-DRAFT Expires: January 12, 1996 [Page 1] IP Multicasting over ATM: System Architecture Issues The end-to-end models discussed here include: o the Classical IP model, o Ohta's "Conventional" model, o the SVC-based WATM (ROLC) model, o the "integrated" model, and o peer models. 2. Introduction The IP over ATM Working Group of the Internet Engineering Task Force (IETF) is chartered to develop standards for routing and forwarding IP packets over ATM sub-networks. One component of this work is the development of standards for support of IP multicast in such an environment. This document provides an overview of the issues associated with support of IP multicasting over ATM, and discusses the basic components needed for an IP over ATM multicast system architecture. The remainder of this document is organized as follows: o Section 3 defines several terms relating to multicasting, o Section 4 describes some characteristics of multicast applications and their use of multicast services, o Section 5 is a basic service model for IP multicasting over ATM, o Section 6 discusses some of major issues for implementing IP multicast services over ATM, o Section 7 discusses the specific issues in implementing IP multicasting over several types of ATM-based subnets defined in [1], o Section 8 addresses issues related to IP multicasting in the context of several end-to-end models defined in [1], and o Section 9 summarizes the issues that need to be addressed in an IP multicast over ATM system architecture. Milliken INTERNET-DRAFT Expires: January 12, 1996 [Page 2] IP Multicasting over ATM: System Architecture Issues 3. Definitions and Terminology The definitions from [1] are used in this document. In addition, a few more terms related specifically to multicasting are defined here. A Multicast-capable Host is one that can send and receive IP packets addressed to IP multicast (class D) addresses. A Multicast-capable Router is one that can relay IP packets addressed to IP multicast (class D) addresses. (No particular multicast routing protocol is assumed here.) An RSVP-capable Host is one which can send and receive RSVP protocol packets to establish quality of service for a flow of IP packets. An RSVP-capable Router is one which can process RSVP protocol packets and use the information therein to enforce a requested quality of service along a path between hosts. 4. Characteristics of IP Multicast The use of IP multicast differs in a number of ways from how traditional IP unicast services have been used. Besides the fundamental use of multicast distribution, many multicast applications make additional demands on network services, which should be addressed in any design for IP multicast over ATM. 4.1. Distribution Models The basic service provided by IP multicast is a logical address to which any sender may send datagrams, without knowledge of the actual addresses of any receivers. Any receiver may join at will, receiving any datagrams addressed to the group address. This service is described in RFC 1112, "Host Extensions for IP Multicasting [2]. The current ATM standard, UNI 3.0 [3], provides a connection- oriented, unidirectional point-to-multipoint service, with sender-initiated connection to each individual receiver. Extensions for receiver-initiated joining and for multipoint- to-multipoint service have been proposed. Multicast applications tend to use a few basic distribution patterns: o 1-to-Any: used primarily for service location, to locate a server whose actual address is not known. A "session" is typically one packet. The set of receivers is relatively static, the transmitter set is highly dynamic. Milliken INTERNET-DRAFT Expires: January 12, 1996 [Page 3] IP Multicasting over ATM: System Architecture Issues o 1-to-N: a single source with many receivers, typical of lectures or video distribution. o N-to-N: a number of participants, each is both a sender and a receiver. Typical of audio conferences and some small video conferences, as well as some distributed computations. o M-to-N: a number of senders with a larger number of receivers, typical of large video conferences and some large distributed simulations. o 1-to-All: while not technically a multicast distribution mode, the IP subnet broadcast addresses (x.255.255.255, x.y.255.255, x.y.z.255, etc.) can be thought of as a predefined multicast address. 4.1.1. Distribution Scope: IP Time-To-Live Besides the basic distribution model, many IP multicast applications make use of the IP header Time-To-Live (TTL) field to limit the scope of distribution. Thus, a multicast session can have a distribution limited to a particular segment of the network topology. This is most often used to limit the scope of a multicast group to the local subnet (i.e., TTL=1). 4.2. Real-Time Requirements Examination of the typical multicast applications shows that many of them share a common requirement for real-time performance. Thus, any design for IP multicast over ATM needs to give strong consideration to supporting not only multicast distribution, but also providing quality-of-service (QoS) support. 4.2.1. QoS Support (Resource Reservation) In the IP multicast service model, QoS support is provided by the RSVP resource reservation protocol [4]. RSVP is a receiver-oriented reservation protocol using soft-state in IP routers to maintain reservation information based on receiver capabilities. One of its primary goals is the support of heterogenous receivers. Thus, reserved bandwidths over different links may be different, and is based on the requirements of downstream receivers. Traffic in excess of the reserved bandwidth is sent "best-effort" and is subject to additional delay and/or discard. Milliken INTERNET-DRAFT Expires: January 12, 1996 [Page 4] IP Multicasting over ATM: System Architecture Issues RSVP also supports the notion of "traffic filters" to classify packets as belonging to a particular reservation. Packets are classified by inspecting arbitrary fields specified by the application. Another characteristic of RSVP is that reservations can be dynamic -- the application can change them at any time. In contrast, current ATM QoS support is static, set up at connection time, and parameters cannot be changed unless the connection is torn down and re-initiated with the new parameters 4.3. Second-generation multicast applications Recently, some applications have appeared that make additional use of multicast services beyond a simple "broadcast to all session participants" model. These applications include multiple data flows to different subsets of participants, or use multicast groups in a more dynamic way than joining a single group for the duration of the session. Applications that could be classified as "second-generation" are multilevel-encoded video (both "lecture" and "conference" distribution models) and distributed interactive simulation. 4.3.1. Multiple flows Some applications may use multiple multicast flows per session. Multilevel-encoded video, for example, can use a group per encoding level, while distributed simulations can use hundreds or thousands of flows per session, representing different "communities of interest" in the simulation -- neighboring objects often need to communicate state to each other, while they do not need state from distant objects. (This characteristic applies to both virtual-reality applications like distributed interactive simulation, and to some types of scientific simulations.) 4.3.2. Low join latency Another characteristic of some second-generation multicast applications is the need for low group-join latency. This is especially important for distributed interactive simulations, since they involve "human in the loop" aspects, but other real-time distributed systems may have similar requirements. Even non-real-time distributed algorithms may perform much better if communication patterns can shift rapidly, since distributed algorithms are often constrained in performance by the slowest action of the slowest participant. Milliken INTERNET-DRAFT Expires: January 12, 1996 [Page 5] IP Multicasting over ATM: System Architecture Issues 4.4. Link-sharing While not an actual end-user requirement, sharing of link bandwidth among different flows within a session is highly desirable -- many multicast applications exhibit behavior that allows more efficient network utilization if resources can be shared between flows from different sources, or even different multicast groups, within the same session. (This is separate from any statistical multiplexing the network might choose to do between flows from different sessions.) RSVP offers a limited form of link-sharing for applications using the N-to-N or M-to-N distribution models: a receiver reservation may be a "wildcard" reservation, applying to all sources. Thus, if flows from different sources, but to the same multicast group, share a link, both may use the same reservation. This is especially useful in audio conferencing applications, where there are many potential senders, but only one or two are usually sending at any given time. (This type of link-sharing does not have any exact mapping to existing ATM services, due to the separation of flows by sender.) Beyond any existing capability in either IP/RSVP or ATM is the concept of general sharing of link bandwidth between groups. This is primarily useful in applications using large numbers of small, dynamic flows, such as distributed interactive simulation. In such an application, the overall traffic is often fairly predictable, but gets subdivided into a shifting set of flows to different destination subsets. The ATM virtual path capability may be useful in this situation, but is limited to one level of hierarchical decomposition, and either may not be accessible to end-users, or may not be supported by the ATM networks in use. 5. Basic IP Multicast over ATM Service Model This section proposes a basic service model for IP multicast over ATM. The service model consists of three components: o Subscription (attachment of receiving hosts to the multicast group), o Distribution (delivery of multicast packets to appropriate destinations), and o Quality of service support (resource reservation and real-time support). Milliken INTERNET-DRAFT Expires: January 12, 1996 [Page 6] IP Multicasting over ATM: System Architecture Issues 5.1. Subscription IP multicast hosts must be able to: o Send to any multicast group at any time, and o Register as a receiver of a multicast group. IP multicast routers must be able to: o Receive all multicast packets on each subnet to which they are attached, and o Determine whether any hosts on an attached subnet are members of a specific multicast group. All multicast-capable hosts also must subscribe to the "all- hosts" group at host initialization time, and remain a member of that group as long as they remain up. (This is required to support IGMP, and is mandated by RFC 1112. If IGMP is not to be supported, then this may no longer be necessary.) All hosts are implicitly members of the IP subnet broadcast "group" (x.255.255.255, etc.) for their local subnet. 5.2. Distribution The standard IP distribution model should be supported, which includes the ability of any IP host to send traffic to any group, and the ability to limit distribution scope using the IP TTL field. The restriction of distribution to the local subnet (TTL=1) must be supported to avoid problems with a number of multicast protocols which assume a TTL of 1 will keep traffic from entering other subnets. Per RFC-1112 [2], each multicast-capable subnet must support the "all-hosts" multicast group (224.0.0.1), which must include every multicast-capable host and router. (If IGMP is not supported, then there may be no need for this requirement.) The IP subnet broadcast address should be supported. Support for the IP "global broadcast" address 255.255.255.255 is not required, and indeed should not be provided (according to the "Router Requirements" RFC-????). 5.3. Quality of Service IP hosts use RSVP to indicate quality of service requirements to the network for IP multicast groups. This should be true on ATM subnets as well, since RSVP, unlike IGMP, is an end- to-end protocol. Using a different QoS mechanism for ATM subnets will raise major interoperability issues with non-ATM subnets. Milliken INTERNET-DRAFT Expires: January 12, 1996 [Page 7] IP Multicasting over ATM: System Architecture Issues 6. Major Issues for IP Multicast over ATM This section discusses various aspects of the IP multicast service model as it impacts design and implementation over ATM services. 6.1. Distribution The primary function of IP multicasting is to distribute a packet to multiple receivers, preferably in an efficient way. In an ATM-based network, this can be done in three ways: o The source uses a point-to-multipoint VC to the destinations; packet replication is then performed by the ATM switch, which replicates the cells in each packet. o The source sends the packet to an ATM multicast server (which must operate at the AAL5 level if there are multiple sources). The server distributes its input either over multiple point-to-point VCs or a point-to- multipoint VC. o The source sends a copy of the packet to each destination over a point-to-point VC. (This is how IP multicast routers classically handle NBMA wide-area networks.) An IP multicast implementation could make use of any of the ATM link-layer multicast mechanisms. The best technique will depend on traffic characteristics, especially the size and number of multicast groups. The MARS server of [5] supports either of the first two techniques, on a per-group basis. Hybrid schemes using elements of both IP-level and ATM-level multicast, such as that proposed in [6], are also possible. 6.1.1. Reflected packets from multicast servers One problem that arises with the use of multicast servers at the ATM level is that packets arriving from them are anonymized at the link level. That is, it is not possible to determine the ATM address of the link-layer source, since it always appears to come from the multicast server. This is a minor problem for hosts, since they can look at the source address in the IP header to determine if it came from themselves. Also, hosts may wish to receive their own multicast traffic -- in current implementations, this is usually configurable on a per-group basis, and can require the host software to duplicate the packet internally, for half-duplex interfaces. Milliken INTERNET-DRAFT Expires: January 12, 1996 [Page 8] IP Multicasting over ATM: System Architecture Issues Reflected packets from a multicast server can be a much more significant problem for IP multicast routers, however. The router must be able to distinguish its own packets to avoid looping them back again. In this case, the IP source address isn't helpful, since it is the address of the originator of the packet, not the gateway. Some, but not all, IP routing algorithms can detect such looped packets by the fact that they arrived on the "wrong" interface. Others cannot. One solution to the reflected packet problem is to make the router itself into the multicast server, and forbid the use of ATM-level multicast servers that are not also IP routers. In this case, routers would be forbidden to allow other routers to be destinations of point-to-multipoint connections being used for multicast servers connections. Traffic arriving for the multicast server would be delivered internally to the IP routing level, and the router's own transmissions would never return to it. In the case of multiple routers on the ATM subnet, separate point-to-point (or point-to-multipoint) connections could be set up between routers for traffic being forwarded explicitly between routers. Copies for local hosts would be sent separately. 6.1.2. Address resolution In ATM without LAN emulation (i.e., an NBMA ATM LAN), one problem that arises is that the sender of an IP multicast packet must map an IP multicast group address to an appropriate VC, with the additional complication that such a VC may not already exist, and hence may need to be set up. This mapping cannot be algorithmic, as it is for 802.x, because VPI/VCIs are dynamically assigned by the ATM switch. A host sending an initial packet to an IP multicast address on an ATM subnet cannot locally determine where it must be sent, since there is no easy way for it to learn the identities of recipients in the absence of a broadcast medium. Armitage's MARS proposal [5] uses a variant of the ATMARP server described in [7] to provide a list of ATM addresses of registered group members; the sending host must then set up a point-to-multipoint VC to this list. (The "list" may also be the address of one or more multicast servers serving the group, and may also include IP routers.) Reference [6] proposes that multicast hosts on an NMBA ATM subnet forward initial packets over a point-to-point VC to a local IP multicast router, which distributes them appropriately; the host may subsequently set up a point-to- multipoint VC, if desired. Milliken INTERNET-DRAFT Expires: January 12, 1996 [Page 9] IP Multicasting over ATM: System Architecture Issues 6.2. Subscription Receiving hosts classically subscribe to IP multicast groups using IGMP, per RFC 1112 [2] -- they send out an IGMP Host Membership Report to the "all-hosts" IP multicast group (224.0.0.1). The difficulty with IGMP on an NBMA ATM subnet is that the protocol is designed with a broadcast subnet in mind. In particular, it presumes the ability for all hosts to trivially send to and receive from the "all-hosts" group, and the ability of IP routers to trivially receive all multicast traffic on attached LANs, including IGMP messages. The primary requirements for a subscription mechanism are: o It receives join requests from group members, o It maintains a mapping from an IP multicast address to one or more destination addresses for use by the distribution mechanism (either ATM or IP host addresses will serve, since ATMARP can map one to the other), and o It provides sufficient information to senders for them to set up a VC to the proper distribution (though this may be a point-to-point VC to a multicast server). The MARS proposal in [5] substitutes part of the MARS protocol for IGMP on ATM subnets. Instead of sending IGMP Host Membership Reports, receivers register interest in an IP multicast address with the MARS server. IP multicast routers use special features in the MARS protocol to register as receivers of blocks of multicast address. Reference [6] proposes that IGMP be retained for subscription on ATM subnets, but requires some changes in router handling of IGMP traffic. Routers also are required to function as multicast servers for IP multicast traffic originating on attached subnets, so that hosts are not required to set up point-to-multipoint VCs. In this case, hosts need only know the ATM address of the IP router, and only the router needs to know the ATM addresses of group members (which can be obtained using ATMARP). For the various advanced end-to-end models employing "cut- through" routing, more problems may arise, since the sending IP host normally has no way of determining the set of receivers. The MARS notion in [5] could be extended to cover such cases, though there may be scaling and routing protocol issues, or the ROLC NHRP protocol may address it using a routing server. The proposal in [6] cannot support "cut- through" for best-effort traffic, but can use a mechanism such as NHRP when making QoS-based connection setups. Milliken INTERNET-DRAFT Expires: January 12, 1996 [Page 10] IP Multicasting over ATM: System Architecture Issues 6.2.1. Subnet broadcast There is no guarantee that all hosts are multicast-capable, and thus registered with a MARS server (per [5]) or IP multicast gateway (per [6]). However, the ATMARP server (described in [7]) does have a list of every currently- running host on the subnet (by definition). Since subnet broadcasts are infrequently used, it may be possible to add subnet broadcast support to the ATMARP server. Subnet broadcast packets would be sent to the ATMARP server, which would maintain a point-to-multipoint VC on which such packets would be forwarded. Alternatively, access to the ATMARP database could be given to an IP multicast server, MARS, or IP multicast router, either by a protocol extension, or by making the ATMARP server coresident with one of the above functions. Arguments for VC conservation suggest that it is likely that ATMARP, MARS, and IP multicast router functions are all likely to be coresident in the same box, otherwise a host is likely to need a separate VC open to each. 6.3. QoS Setup and RSVP 6.3.1. RSVP overview This is a simplified overview of the RSVP protocol. Details can be found in reference [4]. The RSVP protocol provides receiver-specified resource reservation, including support for heterogenous receivers. Senders send RSVP "Path" messages to the IP multicast group, while receivers send "Resv" reservation messages to RSVP- capable upstream nodes (which may be hosts or routers), along the reverse route established by the Path messages. RSVP-capable routers and hosts receive Resv messages and extract reservation information. When multiple receivers specify a reservation relating to the same source(s) and multicast group, the reservations are combined into a single reservation, representing the largest of the receiver requirements. Routers then forward the combined reservation further upstream in a new Resv message. This reduces the implosion problem when there are many receivers. An interesting consequence of the design of the RSVP protocol is that every receiver of a group on a subnet periodically sends a Resv message to every sender on the subnet, where receivers and senders may either be hosts, or routers "proxying" for collections of off-subnet senders and receivers. This is exactly the information needed to set up ATM multicast VCs with proper QoS (except for the ATM addresses, which can be obtained from ATMARP). This suggests that RSVP exchanges might be used to set up appropriate VCs, Milliken INTERNET-DRAFT Expires: January 12, 1996 [Page 11] IP Multicasting over ATM: System Architecture Issues given that some method exists of sending a small amount of multicast traffic so that the necessary RSVP messages can be exchanged. This observation is the basis for the mechanism proposed in [6]. 6.3.2. ATM interactions with RSVP Because Resv messages are sent to a source along the route established by a Path message, at least one Path message must arrive at a receiver (or intermediate router with an existing reservation including that source) before a Resv message can be sent to that source. This implies that if RSVP signalling is used for QoS support, ATM SVCs cannot be set up with proper QoS parameters until a Resv message arrives at the source. Yet the first Path message must be sent out from the source to the multicast group, before the QoS parameters can be known. This implies that QoS support must be a two-stage process. The first stage sets up a temporary best-effort multicast VC, or uses some existing VC (such as one to a multicast IP server) to send the initial RSVP messages. Once Resv messages start arriving, a QoS-based VC can be set up, and the best-effort path is no longer needed. Another problem with RSVP is that ATM currently lacks the ability to change QoS parameters "on the fly" or support different QoS parameters to heterogenous multicast receivers. Changing parameters would require the ability to signal changes to ATM VCs after setup. Heterogeneous receiver requirements would require ATM switches to selectively drop traffic at branch points in the distribution tree forming the point-to-multipoint VC; this is likely to be difficult to implement without auxiliary hardware operating on AAL5 frames. Since RSVP allows both reservation changes and heterogenous receivers, something needs to be done when differing Resv messages arrive for the same session at the same location. Resv messages from different receivers can arrive in any order, and the number arriving is unknown (since receivers may join at will), so there is no way to determine when the "largest" reservation has arrived. There are three obvious solutions for this problem: o Require that all receivers in a session over ATM multicast links use identical Resv messages. o Use the source description(s) in the Path message(s) to set bandwidth requirements for the ATM multicast VC. The first Resv message sets the other QoS parameters (particularly delay). Subsequent Resv messages just identify additional receivers to be added to the VC. Milliken INTERNET-DRAFT Expires: January 12, 1996 [Page 12] IP Multicasting over ATM: System Architecture Issues This behaves in an acceptable way, but is inefficient -- receivers may get more data than requested, and the technique can prevent link sharing between multiple sources for the same session. o Require that a router (or source host) set up one VC for each distinct set of QoS parameters in received Resv messages for the same session. The router (or host) would then have to do additional packet replication. This solution also has the potential to make less efficient use of the network -- VCs for the same session would each get a copy of every packet, and such VCs might easily share network links. Note that either of the latter two solutions can be applied locally, and different routers in the same session can pick either without compromising the requested QoS to the application. Which solution is the most efficient overall will depend on the distribution of the heterogenous receivers, the number of distinct sets of QoS parameters requested, and the topology of the network. Thus, it is probably best to start with a simple solution and develop experience with it, to see if a more complex technique is required. 6.3.3. QoS support within a subnet ATM connections within the sender's subnet must somehow be set up using the appropriate QoS, without router intervention. This requires hosts to implement additions to the RSVP Resv processing to set up QoS-based VCs for outgoing traffic. Alternatively, if an RSVP-capable router is available, QoS-based traffic could be sent directly to it, for redistribution on the local subnet. Both approaches are allowed in [6], with different levels of service guarantees. 6.4. Routing over point-to-multipoint PVCs and SVCs Another area that may need attention is the handling of IP multicast routing in the presence of ATM point-to-multipoint interconnections, especially PVCs. Current network technologies support link-layer broadcast and multicast primarily on LANs, and wide-area networks have historically used point-to-point trunking between routers. Also, ATM link-layer multicast differs from current LAN multicast technologies because it is neither commutative nor transitive -- the fact that A can send to B and C in a link layer multicast implies nothing about the connectivity of B or C to A, or to each other. Milliken INTERNET-DRAFT Expires: January 12, 1996 [Page 13] IP Multicasting over ATM: System Architecture Issues Thus, IP multicast routing technology may need significant adaptation to operate over ATM point-to-multipoint links. The problem may be less significant in an SVC environment, since the routers can set up connectivity as needed, rather than having to discover and make use of point-to-multipoint PVCs. One technique for exploiting point-to-multipoint PVCs is bi- level multicast, but it is currently restricted to certain topologies (LANs interconnected by a single ATM WAN), and it may have problems scaling to a "large ATM cloud" environment like ROLC. 7. IP Multicasting over ATM Subnets This section discusses some basic issues in implementing IP multicasting over various types of ATM subnets. IP connectivity between subnets is assumed here to be handled normally by IP routers (i.e., the "classical" end-to-end model). Adapting to other end-to-end models is discussed in the following section. 7.1. SVC-based ATM LAN (This subnet model is essentially the same as the LATM subnet model in [1].) This is the typical "campus network" of many hosts connected on a link-layer network based on ATM technology. Transit delays are likely to be small, and the network may or may not be connected to an IP internet. An RFC 1577 ATMARP server is assumed to exist on the subnet to support unicast IP over point-to-point ATM connections. The multicast subscription problem can be handled by the MARS server approach of [5]. If an IP multicast router is attached to the subnet,the IGMP router-based solution of [6] will also work. Distribution can be supported through point-to-multipoint VCs (requiring host support) or multicast servers, or a combination. The approach in [6] requires an IP router to serve as a multicast server; it also supports direct point- to-multipoint connections for QoS-based data flows. If host-originated point-to-multipoint VCs are used, a scaling problem arises with applications using N-to-N and M- to-N distribution models -- there is an incoming VC at each host for each other host participating in a session. Since SAR resources need to be allocated to each VC, and SAR resources are often limited in number, direct multicast VCs are best used only for sessions with small numbers of senders, and multicast servers may be needed for sessions with many senders. Milliken INTERNET-DRAFT Expires: January 12, 1996 [Page 14] IP Multicasting over ATM: System Architecture Issues The MARS mechanism currently offers no solution to the QoS problem -- connections are set up when the sender first tries to send a packet, so it can only set up best-effort service. Additional mechanism would be needed to set up QoS-based connections later, which would have to supplant the MARS- generated ones. Or applications could be required to declare QoS parameters before sending traffic -- these could then be used in the VC setup. But this is contrary to the current philosophy of IP multicast, and will not interoperate well with RSVP-based QoS support. The IGMP/RSVP design in [6] inherently supports QoS-based service -- in fact, it only works well if most high-bandwidth IP multicast traffic uses QoS services. 7.2. PVC-based ATM WAN (This is essentially the WATM subnet model from [1].) This model assumes that most or all "hosts" on the ATM subnet are routers. Delays across VCs will often be substantial, on the order of tens of milliseconds. This subnet model corresponds to current WAN technology where routers are interconnected by point-to-point leased lines. If the PVCs are not set up as constant bit rate (CBR) circuits, routers may have difficulty providing QoS support, unless a variety of PVCs with different qualities of service are configured. This raises routing issues, however, unless every inter-router "link" is composed of a similar set of QoS-based PVCs. There are two subcases of this model, one where point-to- multipoint circuits are unavailable, or unused, and one where PVC point-to-multipoint connections are part of the topology. 7.2.1. Without point-to-multipoint Multicasting in this environment is the same as current IP multicast architectures based on leased lines -- routers replicate packets, once for each outgoing interface the packet is forwarded on. No new work is required to support this subnet model, beyond the basic ability to use ATM as a link layer below IP. One drawback to this subnet model is that IP routers may be far from the ATM backbone, down long-delay tail ciruits. Traffic may travel down the tail circuit to the router, merely to be rerouted back up it along a different PVC. This both reduces available bandwidth, and increases end-to-end delay -- an undesirable characteristic for multicast applications, which are often both bandwidth-hungry and delay-sensitive. Thus, the ideal topology for this subnet model is to have IP routers co-located with the ATM backbone switches. Milliken INTERNET-DRAFT Expires: January 12, 1996 [Page 15] IP Multicasting over ATM: System Architecture Issues 7.2.2. With point-to-multipoint This subnet model adds to the previous model the ability to use point-to-multipoint PVCs. IP routers could make use of such links for sending multicast traffic to other routers to which they are directly connected by point-to-multipoint PVCs, and let the ATM switching fabric perform the necessary packet replication. More complex schemes are possible, such as using sets of PVCs covering different subsets of "adjacent" routers. This is the technique used in the "bi-level multicast" design. Current routers do not make use of multicast connections -- additional work is required in routing protocols and forwarding techniques to do this. Besides offloading packet replication to the ATM fabric, this technique also lowers delay (since packets don't need to be copied in routers), and delays introduced by tail circuit round-trips can also be minimized, if the PVC topology is cleverly chosen. The multicast PVC topology can also be simplified if packets can be delivered to routers that wouldn't ordinarily want them. This consumes additional bandwidth, but may be a useful tradeoff, since the more complex the PVC topology is, the more likely individual PVCs will be underutilized. 7.3. SVC-based ATM WAN This subnet model is a composite of the two previous models. It corresponds to the current vision of a public ATM network with an IP overlay. Like the PVC WAN, most "hosts" are actually routers, though hosts may also be present. Delays across VCs may be substantial. Like the SVC LAN model, hosts and routers may open VCs dynamically, to support traffic. This model is similar to the ATM network environment assumed by the ROLC end-to-end model. Two subcases are considered, depending on whether point-to- multipoint SVCs are supported. 7.3.1. Without point-to-multipoint This case is analogous to current WANs based on IP routers supporting dial-up trunking. Current IP multicast routers should function normally in this environment, though changing the topology frequently may cause difficulty with the RSVP protocol, which depends on periodic refreshes of reservation data to track routing changes. (It is possible, however, for the router to trigger such an update when it detects a relevant change in the IP topology.) Milliken INTERNET-DRAFT Expires: January 12, 1996 [Page 16] IP Multicasting over ATM: System Architecture Issues Multicast support to hosts on such a network would be similar to that needed in the SVC LAN case, except that the router would be forced to function as a multicast server, replicating packets out multiple point-to-point VCs to hosts. 7.3.2. With point-to-multipoint This subcase combines aspects of both the SVC LAN model (to support point-to-multipoint connects between hosts, or hosts and a router), and the PVC WAN with multipoint VC model, in that routers must support inter-router multicast circuits, and deal with the consequent routing issues. Since point-to-multipoint SVCs can be set up between routers when needed, they can better correspond to IP multicast distribution trees, and quality of service support is fairly straightforward. If routers function as local subnet multicast servers, it may be inappropriate to mix hosts and routers as destinations in the same multicast VC, even if the VC is associated with a single multicast group -- otherwise, there is a potential for routing loops. (This is one of the problems with the model proposed in [6].) 8. IP Multicasting over ATM End-to-end Models This section discusses various IP/ATM end-to-end models, and issues related to IP multicasting in the context of each. These models are described in further detail in [1]. 8.1. The Classical IP model The classical IP model of subnets connected by IP routers presents no major difficulties, once the problem of handling IP multicast on ATM subnets is solved. This model has several desirable features, beyond straightforward implementation. In particular, it offers reduced numbers of VCs, both at hosts and routers. Receiving hosts may only need a single VC per multicast group (depending on whether "shortcut routing" is done at the subnet level, or whether all subnet multicast is done through the router). Multiple flows between routers with the same, or similar, destination sets and QoS parameters may share VCs in the wide area network. Drawbacks to this approach are primarily logistical in nature. Routers may need to terminate large numbers of VCs, especially when supporting QoS-based data flows, requiring substantial SAR resources. Also, with public ATM networks, routers are likely to be on the fringes of the ATM "cloud", and may be at the end of long ATM tail circuits, leading to a small, high-speed ATM backbone. Such an IP and ATM topology Milliken INTERNET-DRAFT Expires: January 12, 1996 [Page 17] IP Multicasting over ATM: System Architecture Issues may introduce excessive link delays. This problem can be eliminated by co-locating IP routers with the ATM backbone nodes. In this end-to-end model, the RSVP mechanism will work almost unchanged -- routers use point-to-point links in the WAN, though they may need to set up and tear down SVCs supporting appropriate QoS between routers. This setup and teardown can be triggered straightforwardly by RSVP messages. 8.2. Ohta's "Conventional" model Ohta's "Conventional" model is similar to the "classical" model, except that routers forward ATM cells, rather than IP packets. Except for the potential for latency advantages, the conventional model should support IP multicast in exactly the same way as the classical model. The only issue is that care must be taken to insure that TTL- based distribution scope control works properly, since Ohta's model does not forward IP packets, but rather the individual cells comprising them. For point-to-point ATM connections, this is not a serious problem, since the appropriate TTL modification can be performed before the IP packet is injected into the ATM connection, assuming the number of IP router "hops" covered by the ATM connection is known. For point-to-multipoint connections, however, different receivers may be different numbers of IP "hops" from the sender. This implies that TTL modification would have to be done at the receiving end of the connection (again requiring some knowledge of the "distance" to the sending end of the ATM connection). The primary consequence of this is that some routers (or hosts) may receive multicast packets that they otherwise shouldn't have gotten, which would then have to be discarded. This could have both security and network efficiency implications, depending on the overall system topology. Additionally, there may be problems with RSVP protocol message implosions where large numbers of receivers are connected at the ATM level to a single sender. Solving these problems may require hybrid models where intermediate IP routers are used as multicast servers, or the RSVP protocol is modified to surpress periodic Resv refresh over ATM connections. Milliken INTERNET-DRAFT Expires: January 12, 1996 [Page 18] IP Multicasting over ATM: System Architecture Issues 8.3. The SVC-based WATM (ROLC) model This model assumes a "large ATM cloud" with IP hosts and subnets at its periphery. ATM connections are "cut-through" end-to-end between hosts attached to interconnected ATM networks. Conventional IP routers are used only to connect non-ATM subnets, or ATM networks that are not interconnected at the ATM level. Presuming the solution of the point-to-point connection problem, such as a "next hop" routing mechanism that translates a destination IP address into an ATM endpoint address, the ROLC model can straightforwardly support IP multicast by a mechanism such as that described in [6], where RSVP is used as the basis for VC setup. However, an initial multicast distribution path must be set up to exchange RSVP messages. This would probably best be implemented using a multicast packet overlay, similar to the classical model, though actual subnets (in the sense of IP address blocks) are not required. The MARS mechanism of [5] can also be generalized to the ROLC model, though this presupposes that multicast routing protocols such as PIM [8] and CBT [9] can be adapted to "next hop" routing. Also, setup of VCs with appropriate QoS still requires some additional mechanism beyond basic routing. The ROLC model has the advantage that routers (or more appropriately, route servers) do not need to be as concerned about the number of potential VCs terminated, since this depends only on the number of clients the route server has. However, there are also two disadvantages to the ROLC model for IP multicast traffic, both "implosion" problems. First, RSVP may produce an implosion problem when a node has many "downstream" receivers sending it Resv messages periodically. Fixing this probably requires adjusting the rate of Resv messages, or using the "hard" state of the ATM VC to eliminate the need for sending Resv messages over ATM-based links. The second implosion problem is that a large session using the N-to-N or M-to-N distribution model can result in a large number of VCs terminating at a host. The classical model avoids this, since all sources sending to the same multicast group from outside the subnet share a single VC. Another problem with the ROLC model is support for IP subnet broadcast. This would likely have to be handled in the route server, since determining the extent and membership of a "subnet" in the ROLC model is somewhat difficult. Milliken INTERNET-DRAFT Expires: January 12, 1996 [Page 19] IP Multicasting over ATM: System Architecture Issues Also, like the "conventional" model, the ROLC model will have difficulty in limiting the distribution scope of IP packets based on TTL values. 8.4. The Integrated Model This model allows ATM switches to share routing information with IP routers, allowing IP routing computations to take ATM topology into account. The integrated IP/ATM routing model does not appear to present any special difficulties with regards to multicasting, though developing an integrated multicast routing algorithm may be troublesome in such an environment, given the difference in orientation between receiver-oriented IP and source-oriented ATM. The remaining issues posed by this model appear to be similar to those posed by the classical and conventional subnet models, and the ROLC model. 8.5. Peer Models Peer models place IP routers and ATM switches as peers in routing information exchanges. Given the current state of peer models, it isn't yet clear what specific multicast issues will arise in that context. It seems that at least one issue likely to arise is the conversion between ATM call setup QoS signalling and RSVP. Many of the multicast issues discussed under previous models may also arise. 9. System Architecture Issues Summary This section summarizes the issues that must be addressed by any IP multicasting-over-ATM system architecture. These are broken down into issues for host and router implementations, and a number of service mismatches between IP multicasting and ATM. 9.1. Host Issues Given that there are far more hosts, and types of hosts, than there are routers or switches in the Internet, a major consideration of the system architecture must be what is required for host support for IP multicast over ATM. It is assumed that all IP multicast hosts with ATM interfaces: o Support IGMP per RFC-1112, or IGMPv2 (for non-ATM interfaces), o Support RSVPv1 per [4], and Milliken INTERNET-DRAFT Expires: January 12, 1996 [Page 20] IP Multicasting over ATM: System Architecture Issues o Support unicast IP over ATM per RFC-1577, including ATMARP. In addition, ATM IP multicast hosts must either implement the MARS protocol [5] or equivalent, or they must be configured to send multicast traffic to an IP multicast router (as described in [6]). As an optimization, hosts may wish to support direct point- to-multipoint ATM connections to: o Other hosts on the same subnet (as with MARS [5]), and o Hosts on other subnets ("cut-through" ATM connections, a la ROLC). It appears likely that the majority of multicast traffic will require quality of service support, so such mechanisms for setting up multicast ATM VCs must include a way of handling application QoS requirements. For interoperability with non- ATM subnets, this mechanism needs to be interoperable with RSVP. Hosts need to implement mechanisms to detect reflected traffic from ATM multicast servers. This can be done with a simple check of the IP source address. In some cases, the host may want a copy of its own traffic. Since its traffic may or may not be reflected back to it, depending on the implemenation of the local subnet, hosts should make provisions to copy outgoing multicast traffic for local delivery, if desired, and all traffic arriving from the ATM interface with a source address equal to the host's should be dropped. One system issue of major import to hosts is the problem of "VC implosion", where large numbers of point-to-multipoint VCs from multiple sources are required in a single session. Since each VC requires SAR resources, which may be in short supply in cost-sensitive host interface designs, this could eventually prove to be a major problem, especially when large multiway conferences or similar multicast applications involving large numbers (>10???) of hosts become common. ATM support for a multipoint-to-multipoint connection paradigm would partially solve the VC implosion problem, but the problem would remain for certain advanced multicast applications, such as distributed interactive simulations, which use multiple distribution subsets within a single session. Such applications need an M-to-N distribution capability. From the host's point of view, the simplest solution to the VC implosion problem seems to be the use of a multicast server. Milliken INTERNET-DRAFT Expires: January 12, 1996 [Page 21] IP Multicasting over ATM: System Architecture Issues Note that the VC implosion problem for hosts is primarily a side effect of multicast applications with multiple sources, and does not normally arise with unicast applications, or simple 1-to-N distribution applications. RSVP "Resv implosion" may occur in hosts with certain system architectures, especially "cut-through" end-to-end models, that do not employ special techniques to handle RSVP signalling. This would likely become most troublesome for 1- to-N applications like video distribution, where N can become very large. 9.2. Router Issues Issues of concern to router (and route server) designs include replication, address list management, QoS support, and techniques for "cut-through routing", as well as maintaining multicast routing information. Routers may also be forced to deal with large numbers of VCs, though it isn't clear if multicast services are significantly more demanding than unicast services in this regard. Multicast routing over generalized point-to-multipoint connections needs to be resolved, especially where routers and hosts might be intermixed on the same connection. The impact of the system architecture is most severe on the router. Both the end-to-end and subnet designs strongly influence the choice of router services and the appropriate design. For ATM networks that include multicast servers, the router must deal with its own multicast transmissions being reflected back to it, with no way of distinguishing the origin from the link layer. Since IP layer headers may not be sufficient to identify reflected packets, then either: o The link layer header needs to be augmented to help solve this problem, or o Use of multicast servers operating strictly at the ATM layer must be forbidden. (Routers themselves can safely serve as multicast servers, however.) For basic subnet models, without "cut-through" routing, relatively little additional support is needed in the router. Some technique for obtaining appropriate ATM addresses for receivers on local subnets is required, either an ARP-like mechanism like MARS [5], or the modified IGMP implementation proposed in [6]. Reference [6] discusses a design for QoS-based multicast IP over ATM, and its impact on router design. Milliken INTERNET-DRAFT Expires: January 12, 1996 [Page 22] IP Multicasting over ATM: System Architecture Issues 9.3. Service Mismatches There are a number of service mismatches between ATM and the IP protocol suite that need to be addressed in any system architecture for IP multicasting over ATM. These are discussed in detail in previous sections, they are merely summarized here: o IP uses receiver-initiated join; in ATM membership is initiated by the sender. o IP multicast addresses are logical, and receivers are anonymous; in ATM, receivers must be known to the sender, and are addressed by their normal unicast addresses. o In IP multicast, any IP host may send to any multicast group, at any time, without prior notice; in ATM, each multicast connection has a single sender, which must set up the connection. o In IP, group membership must be established and the receiver must be receiving packets from the sender(s) before QoS setup can be performed; in ATM, QoS parameters must be known by the sender before connection setup can be done. o IP QoS parameters are controlled by both the sender and the receiver, and are primarily set by the receiver; in ATM, only the sender sets QoS parameters for a multicast connection. o IP QoS parameters may change over the lifetime of the session; ATM QoS parameters are fixed at connection establishment, and cannot be subsequently changed. o IP supports different QoS parameters to different receivers of a multicast group; ATM uses identical QoS parameters to all receivers on a given multicast connection. Acknowledgements This work was sponsored by the Advanced Research Projects Agency under contract No. MDA 903-92-C-0081. The views expressed in this document are not necessarily those of the Advanced Research Projects Agency. Milliken INTERNET-DRAFT Expires: January 12, 1996 [Page 23] IP Multicasting over ATM: System Architecture Issues Author's Address Walter Milliken Bolt, Beranek and Newman, Inc. 10 Moulton St. Cambridge, MA 02138 Phone: (617) 873-3382 Fax: (617) 873-6091 Email: milliken@bbn.com References [1] R. G. Cole and D. Shur. IP over ATM: A Framework Document. Internet Draft (Work in Progress) draft-ietf- atm-framework-doc-01.txt, Internet Engineering Task Force, February 1995. [2] S. Deering. Host Extensions for IP Multicasting. Request for Comments (Recommended Standard) RFC 1112, Internet Engineering Task Force, January 1994. [3] ATM Forum, "ATM User-Network Interface Specification", Version 3.0, PTR Prentice Hall, ISBN 0-13-225863-3, September 1993. [4] R. Braden, L. Zhang, D. Estrin, S. Herzog, and S. Jamin. Resource ReSerVation Protocol (RSVP) -- Version 1 Functional Specification. Internet Draft (Work in Progress) draft-ietf-rsvp-spec-05.txt, Internet Engineering Task Force, March 1995. [6] W. Milliken. Integrated Services IP Multicast over ATM. Internet Draft (Work in Progress) draft-milliken-ipatm- services-00.txt, July 1995. [5] G. Armitage. Support for Multicast over UNI 3.1 based ATM Networks. Internet Draft (Work in Progress) draft- ietf-ipatm-ipmc-04.txt, Internet Engineering Task Force, February 1995. [7] M. Laubach. Classical IP and ARP over ATM. Request for Comments (Proposed Standard) RFC 1577, Internet Engineering Task Force, January 1994. [8] S. Deering, D. Estrin, D. Farinacci, V. Jacobson, C. Liu, and L. Wei. Protocol Independent Multicast (PIM): Motivation and Architecture. Internet Draft (Work in Progress) draft-ietf-idmr-pim-arch-01.txt, Internet Engineering Task Force, January 1995. [9] A. J. Ballardie. Core Based Trees (CBT) Multicast -- Architectural Overview and Specification. Internet Draft (Work in Progress) draft-ietf-idmr-cbt-spec- 01.txt, Internet Engineering Task Force, April 1995. Milliken INTERNET-DRAFT Expires: January 12, 1996 [Page 24]