Network Working Group David Melman Internet Draft Tal Mizrahi Intended status: Informational Marvell Expires: January 2013 Donald Eastlake Huawei July 10, 2012 FCoE over TRILL draft-mme-trill-fcoe-02.txt Abstract Fibre Channel over Ethernet (FCoE) and TRILL are two emerging standards in the data center environment. While these two protocols are seemingly unrelated, they have a very similar behavior in the forwarding plane, as both perform hop-by-hop forwarding over Ethernet, modifying the packet's MAC addresses at each hop. This document describes an architecture for the integrated deployment of these two protocols. Status of this Memo This Internet-Draft is submitted to IETF in full conformance with the provisions of BCP 78 and BCP 79. Internet-Drafts are working documents of the Internet Engineering Task Force (IETF), its areas, and its working groups. Note that other groups may also distribute working documents as Internet- Drafts. Internet-Drafts are draft documents valid for a maximum of six months and may be updated, replaced, or obsoleted by other documents at any time. It is inappropriate to use Internet-Drafts as reference material or to cite them other than as "work in progress." The list of current Internet-Drafts can be accessed at http://www.ietf.org/ietf/1id-abstracts.txt. The list of Internet-Draft Shadow Directories can be accessed at http://www.ietf.org/shadow.html. This Internet-Draft will expire on January 10, 2013. Copyright Notice Copyright (c) 2012 IETF Trust and the persons identified as the document authors. All rights reserved. Melman, et al. Expires January 10, 2013 [Page 1] Internet-Draft FCoE over TRILL July 2012 This document is subject to BCP 78 and the IETF Trust's Legal Provisions Relating to IETF Documents (http://trustee.ietf.org/license-info) in effect on the date of publication of this document. Please review these documents carefully, as they describe your rights and restrictions with respect to this document. Code Components extracted from this document must include Simplified BSD License text as described in Section 4.e of the Trust Legal Provisions and are provided without warranty as described in the Simplified BSD License. Table of Contents 1. Introduction ................................................. 2 2. Abbreviations ................................................ 3 3. FCoE over TRILL .............................................. 4 3.1. FCoE over a TRILL Cloud ................................. 4 3.2. FCoE over RBridge ....................................... 5 3.2.1. FCRB ............................................... 5 3.2.2. Topology ........................................... 7 3.2.3. The FCRB Flow ...................................... 9 3.2.3.1. Example - ENode to ENode ...................... 9 3.2.3.2. Example - ENode to Native FC Node ............ 10 3.2.3.3. Example - ENode to ENode with non-FCRB EoR ... 10 4. Security Considerations ..................................... 12 5. IANA Considerations ......................................... 12 6. Acknowledgments ............................................. 12 7. References .................................................. 12 7.1. Normative References ................................... 12 7.2. Informative References ................................. 12 1. Introduction Data center networks are rapidly evolving towards a consolidated approach, where Ethernet is used as the common infrastructure for all types of traffic. Storage traffic was traditionally dominated by the Fibre Channel (FC) protocol suite. At the intersection between these two technologies a new technology was born, Fibre Channel over Ethernet (FCoE), where native Fibre Channel (FC) packets are encapsulated with an FCoE encapsulation over an Ethernet header. Traffic between two FCoE end nodes (ENodes) is forwarded through one or more FCoE Forwarders (FCF). An FCF takes a forwarding decision based on the Fibre Channel destination ID (D_ID), and enforces security policies between ENodes, also known as zoning. Once an FCF takes a forwarding decision, it modifies the source and destination MAC addresses of the packet, to reflect the path to the next hop FCF Melman, et al. Expires January 10, 2013 [Page 2] Internet-Draft FCoE over TRILL July 2012 or ENode. FCFs use a routing protocol called Fabric Shortest Path First (FSPF) to find the optimal path to each destination. An FCF typically has one or more native Fibre Channel interfaces, allowing it to communicate with native Fibre Channel devices, e.g., storage arrays. TRILL [RFCTRILL] is a protocol for transparent least cost routing, where RBridges forward traffic to their detination based on a least cost route, using a TRILL encapsulation header. RBridges forward TRILL-encapsulated packets based on the Egress RBridge Nickname in the TRILL header. An RBridge forwards a TRILL-encapsulated packet after modifying its MAC addresses to reflect the path to the next-hop RBridge, and decrementing a Hop Count field. TRILL and FCoE bear a strong resemblance in their forwarding planes. Both protocols take a forwarding decision based on protocol addresses above Layer 2, and modify the Ethernet MAC addresses on a per-hop basis. Each of the protocols uses its own routing protocol rather than using any type of bridging protocol such as spanning tree protocol [802.1Q] or the Shortest Path Bridging protocol [802.1aq]. FCoE and TRILL are both targeted at the data center environment, and their concurrent deployment is self-evident. This document describes an architecture for the integrated deployment of these two protocols. 2. Abbreviations DCB Data Center Bridging ENode FCoE Node such as server or storage array EoR End of Row FC Fibre Channel FCF Fibre Channel Forwarder FCoE Fibre Channel over Ethernet FCRB Fibre Channel forwarder over RBridge FSPF Fabric Shortest Path First LAN Local Area Network RBridge Routing Bridge Melman, et al. Expires January 10, 2013 [Page 3] Internet-Draft FCoE over TRILL July 2012 SAN Storage Area Network ToR Top of Rack TRILL Transparent Interconnection of Lots of Links WAN Wide Area Network 3. FCoE over TRILL 3.1. FCoE over a TRILL Cloud The simplest approach for running FCoE traffic over a TRILL network is presented in Figure 1. The figure illustrates a TRILL-enabled network, where FCoE traffic is transparently forwarded over the TRILL cloud. The figure illustrates two ENodes, a Server and an FCoE Storage Array, an FCF, and a native Fibre Channel SAN connected to the FCF. FCoE traffic between the two ENodes is sent from the first ENode over the TRILL cloud to the FCF, and then back through the TRILL cloud to the second ENode. +---+ | |_________ | | \ ___ _ +---+ \/ \_/ \__ _ __ FCoE Storage _/ \ / \_/ \_ Array / TRILL / +---+ \_ \ (ENode A) \_ Cloud /________| |____/ SAN _/ / \ | | \__ _/ \__/\_ ___/ +---+ \_/ +---+ / \_/ FCF | |________/ | | +---+ Server (ENode B) Figure 1 The "Separate Cloud" Approach The configuration in Figure 1 separates the TRILL cloud(s) and the FCoE cloud(s). The TRILL cloud forwards FCoE traffic as standard Ethernet traffic, and appears to the ENodes and FCF as an Ethernet LAN. (To eliminate frame loss due to queue overflow, the switches in Melman, et al. Expires January 10, 2013 [Page 4] Internet-Draft FCoE over TRILL July 2012 any TRILL Cloud used with FCoE would likely implement and use the relevant DCB protocols [TRILDCB].) The main drawback of the Separate Cloud approach is that RBridges and FCFs are separate nodes in the network, resulting in more cabeling and boxes, and communication between Enodes usually requires two TRILL cloud traversals with twice as many hops. As mentioned above, data center networking is converging towards a consolidated and cost effective approach, where the same infrastructure and equipment is used for both data and storage traffic, and where high efficiency and minimal number of hops are important factors when designing the network topology. Clearly the Separate Cloud approach is not practical, and is not commonly deployed. It is presented as a background and motivation for the approach discussed in the next section. 3.2. FCoE over RBridge 3.2.1. FCRB Rather than the Separate Cloud approach discussed in the previous subsection, an alternate approach is presented, where each switch incorporates both an FCF entity and an RBridge entity. This consolidated entity is referred to as FCoE-forwarder-over-RBridge (FCRB). Figure 2 illustrates an FCRB, and its main building blocks. An FCRB can be functionally viewed as two independent entities: o An FCoE Forwarder (FCF) entity. o An RBridge entity. The FCF entity is connected to one of the ports of the RBridge, and appears to the RBridge as a native Ethernet host. A detailed description of the interaction between the layers is presented in Section 3.2.3. Melman, et al. Expires January 10, 2013 [Page 5] Internet-Draft FCoE over TRILL July 2012 +--------------------+ |FCRB | | +-----------+ | | | FCF |-+ | | +-----+-----+ | | | | | | | +-----+-----+ | | | | RBridge | | | | +-+-+-+-+-+-+ | | | | | | | | | | | +---|-|-|-|-|-|---|--+ _ __ FCoE/ / | | | | | \ Native / \_/ \_ +---+ Ethernet / / | | | | \ FC \_ \ | |_________________/ / | | | | \______________/ SAN _/ | | / | | | | \__ _/ +---+ / | | | | \_/ FCoE Storage / | | | | FCoE / Ethernet Array / |_| | | over TRILL (ENode A) / / \_/ \__ / _/ \ +---+ / / TRILL / | |____________/ \_ Cloud / | | / \ +---+ \__/\_ ___/ Server \_/ (ENode B) Figure 2 FCRB Entity in the Network The FCRB entity maintains layer independence between the TRILL and FCoE protocols, while enabling both protocols on the same network. It is noted that FCoE traffic is always forwarded through an FCF, and cannot be forwarded directly between two ENodes. Thus, FCoE traffic between ENodes A and B in the topology in Figure 1 is forwarded through the path ENode A-->TRILL cloud-->FCF-->TRILL cloud-->ENode B Traffic between A and B in the topology in Figure 2 is forwarded through the path ENode A-->FCRB-->ENode B Melman, et al. Expires January 10, 2013 [Page 6] Internet-Draft FCoE over TRILL July 2012 Hence, the usage of FCRB entities allows TRILL and FCoE to use common infrastructure and equipment, as opposed to the Separate Cloud topology presented in Figure 1. 3.2.2. Topology The network configuration illustrated in Figure 3 shows a typical topology of a data center network. Servers are hierarchically connected through Top-of-Rack (ToR) switches, also known as access switches, and each set of racks is aggregated through an End-of-Row (EoR) switch. The EoR switches are aggregated to the Core switches, which may be connected to other clouds, such as an external WAN or a native FC SAN. Melman, et al. Expires January 10, 2013 [Page 7] Internet-Draft FCoE over TRILL July 2012 _ __ _ __ / \_/ \_ / \_/ \_ \_ \ \_ \ .... / SAN _/ / WAN _/ \__ _/ \__ _/ \_/ \_/ | | | | | | +------+ +------+ Core | | | | FCoE over | | | | RBridge | | | | (FCRB) +------+ +------+ | \___ ___/ | | \ / | | \/ | EoR +----+_______/\_______+----+ FCoE over | | | | RBridge | | | | (FCRB) +----+ +----+ / \ / \ / \ / \ ToR +---+ +---+ +---+ +---+ FCoE over | | | | | | | | RBridge | | | | | | | | (FCRB) +---+ +---+ +---+ +---+ / \ / \ / \ / \ / \ / \ / \ / \ +-+ +-+ +-+ +-+ +-+ +-+ +-+ +-+ Servers/ | | | | | | | | | | | | | | | | ENodes +-+ +-+ +-+ +-+ +-+ +-+ +-+ +-+ A B C D E F G H Figure 3 FCoE over RBridge Topology Note that in the example in Figure 3 all the ToR, EoR and core switches are FCRB entities, but it is also possible for some of the network nodes to be pure RBridges, creating a topology where FCRBs are interconnected through TRILL clouds. Melman, et al. Expires January 10, 2013 [Page 8] Internet-Draft FCoE over TRILL July 2012 3.2.3. The FCRB Flow 3.2.3.1. Example - ENode to ENode FCoE traffic sent between two ENodes, A and B, is transmitted through the ToR FCRB, since A and B are connected to the same ToR. Traffic between A and C must be forwarded through the EoR FCRB. +--------+ +--------+ +--------+ +--------+ +--------+ | FCoE |.....| FCF |.....| FCF |.....| FCF |.....| FCoE | | ENode | +--------+ +--------+ +--------+ | ENode | | | |RBridge |.....|RBridge |.....|RBridge | | | +--------+ +--------+ +--------+ +--------+ +--------+ |Ethernet|<===>|Ethernet|<===>|Ethernet|<===>|Ethernet|<===>|Ethernet| +--------+ +--------+ +--------+ +--------+ +--------+ Server ToR EoR ToR FCoE Storage ENode A FCRB FCRB FCRB Array ENode C Figure 4 Traffic between two ENodes - Example Figure 4 illustrates the traffic between ENodes A and C that are not connected to the same ToR. o FCoE traffic from A is sent to the ToR over the Ethernet interface. The destination MAC address is the address of the FCF entity at the ToR. o ToR: o The packet is forwarded to the FCF entity at the ToR. Thus, forwarding between A and the FCF at the ToR is analogous to forwarding between two Ethernet hosts. o The FCF entity at the ToR takes a forwarding decision based on the FC addresses. This decision is based on the FSPF routing protocol at the FCF layer. The next hop at the FCF layer is the EoR FCF entity. o The FCF then updates the destination MAC address of the packet to the address of the EoR FCF. o The packet is forwarded to the RBridge entity, where it is encapsulated in a TRILL header, and sent to the RBridge at the EoR over a single hop of the TRILL network. Melman, et al. Expires January 10, 2013 [Page 9] Internet-Draft FCoE over TRILL July 2012 o The RBridge entity in the EoR FCRB, acting as the egress RBridge, decapsulates the TRILL header and forwards the FCoE packet to the FCF entity. From this point the forwarding process is similar to the one described above for the ToR. o A similar forwarding process takes place at the next hop ToR FCRB, where the FCRB finally forwards the FCoE packet to the target ENode. 3.2.3.2. Example - ENode to Native FC Node +--------+ +--------+ +--------+ +---------+ +--------+ | FCoE |.....| FCF |.....| FCF |.....| FCF |.....| FC | | ENode | +--------+ +--------+ +----+----+ |protocol| | | |RBridge |.....|RBridge |.....| RB | | | stack | +--------+ +--------+ +--------+ +----+ FC | | | |Ethernet|<===>|Ethernet|<===>|Ethernet|<===>|Eth | |<===>| | +--------+ +--------+ +--------+ +----+----+ +--------+ Server ToR EoR Core Native FC ENode FCRB FCRB FCRB Storage Array Figure 5 Example Traffic between ENode & Native FC Storage Array Figure 5 illustrates a second example, where traffic is sent between an ENode and an FC Storage Array, following the network topology in Figure 3. o FCoE traffic from the ENode is sent to the ToR over the Ethernet interface. The forwarding process through the ToR FCRB and through the EoR is similar to the corresponding steps in Section 3.2.3.1. o When the packet reaches the core FCRB, the egress RBridge entity decapsulates the TRILL header and forwards the FCoE packet to the FCF entity. The packet is then forwarded as a native FC packet through the FC interface to the native FC node. 3.2.3.3. Example - ENode to ENode with non-FCRB EoR The example illustrated in Figure 6 is similar to the one shown in Figure 4, except that the EoR is an RBridge rather than an FCRB. Melman, et al. Expires January 10, 2013 [Page 10] Internet-Draft FCoE over TRILL July 2012 +--------+ +--------+ +--------+ +--------+ | FCoE |.....| FCF |....................| FCF |.....| FCoE | | ENode | +--------+ +--------+ +--------+ | ENode | | | |RBridge |.....|RBridge |.....|RBridge | | | +--------+ +--------+ +--------+ +--------+ +--------+ |Ethernet|<===>|Ethernet|<===>|Ethernet|<===>|Ethernet|<===>|Ethernet| +--------+ +--------+ +--------+ +--------+ +--------+ Server ToR 1 EoR ToR 2 FCoE Storage ENode A FCRB FCRB FCRB Array ENode C Figure 6 Traffic between two ENodes - Example An FCoE packet sent from A to C is forwarded as follows: o The packet is sent to the FCF in ToR 1, as in the previous example. o The FCF in ToR 1 takes a forwarding decision based on the FC addresses, and forwards the packet to the next hop FCF, which resides in ToR 2. This forwarding decision is taken at the FCF layer, and is based on the FSPF routing protocol. o The packet is then forwarded to the RBridge entity in ToR 1, where it is encapsulated in a TRILL encapsulation, and forwarded to the RBridge at ToR 2. The packet is forwarded over the TRILL cloud through the RBridge at the EoR. The path through the TRILL cloud is determined by TRILL's IS-IS routing protocol. o Once the packet reaches ToR 2, it is forwarded in a similar manner to the description in Section 3.2.3.1. This example demonstrates that it is possible to have a hybrid network, where some of the nodes are FCRBs, and some of the nodes are RBridges. A network configuration where some of the nodes are FCFs and others are not FCoE aware is sometimes referred to in the FCoE jargon as sparse mode. In dense mode, on the other hand, all nodes in the network are FCFs. Thus, while the example in Figure 4 illustrates a dense mode topology where all bridges are FCRBs, the example in Figure 6 shows a sparse mode, where traffic between FCRBs can be forwarded through a TRILL cloud with several RBridge hops. Melman, et al. Expires January 10, 2013 [Page 11] Internet-Draft FCoE over TRILL July 2012 4. Security Considerations For general TRILL Security Considerations see [RFCTRILL]. For general FCoE Security Consideration see Annex D of [FC-BB-5]. There are no additional security implications imposed by this document. 5. IANA Considerations There are no IANA actions required by this document. RFC Editor: please delete this section before publication. 6. Acknowledgments This document was prepared using 2-Word-v2.0.template.dot. 7. References 7.1. Normative References [RFCTRILL] Perlman, R., Eastlake, D., Dutt, D., Gai, S., Ghanwani, A., "Routing Bridges (RBridges): Base Protocol Specification", RFC6325, July 2011. 7.2. Informative References [FC-BB-5] ANSI INCITS 462: Information Technology - Fibre Channel - Backbone - 5 (FC-BB-5). [802.1Q] "IEEE Standard for Local and metropolitan area networks - Virtual Bridged Local Area Networks", IEEE Std 802.1Q-2011, May 2011. [802.1aq] "IEEE Standard for Local and metropolitan area networks - Shortest Path Bridging", work in progress, June 2011. [TRILLDCB] Eastlake, D., Wadekar, M., Ghanwani, A., Agarwal, P., Mizrahi, T., "RBridges: Support of IEEE 802.1Qbb, 802.1Qaz, and 802.1Qau", draft-eastlake-trill-rbridge- dcb, work in progress, 2012. Melman, et al. Expires January 10, 2013 [Page 12] Internet-Draft FCoE over TRILL July 2012 Authors' Addresses David Melman Marvell 6 Hamada St. Yokneam, 20692 Israel Email: davidme@marvell.com Tal Mizrahi Marvell 6 Hamada St. Yokneam, 20692 Israel Email: talmi@marvell.com Donald Eastlake 3rd Huawei USA R&D 155 Beaver Street Milford, MA 01757 USA Phone: +1-508-333-2270 EMail: d3e3e3@gmail.com Melman, et al. Expires January 10, 2013 [Page 13]