Internet DRAFT - draft-zeng-nvo3-l3-vm-migration
draft-zeng-nvo3-l3-vm-migration
Network Virtualization Overlays L. Zeng
Internet Draft Tsinghua University
Intended status: Informational June 12, 2014
Expires: December 2014
Layer 3 Cross-site VM Live Migration Scheme
draft-zeng-nvo3-l3-vm-migration-00.txt
Status of this Memo
This Internet-Draft is submitted in full conformance with the
provisions of BCP 78 and BCP 79.
Internet-Drafts are working documents of the Internet Engineering
Task Force (IETF), its areas, and its working groups. Note that
other groups may also distribute working documents as Internet-Drafts.
Internet-Drafts are draft documents valid for a maximum of six months
and may be updated, replaced, or obsoleted by other documents at any
time. It is inappropriate to use Internet-Drafts as reference
material or to cite them other than as "work in progress."
The list of current Internet-Drafts can be accessed at
http://www.ietf.org/ietf/1id-abstracts.txt
The list of Internet-Draft Shadow Directories can be accessed at
http://www.ietf.org/shadow.html
This Internet-Draft will expire on December 12, 2014.
Copyright Notice
Copyright (c) 2014 IETF Trust and the persons identified as the
document authors. All rights reserved.
This document is subject to BCP 78 and the IETF Trust's Legal
Provisions Relating to IETF Documents
(http://trustee.ietf.org/license-info) in effect on the date of
publication of this document. Please review these documents carefully,
as they describe your rights and restrictions with respect to this
document. Code Components extracted from this document must include
Simplified BSD License text as described in Section 4.e of the Trust
Legal Provisions and are provided without warranty as described in
the Simplified BSD License.
Zeng Expires December 12, 2014 [Page 1]
Internet-Draft Layer 3 Cross-site VM Live Migration Scheme June 2014
Abstract
This document utilizes software-defined network (SDN) to propose a
cross-site live VM migration scheme in layer 3 environment. The
architecture and main components are described in detail. The
components include central controller, local controller, gateway and
virtual switch, which are in charge of cross-site communication
management, in-site network configuration, inter-site connection and
VM address resolution respectively. This scheme can implement layer 3
cross-site VM migration without using complex mechanism.
Table of Contents
1. Introduction ................................................ 2
2. Conventions used in this document............................ 3
2.1. Requirements Language................................... 3
2.2. Definitions ............................................ 3
3. Scheme Architecture ......................................... 4
4. Component Detail ............................................ 5
4.1. Central SDN Controller.................................. 5
4.2. Local SDN Controller.................................... 6
4.3. Gateway ................................................ 7
4.4. Virtual Switch ......................................... 7
5. Migration Process ........................................... 7
6. Security Considerations..................................... 11
7. IANA Considerations ........................................ 11
8. Conclusions ................................................ 11
9. References ................................................. 11
9.1. Normative References................................... 11
10. Acknowledgments ........................................... 11
1. Introduction
The support of Virtual Machine (VM) mobility is an important feature
of nvo3. This document describes a VM live migration scheme which
addresses several network related issues of this problem. The scheme
mainly concerns VM live migration between different data centers,
especially the ones with different subnets. As for live migration of
VM, it is defined as the mobility of VM while retaining the VM's
addresses and communication status.
The scheme in this document utilizes SDN to control and configure
data center network. SDN's global network view and flexible
forwarding control make it possible to quickly change routing and
provide peer to peer forwarding which enables cross-subnet connection.
These features help simplify the scheme.
Zeng Expires December 12, 2014 [Page 2]
Internet-Draft Layer 3 Cross-site VM Live Migration Scheme June 2014
2. Conventions used in this document
2.1. Requirements Language
The key words "MUST", "MUST NOT", "REQUIRED", "SHALL", "SHALL NOT",
"SHOULD", "SHOULD NOT", "RECOMMENDED", "MAY", and "OPTIONAL" in this
document are to be interpreted as described in RFC-2119 [RFC2119].
In this document, these words will appear with that interpretation
only when in ALL CAPS. Lower case uses of these words are not to be
interpreted as carrying RFC-2119 significance.
2.2. Definitions
live migration: VM can keep communication while migration.
flow: forwarding rules configured in switches.
host: physical machine which hosts VMs and virtual switch.
Zeng Expires December 12, 2014 [Page 3]
Internet-Draft Layer 3 Cross-site VM Live Migration Scheme June 2014
3. Scheme Architecture
+--------------------------------------------------------------+
| +----------------+ |
| | Central | |
| | Controller | |
| +-----*----*-----+ |
| | | |
| +----------+_ +-----------+ |
| | | |
| | +------*-------+ |
| Control | | Data Center | |
| API | +----------**--+ |
| | || Internet |
| | || Connection |
| +----------|-------------------------------||----------+ |
| | | || | |
| | +------*------+ Data Center +----**-----+ | |
| | | Local | | Gateway | | |
| | | Controller | +--*----*---+ | |
| | +---*------*--+ | | | |
| | | | | | | |
| | | +-------------------------+ | | |
| | | Control | | |
| | | API +--------+ | | |
| | +--------------* SDN *-------------+ | |
| | | | Switch | Data | |
| | | +--*--*--+ Path | |
| | | +--------+ | | +-------+ | |
| | +--* SDN *-----+ +--------* | | |
| | | | Switch | | SDN | | |
| | | +---*----+ | Switch| | |
| | +------|----------------------* | | |
| | +--|------|--------------+ +---*---+ | |
| | | | | | | | |
| | | | +-*---------+ | +--------+ | |
| | | +----* Virtual | | | Host | | |
| | | | Switch | | +--------+ | |
| | | +--*------*-+ | | |
| | | Host | | | | |
| | | | | | | |
| | | +---*--+ +-*----+ | | |
| | | | VM | | VM | | | |
| | | +------+ +------+ | | |
| | +------------------------+ | |
| +------------------------------------------------------+ |
+--------------------------------------------------------------+
Zeng Expires December 12, 2014 [Page 4]
Internet-Draft Layer 3 Cross-site VM Live Migration Scheme June 2014
Figure 1 VM Live Migration Scheme Architecture
The scheme architecture is shown in figure 1. VMs are interconnected
by intra data center network, which is controlled by local SDN
controller. All SDN controllers in different data center are
connected with central SDN controller, which is in charge of cross-
site VM migration. Cross-site migration process is invoked by central
SDN controller. Central SDN controller stores VMs' location and
communication information, using this information, it can calculate
which data centers are involved in the migration process and the job
types each data center needs to handle, e.g. setting up migration
path, changing routing. After calculation, central controller notices
the involved local controllers to configure each intra data center
network to complete the migration process.
4. Component Detail
4.1. Central SDN Controller
Central SDN controller is in charge of cross-site communication. It
needs to store information like VMs' location and communication pairs,
and to have the ability to calculate and dispatch the configuration
jobs.
Central SDN controller do not need to configure the network devices
by itself, this work is done by local controllers. The central
controller only needs to notice the involved local controller its job
type and configuration parameters.
Job types include setting up migration path and changing routing.
Setting up migration path refers to connect the source host and the
destination host before copying the VM's disk and memory, it involves
intra and inter data center connection configuration. Changing
routing means to calculate and configure the changed routing path
caused by VM migration. This work can be conducted during the
migration process to minimize the service down time.
Setting up migration path needs parameters like source host, source
data center, VM IP and MAC addresses, destination host and
destination data center. Central controller divides the connection
configuration into three parts, source host to source data center
gateway, source data center gateway to destination data center
gateway and destination data center gateway to destination host and
respectively notices the source and destination data centers' local
controllers to configure the switches and gateways.
Zeng Expires December 12, 2014 [Page 5]
Internet-Draft Layer 3 Cross-site VM Live Migration Scheme June 2014
Changing routing needs parameters like VM IP and MAC addresses and
virtual network (VN) ID. Central controller needs to search its
database to find VMs of the same VN belongs to which data centers and
then notices the data centers' local controller to calculate and
modify the flows related to the migrating VM.
4.2. Local SDN Controller
Local SDN controller is in charge of the network configuration issues.
Each local controller controls the physical network (include physical
switches, virtual switches, gateway, etc.) of the data center it
belongs to. Local controller should have at least two functions:
network configuration and communication with central controller.
Network configuration function mainly manages the forwarding logic of
the data center, more specifically, it manages network virtualization,
communication within VNs and communication between different data
centers.
Network virtualization can be realized by setting VLAN, classifying
MAC address, etc. The way to virtualize the network is not specified
in this document, different data centers can even implement network
virtualization in different ways as long as VMs in different data
centers, but of the same VN can communicate with each other.
Communication within VNs function can be provided by classical
forwarding methods, like MAC-learning, OSPF, etc. These methods
should be implemented as controller modules which can automatically
set up the route. Cross-site forwarding is like intra-site forwarding
in most aspects, if the local controller identifies a certain
communication as cross-site communication, it should forward the
packets to gateway and notices the central controller to configure
the relative data center's local controller to set up the remaining
path. After receiving the notice from central controller, the other
local controller needs to forward the packets from gateway to its
destination VM.
Communication between different data centers is implemented by
inserting NAT flows to the gateways of the involved data centers.
Gateways should have one or more public IPs, if a flow is identified
as a cross-site flow, central controller assigns a source public IP,
destination public IP and TCP port tuple as the flow's ID. After that,
the central controller notices the local controllers to insert NAT
flows to the gateways. At the source gateway, the NAT flow replaces
the packets' source IP, MAC addresses with the gateway's addresses,
replaces the TCP port with the assigned port number, and replaces the
packets' destination IP, MAC addresses with the destination gateway's
addresses. At the destination gateway, the NAT flow does the reverse
Zeng Expires December 12, 2014 [Page 6]
Internet-Draft Layer 3 Cross-site VM Live Migration Scheme June 2014
process and recovers the packets' headers. If VM's status is changed
or the flow is no longer needed, central controller should notice the
local controllers to delete the flows and release the TCP port.
4.3. Gateway
Gateway's main function is to support inter data center communication
and user to VM's communication. Inter data center communication's
process is explained in 4.2. User to VM's communication's process is
almost the same but only involves one gateway.
One data center can deploy one or more gateways to balance the load,
or even one gateway per VN per data center to simplify the VN
implementation. To support NAT flows, gateway should support packet
header modification action of at least MAC, IP, and TCP fields.
4.4. Virtual Switch
Virtual switches are usually deployed at the host server in which the
VMs nest. As the last hop of the whole network, virtual switches
provide not only the basic forwarding function but also the address
resolution function. After migration, VM is usually in a different
subnet, which can impede the normal address resolution process. To
avoid this, virtual switches need to identify all ARP packets and
upload them to the local controller to process them. With the ARP
packets, local controller can recognize the destination VM and set up
the forwarding path accordingly. After that, it can reply the source
VM with the destination VM's MAC address and the source VM can start
communication.
5. Migration Process
This section describes the timeline of the migration process. The
whole migration process is divided into three parts: pre-migration,
migration and post-migration.
Zeng Expires December 12, 2014 [Page 7]
Internet-Draft Layer 3 Cross-site VM Live Migration Scheme June 2014
+--------------------------------------------------------------+
| Central Controller(CC) Source Local Destination |
| | Controller (SLC) Local Controller|
| +----*----+ | (DLC) |
| | receive | | | |
| |migration| | | |
| | request | | | |
| +----*----+ | | |
| | | | |
| +----*----+ | | |
| | query | | | |
| | database| | | |
| +----*----+ | | |
| | deploy jobs | | |
| *------------------* | |
| | | | |
| *------------------|----------------* |
| Pre-migration | | | |
| Phase | +-----*-----+ +-----*------+|
| | | calculate | | calculate ||
| | | route | | route ||
| | | from host | |from gateway||
| | | to gateway| | to host ||
| | +-----*-----+ +-----*------+|
| | | | |
| | +-----*-----+ +-----*------+|
| | | configure | | configure ||
| | | switches | | switches ||
| | +-----*-----+ +-----*------+|
| +----*----+ | | |
| | Allocate| | | |
| | TCP port| | | |
| | for NAT | | | |
| +----*----+ | | |
| | deploy NAT jobs | | |
| *------------------* | |
| | | | |
| *------------------|----------------* |
| | | | |
| | +----*----+ +----*----+ |
| | |configure| |configure| |
| | | gateway | | gateway | |
| +---------+ +---------+ |
+--------------------------------------------------------------+
Figure 2 Pre-migration Phase
Zeng Expires December 12, 2014 [Page 8]
Internet-Draft Layer 3 Cross-site VM Live Migration Scheme June 2014
o Pre-migration Phase
Pre-migration phase has one main object, which is to set up the
routing path for image copy. This object is fulfilled by three steps:
First, central SDN controller receives the migration request, which
contains the VM's IP address and the destination host name, then the
central controller queries the VM's MAC address, source and
destination data centers' local controllers' addresses. After the
query, the central controller deploys the forwarding job to the
involved local controllers. Second, the local controllers set up the
path from host to gateway. And last, the local controllers configure
the gateway to connect the two data centers. The NAT parameters are
provided by the central controller within the job request.
+--------------------------------------------------------------+
| CC SLC DLC |
| | start | | |
| | VM migration | | |
| *------------------* | |
| | | | |
| | +-----*-----+ | |
| | | start | | |
| +------*------+ |disk/memory| | |
| | calculate | | copy | | |
| |route changes| +-----*-----+ | |
| +------*------+ | data copy | |
| | *----------------* |
| | deploy jobs | | |
| *------------------* | |
| | | | |
| *------------------|----------------* |
| Migration | | | |
| Phase | +-----*-----+ +-----*-----+ |
| | | configure | | configure | |
| | | switches | | switches | |
| | | to update | | to update | |
| | | flows | | flows | |
| | +-----------+ +-----------+_|
+--------------------------------------------------------------+
Figure 3 Migration Phase
o Migration Phase
In migration phase, hypervisors conduct the VM migration process.
During this time, the SDN controllers can calculate and configure the
route changes caused by migration to minimize the VM's service down
Zeng Expires December 12, 2014 [Page 9]
Internet-Draft Layer 3 Cross-site VM Live Migration Scheme June 2014
time. In order not to disturb the ongoing communication, in this
phase, only the flows which do not impact the service are added or
updated. Other flows are updated after migration is completed.
+--------------------------------------------------------------+
| CC SLC DLC |
| | | | |
| | | VM migration | |
| | | complete | |
| | *----------------* |
| Post-migration | | | |
| Phase | +-----*-----+ +-----*------+|
| | | configure | | configure ||
| | | switches | | switches ||
| | | to update | | to update ||
| | | flows | | flows ||
| | +-----*-----+ +-----*------+|
| |migration complete| | |
| | notification | | |
| *------------------*----------------* |
| | | | |
| +---*----+ +-----*-----+ +----*----+ |
| | update | | unplug | | boot up | |
| |database| | virtual | | VM | |
| +---*----+ | interface | +----*----+ |
| | +-----*-----+ | |
| | | | |
| | +-----*-----+ +-----*-----+ |
| | | delete | | delete | |
| | | NAT and | | NAT and | |
| | | disk copy | | disk copy | |
| | | flows | | flows | |
| | +-----*-----+ +-----*-----+ |
| | clean complete | | |
| *------------------* | |
| | notification | | |
| *------------------|----------------* |
| | | | |
| +-----*-----+ | | |
| | release | | | |
| | TCP port | | | |
| +-----------+ | | |
+--------------------------------------------------------------+
Figure 4 Post-migration Phase
o Post-migration Phase
Zeng Expires December 12, 2014 [Page 10]
Internet-Draft Layer 3 Cross-site VM Live Migration Scheme June 2014
This phase mainly does some clean work, including deleting disk copy
flows, release NAT resources, etc. Besides these, some route changes
need to be configured after the VM migration to avoid unnecessary
service down (like ingress and egress flows).
6. Security Considerations
This requirements document does not raise in itself any specific
security issues.
7. IANA Considerations
IANA does not need to take any action for this draft.
8. Conclusions
This document proposes a scheme for layer 3 cross-site VM live
migration and describes its architecture, main components and process
phases.
9. References
9.1. Normative References
[RFC2119] Bradner, S., "Key words for use in RFCs to Indicate
Requirement Levels", BCP 14, RFC 2119, March 1997.
[I-D.ietf-nvo3-vm-mobility-issues] Rekhter, Y., Henderickx, W.,
Shekhar, R., Fang, L., Dunbar, L., and A. Sajassi,
"Network-related VM Mobility Issues", draft-ietf-nvo3-vm-
mobility-issues-02 (work in progress), June 2014.
10. Acknowledgments
This work is supported by Chinese National Major Scientific and
Technological Specialized Project (No.~2013ZX03002001), National
Basic Research Program of China (973 Program Grant No.~2013CB329105),
China's Next Generation Internet (No.~CNGI-12-03-007), and ZTE
Corporation.
This document was prepared using 2-Word-v2.0.template.dot.
Zeng Expires December 12, 2014 [Page 11]
Internet-Draft Layer 3 Cross-site VM Live Migration Scheme June 2014
Authors' Addresses
Lieguang Zeng
Department of Electronic Engineering, Tsinghua University
Department of Electronic Engineering, Tsinghua University, Beijing,
China
Email: zenglg@mail.tsinghua.edu.cn
Xu Yang
Department of Electronic Engineering, Tsinghua University
Department of Electronic Engineering, Tsinghua University, Beijing,
China
Email: yxpoke@sina.com
Mao Yang
Department of Electronic Engineering, Tsinghua University
Department of Electronic Engineering, Tsinghua University, Beijing,
China
Email: yangmao210@163.com
Yong Li
Department of Electronic Engineering, Tsinghua University
Department of Electronic Engineering, Tsinghua University, Beijing,
China
Email: liyong07@tsinghua.edu.cn
Depeng Jin
Department of Electronic Engineering, Tsinghua University
Department of Electronic Engineering, Tsinghua University, Beijing,
China
Email: jindp@mail.tsinghua.edu.cn
Li Su
Department of Electronic Engineering, Tsinghua University
Department of Electronic Engineering, Tsinghua University, Beijing,
China
Email: lisu@tsinghua.edu.cn
Zeng Expires December 12, 2014 [Page 12]
