P2PSIP Gang.Li Internet Draft China Mobile Intended status: Informational Yunfei.Zhang Expires: September 2010 China Mobile Baohong.He China CATR Shihui.Duan China CATR Cheng.Li China CATR Jianxin.Xie Huawei Technologies March 9, 2010 Implementation of a large scale carrier-level VoIP system using P2PSIP draft-zhang-p2psip-bcp-01.txt Status of this Memo This Internet-Draft is submitted in full conformance with the provisions of BCP 78 and BCP 79. Internet-Drafts are working documents of the Internet Engineering Task Force (IETF), its areas, and its working groups. Note that other groups may also distribute working documents as Internet-Drafts. Internet-Drafts are draft documents valid for a maximum of six months and may be updated, replaced, or obsoleted by other documents at any time. It is inappropriate to use Internet-Drafts as reference material or to cite them other than as "work in progress." The list of current Internet-Drafts can be accessed at http://www.ietf.org/ietf/1id-abstracts.txt The list of Internet-Draft Shadow Directories can be accessed at http://www.ietf.org/shadow.html This Internet-Draft will expire on September 9, 2010. Copyright Notice Copyright (c) 2010 IETF Trust and the persons identified as the document authors. All rights reserved. Expires September 1 2010 [Page 1] Internet-Draft p2psip bcp March 2010 This document is subject to BCP 78 and the IETF Trust's Legal Provisions Relating to IETF Documents (http://trustee.ietf.org/license-info) in effect on the date of publication of this document. Please review these documents carefully, as they describe your rights and restrictions with respect to this document. Abstract This document introduces a practical work for Peer-to-Peer (P2P) SIP system based on Distributed Service Network (DSN), which was proposed by China Mobile in ITU-T. In this document, it introduces some key problem of carrier grade P2PSIP VoIP system. Then it gives a brief introduction on DSN VoIP system and especially discusses some key technologies aimed at those mentioned problems. At last, it presents some validation works for those technologies. Expires September 9, 2010 [Page 2] Internet-Draft p2psip bcp March 2010 Table of Contents 1. Introduction ................................................ 4 2. Requirements and key problems of carrier grade P2P VoIP system 4 2.1. Requirements ........................................... 4 2.2. Some Key problems....................................... 5 3. DSN VoIP System Overview..................................... 6 3.1. System Architecture..................................... 6 3.2. System Implementation and Components ....................8 3.3. System Key technologies................................ 10 4. Validation of DSN VoIP system............................... 12 4.1. Overview .............................................. 12 4.2. Validation of application layer ........................12 4.3. Validation of P2PSIP layer............................. 14 4.3.1. Reliability....................................... 14 4.3.2. Uniformity Distribution........................... 15 4.3.3. SPM Bandwidth Consumption .........................16 4.3.4. others ........................................... 17 5. Security Considerations..................................... 17 6. IANA Considerations ........................................ 17 7. Conclusions ................................................ 17 8. References ................................................. 17 8.1. Normative References................................... 17 8.2. Informative References................................. 17 9. Acknowledgments ............................................ 18 Expires September 9, 2010 [Page 3] Internet-Draft p2psip bcp March 2010 1. Introduction DSN[1],the abbreviation of Distributed Services Network, is a new question being standardized in ITU-T proposed by China Mobile. [2] outlines the DSN Architecture. As described in [2], DSN can support many applications, such as Multimedia telephony services, Streaming services, Content distribution service and so on. DSN VoIP system is designed based on the theory of DSN and P2PSIP. According to [1], DSN VoIP system includes two layers over IP network: P2PSIP layer and application layer. Application layer uses SIP protocol for call and P2PSIP layer uses RELOAD[3] protocol for routing calls in Overlay. It also introduces some key problems in P2P VoIP system and the corresponding requirements in this document. Then we introduce the system architecture, components and key technologies. At last we do some measurement and simulation experiments for validating system. 2. Requirements and key problems of carrier grade P2P VoIP system 2.1. Requirements Peer-to-Peer(P2P) systems have been widely applied in current internet, due to their advantages such as high scalability and costeffectiveness. Many P2P applications have been proposed to implement the traditional telecom voice service(for example, IMS or softswitch) such as Skype. But it's seldom discussed how to make P2P system meet the telecom infrastructure performance requirements, which is usually called as the "Carrier Grade" requirements. According to the current performance of telecom voice service deployments, the requirements of the carrier-grade services can be summarized but not limited to those listed below: 1. Qos guarantee: Current IMS requires that any client-side operations should be able to get the final responses in 300 milliseconds. So P2P VoIP system must guarantee the signaling response time is comparable to this time and the media transmission time is less than 400ms. There also are some specifications for voice quality. Expires September 9, 2010 [Page 4] Internet-Draft p2psip bcp March 2010 2. High Availability: The high availability includes network availability, service availability, and subscriber data availability and so on. The most common practice is to use some special mechanisms to acquire robustness and availability such as hot backup redundancy. High availability is a severe challenge for P2P VoIP systems which intends to achieve the same level of the telecom systems. 3. Scalability: The P2P VoIP system must be adapted to the expanding of the system scale, such as the increasing of the number of the clients or core nodes. 4. Load balance: The resource is distributed among P2P nodes and each node must collaborate with one another to void the emergence of the centralization of resource and traffic. 5. Cost-effectiveness: It's the aim that the P2P VoIP system can achieve the same performance as the telecom commercial system by fully utilizing just commodity PCs, not using the costly hardware. 6. Maintainability: it's quite appealing that it needs to do a little work to configure the nodes and network in order to maintain the natural operation of the P2P VoIP system. P2P VoIP system can realize the self-organization even after any network failure, so as to reduce the demand for emergent response and well experienced operators. 7. Others: TBD. 2.2. Some Key problems To obtain the same performance as the telecom system, there are many key problems to be settled. Some of them are listed as below: 1. Routing performance: The traditional P2P routing performance is relative to the network dimensions. The routing performance will be sharply descends as the increase of the network dimensions, for example, the traditional Chord routing performance is O(logN). 2. Redundant traffic: The P2P network does not match with the underlying network, so the P2P nodes forward the traffic only according to the logic P2P routing and not to the real optimal IP layer network routing, and this causes mass P2P redundant traffic across many Automatic Systems(AS). Expires September 9, 2010 [Page 5] Internet-Draft p2psip bcp March 2010 3. The reliability and availability of subscriber data: Current P2P network can't provide an efficient solution for the reliability and availability of subscriber data in case of the random joining or exiting of the P2P nodes happened in the network. 4. Duplicate data consistency: There are many replicas for subscriber data to acquire enough availability. It's very common that the subscriber data are often modified, so the system must allow for updates and amendments to be propagated to all replicas asynchronously. 5. Hot spot: The subscriber data are distributed uneven in the network, which will cause that some nodes have much more subscriber data than others. Therefore, it will probably leads to the high traffic burdened by these nodes, which might makes these nodes overload and even fault in some cases. 6. Single node failure: If a node fails, its clients will not gain the service. 7. Others: TBD. It discusses some key technologies in section 3.3 with a view to solve the above questions. 3. DSN VoIP System Overview DSN VoIP system is a VoIP system based on P2P overlay. DSN VoIP system uses SIP protocol for call transaction and RELOAD for call routing. 3.1. System Architecture According to the Architecture of RELOAD, DSN VoIP system takes the similar architecture and contains some essential components in RELOAD, such as Usage Layer, Routing Layer, Storage and so on. The architecture of DSN VoIP system is shown as figure 1. Expires September 9, 2010 [Page 6] Internet-Draft p2psip bcp March 2010 +-----------------------------------------------------------+ | +-------------------------------------------------------+ | | |Application layer | | | | +---------- --+ +------------+ | | | | | SIP Usage | | XMPP Usage | ... | | | | +-------------+ +------------+ | | | +-------------------------------------------------------+ | | ^ ^ | | =================Message Routing API===================== | | v v | | +------------------------------------------------------ + | | |P2PSIP layer | | | | +-------------------------------------------------- + | | | | | Data Storage | | | | | |-------------------------------------------------- | | | | | |Parallel Recovery| |Consistency Mgt| |User Index| | | | | | |------------------------------------------------ --| | | | | | Replica Placement Strategy | | User Profile | | | | | +----------------------------------------------- ---+ | | | | ^ ^ | | | | v v | | | | +---------------------------------------------------+ | | | | | Key Based Routing(KBR) | | | | | |---------------------------------------------------| | | | | | ID Mgt | | DHT Protocol | | Topo Maintenance| | | | | | +---------------------------------------------------+ | | | +-------------------------------------------------------+ | | ^ ^ | | ====================Transport API======================== | | v v | | +-------------------------------------------------------+ | | | +------------------------------------------------ --+ | | | | | Communication | | | | | |---------------------------------------------------| | | | | | TCP | | UDP | | Other | | | | | +------------------------------------------------- -| | | | | IP Network | | | | | | | +--------------------------------------------------- ---+ | +----------------------------------------------------- -----+ Figure 1 The architecture of DSN VoIP system Based on the RELOAD architecture, DSN VoIP system has implemented the necessary functions required by RELOAD and some additional functions to make the system more practicability. Expires September 9, 2010 [Page 7] Internet-Draft p2psip bcp March 2010 There are two basic modules in P2PSIP layer: KBR and Data Storage module. The key based routing, KBR, refers to find the best suitable host for an input key, also includes ID management, DHT routing protocol, Topology Maintenance, peers failure detecting and etc. DSN VoIP system implements a unique KBR routing scheme with real-time response and a traffic localization mechanism, including a novel ID assignment method. The data storage module takes charge of management of subscriber data, including storage, restoring, consistency verification and etc. Here DSN VoIP system implements a unique and practical replica placement strategy and an effective consistency strategy. 3.2. System Implementation and Components DSN VoIP system includes distributed servers and subscriber terminals. These servers form the DSN VoIP network through P2P protocol and share load for each other. According to the different function, the server can be Peer Node(PN), Application Server(AS), Enrollment Server(ES), Edge Agent(EA), Super Peer-Maintenance (SPM),Relay Node(RN) and RN Cluster Server(RCS). According to the support for P2P protocol, the subscriber terminal can be the traditional SIP Client(SiC) and Peer Client(PeC). The DSN VoIP system framework is shown as figure 2. Expires September 9, 2010 [Page 8] Internet-Draft p2psip bcp March 2010 +-----+ +----+ +-----+ | ES | | AS | | SPM | +-----+ +----+ +-----+ | | | | | | +----+ +----+ +----+ +----+ | PN |----| PN |----| PN |- ... -| PN | +----+ +----+ +----+ +----+ | | | | +----+ +----+ +----+ +----+ | PN |----| PN |----| PN |- ... -| PN | +----+ +----+ +----+ +----+ | | | | | | +----+ +----+ +----+ | EA | | PeC| | RCS| +----+ +----+ +----+ | | | | +----+ +----+ | SiC| | RN | +----+ +----+ Figure 2 The framework and components of DSN VoIP system Peer Node(PN): PN is the core node in DSN VoIP network and is deployed by telecom operators. PN is responsible for DHT topology Maintenance, overlay routing, storage and access for subscriber/service data, subscriber registration and Authentication, BootStrap, SIP session control and VoIP service. Application Server(AS): AS can provide other services which can't be provided by Peer Node(PN). Enrollment Server(ES): ES provides BootStrap service when the PeC or PN joins the DSN VoIP network for the first time. When new PN joins, ES is responsible for Authentication for the new node. ES is deployed by telecom operators and fixed in the network. Edge Agent(EA):EA provides the relay service for SiC and adapt to all kinds of services and connections to PN. EA support SIP proxy discovery for DSN and connection with PN. SPM(Super Peer Maintenance): SPM is an administrator for one zone. SPM undertakes the registration of the PN nodes which are in the same zone, assigns the node ID to PN nodes and maintain DHT routing information. Expires September 9, 2010 [Page 9] Internet-Draft p2psip bcp March 2010 Relay Node(RN): RN is responsible for NAT traversal and media traffic forwarding. RN can discover the DSN and connection with PN. RN can be deployed by telecom operators or upgraded PeC. RN Cluster Server(RCS): RCS is responsible for the management of RN nodes according to topology information or RTT(Round-Trip Time). RCS manages one AS field and assigns a suitable RN for PeC according to the information about PeC's request or location etc. SIP Client(SiC): SiC support only SIP protocol and can't discover the DSN network if it is directly connected to the DSN. It must connect directly with Edge Agent (EA). Peer Client(PeC): PeC support both SIP protocol and P2P protocol. PeC can be upgraded to be Relay Node (RN) or RN Cluster, but can't be Peer Node(PN). PN nodes form the DSN VoIP network and interconnect via P2P protocol. The subscriber terminal can get valid service once it connects to any PN. If some but not all PN nodes fail or go offline, this will not affect the normal operation of DSN network. We can add more new PN nodes to augment the capacity of DSN network. 3.3. System Key technologies In this section, it presents with some key technologies which is aiming at solving the problem in sec 2.2. 1. SPM assisted one hop route: DSN VoIP system is typically deployed as a two-layer DHT, including a global DHT (correspond to countrywide) and several regional DHTs (correspond to provinces). For each region, there are several special peers dedicated to routing information update and some other management tasks and these special peers are SPM. Multiple SPMs can be deployed for a large region. Each one is responsible for a partition of PN belonging to that region, and these SPMs backup each other. SPMs from different regions are full-mesh connected, they run a gossip- based protocol to maintain a strongly consistent view of their existence. All PN nodes must register to the local SPM of the same region, when they join the network. During runtime, once the PN's state change is detected by its neighboring PN, the neighbor will notify its local SPM, and then the SPM will disseminate the event to other SPMs, finally each SPM will broadcast this notification to all PN nodes under the charge of it. Each PN thus can obtain other PN's information through SPM and can maintains routing table for all PNs and we enlarge the routing table's size on each PN. So SPM assisted one hop route can be implemented. Expires September 9, 2010 [Page 10] Internet-Draft p2psip bcp March 2010 2. Traffic Localization: Recent studies have shown that a large volume of inter-domain redundant traffic by P2P mismatch problem already became the serious problem. As mentioned above, each PN node joins the countrywide global DHT and one regional DHT at the same time. We use a novel peer ID assignment scheme optimized for global load balancing, which can write the regional information into peer ID. With this unique ID scheme, the global DHT and regional DHT can be maintained by only one KBR(Key Based Routing) incarnation, that's different with many other layered DHT proposed. Every object will be put in the regional DHT and global DHT for disaster recovery, according to the normal DHT semantics internally. 3. Replica Placement strategy: The DSN VoIP system raises a unique N:N replica placement which provides a effective way to backup and recovery subscriber data from local and remote region. Under this solution a parallel recovery factor N is defined as how many candidates should be used as the recovery. And B is defined as the number of Backups which means how many duplicated replicas should the overlay hold. subscriber data in each primary peer node is divided into S=N/B slices After that, each B backup peers in a group for this primary peer will backup the same slice of subscriber data. One peer node could execute data backup procedure to send data to different backup peers simultaneously. Compare to 1:1 backup mechanism, not only does our N:N replica placement increase the efficiency of parallel recovery, but also avoids the reservation of half capacities (CPU cycles, bandwidth) for potential migration of the service backed up. 4. Consistency Strategy: In order to ensure better data consistency, DSN VoIP system will trigger "DataCheck" and "DataGoHome" mechanisms periodically. DSN VoIP system through "DataCheck" triggers the comparison between the data version information of the corresponding PN nodes, and updates the inconsistent data to ensure that the eventual consistency of subscriber profiles. "DataGoHome" takes charge of identify those temporary irregularities data, recalculate their key-value pair and push them to correct PN nodes. With the help of timing service always available in telecom offices, DSN VoIP system uses timestamp in order to capture causality between different versions of the same data. One can determine whether two versions of a data have a causal ordering by examine their timestamps. Expires September 9, 2010 [Page 11] Internet-Draft p2psip bcp March 2010 5. Strip Segmentation method Based ID Assignment: We use this method to generate the subscriber ID. This method can evenly distribute the subscriber data on each PN node so as to achieve traffic load balance and avoid hot spot. The detailed method can be referred to [4]. 6. Single node failure handle: If a PN node faults, we design some mechanisms to ensure the natural operation of VoIP service. For the session in progress, the session between the caller and the callee is established, the standby PN checks the primary PN in failure, and the standby PN notifies the associated nodes and switch the session to itself. For the new call originated from the caller, if the standby PN has switch to be the primary PN and notified the subscribers, the call originated from the subscriber will be routed to the standby PN. For the new call originated from the callee, if the callee knows the failure in the primary PN by requesting routing or querying routing table, the callee will route the query to the standby PN and establishes the session. 7. Others: TBD. 4. Validation of DSN VoIP system 4.1. Overview We do some measurement and simulation experiments to validate the DSN VoIP system and its key technologies. The SIP protocol is implemented in the application layer and we validate the call disposal capability in this layer. The P2PSIP protocol is implemented in the P2PSIP layer and we do some simulation experiments to validate some key technologies in this layer. 4.2. Validation of application layer We set up a DSN VoIP demo system in CMCC lab and do some measurement for the practical system. Demo system is made up of twenty PC servers and two management PC servers. Twenty PC servers are PN nodes for overlay network. All these PN nodes have the same hardware and software configuration: Pentium(R) Dual-Core CPU E5200, 2.50GHz,4G memory and 300G hard disk; OS is SUSE Linux Enterprise Desktop 10 SP2 (i586). All PC servers connect through three D-LINK layer2 switch. For measuring the bulk call performance of the demo system, we configure the following parameters: Expires September 9, 2010 [Page 12] Internet-Draft p2psip bcp March 2010 1.call parameters: each call last time=30 second each measurement last time = 40 minutes 2. subscriber data parameters: subscriber number for each PN: 20 thousand caller subscriber number = callee subscriber number = 6000 The format of SIP subscriber terminal ID is like 6xxxxxxx@bj.dsn.com. The callers and callees must be registered before bulk call performance measurement. All these registered subscribers and calls handled by each PN are randomly averagely distributed in system. Our algorithm can ensure the well-proportioned distribution for registered subscribers and calls handled by each PN. We measured the maximal call transaction performance when the demo system respectively included 4,8,12,16,20 PN nodes. We collected the following parameters: 1.Registration statistics: average register time 2.Call Establishment time: average call setup time, average call tear down time, Post dial delay 3.Call finish: call success ratio, call attempts per second(CAPS) If the call lose ratio is below 0.5% and don't persist to increase, we think the call performance is under the limit of the system. The following are the results. 1.average register time Node number 4 8 12 16 20 average register time(ms): 34 25 26 32 32 2. average call setup time Node number 4 8 12 16 20 average setup time(ms): 272 247 251 884 236 Expires September 9, 2010 [Page 13] Internet-Draft p2psip bcp March 2010 3. average call tear down time Node number 4 8 12 16 20 average tear down time(ms): 161 142 152 406 171 4. Post dial delay Node number 4 8 12 16 20 Post dial delay (ms): 260 235 239 1004 229 5. call success ratio Node number 4 8 12 16 20 call success ratio(%): 100 100 99.99 99.94 99.89 6. call attempts per second(CAPS) Node number 4 8 12 16 20 CAPS: 48 84 129 172 216 Total call number: 230400 403200 619196 824880 925546 Through above measurement, the conclusion is: 1. The DSN VoIP system has the capability for call transaction. 2. The system capability can approximately linearly increase as the number of PN nodes increase. We have measure the bulk call performance for the demo system. However due to measurement time limited, these results are just elementary and further measurement will be done. 4.3. Validation of P2PSIP layer We establish an simulation environment with 3000 PN nodes located in three regions separately and There is one SPM node in each region. 4.3.1. Reliability This measure will validate the replica placement and consistency strategy. We evaluate the reliability of DSN VoIP system in the face of a variety of failure and application models. Expires September 9, 2010 [Page 14] Internet-Draft p2psip bcp March 2010 Table 1 the reliability of DSN VoIP system +-----------------------------------------------------------+ | Application Model Failure Model Success Rate | | write:1/s per node Failure1 (1%) 100% | | read:10/s per node Failure1 (3%) 100% | | Failure2 99.9992% | | Failure3 99.951% | +-----------------------------------------------------------+ | write:30/s per node Failure1 (1%) 99.99995% | | read:50/s per node Failure1 (3%) 99.9997% | | Failure2 99.9959% | | Failure3 99.68% | +-----------------------------------------------------------+ | write:50/s per node Failure1 (1%) 99.99992% | | read:100/s per node Failure1 (3%) 99.9973% | | Failure2 99.9934% | | Failure3 99.30% | +-----------------------------------------------------------+ As shown in Table 1, under normal application model, each PN received 1 write and 10 read requests per second. 1% Failure1 model will result in 1% of peers fail in every hour, and in the next hour 80% of fail peers will come back. As is clear from Table 1, the 1% and 3% Failure1 model almost no impact on business success rate. The success rate in 12 hours is almost 100%. The Failure2 and Failure3 model lead to more number of peers' failure, but the system still able to maintain more than 99.9% success rate. With the increased of business requests, the success rate has declined, but still more than 99%. 4.3.2. Uniformity Distribution We propose Strip Segmentation method Based ID Assignment for the uniformity distribution of subscriber data and PN node ID. We randomly generated 10 million subscriber data, and put them into the 3000 peers based on DHT key-value rule. Finally, we made a statistic for distribution of these data in 3000 peers and we can see that the subscriber data obeys a Gaussian-like distribution through the optimization of strip segmentation arithmetic and ID assignment mechanism. The number of data in 98% of PN nodes maintained at around 10000. The model of data stored in DSN VoIP system has perfect load- sharing so that it also improves the traffic balance finally. We also validate this from the application layer measurement. We randomly choose 12 thousand ordinal subscribers and don't know how these subscribers distribute. Through measurement, we can see that the number of registered subscribers on each PN is approximately equal. So the call traffic undertaken by each PN is also Expires September 9, 2010 [Page 15] Internet-Draft p2psip bcp March 2010 approximately equal. Here we give the below results to verify it( for the limited space, here only give the random 8 PN nodes call number). Table 2 the call distribution of DSN VoIP system +-------------------------------------------------------------+ | Node Number | 4 | 8 | 12 | 16 | 20 | +-------------------------------------------------------------+ | Call Distribution | 56695 | 51655 | 53796 | 53198 | 48822 | | | 57835 | 51446 | 52377 | 51234 | 47857 | | | 57778 | 50292 | 52475 | 53987 | 50644 | | | 58092 | 51124 | 50540 | 52334 | 51238 | | | | 50575 | 50589 | 52290 | 46895 | | | | 51848 | 50885 | 50343 | 45838 | | | | 50300 | 48851 | 50022 | 48575 | | | | 48125 | 53523 | 54014 | 46805 | +-------------------------------------------------------------+ So the Strip Segmentation method is useful to implement load balance among the DSN VoIP network. 4.3.3. SPM Bandwidth Consumption Although SPM node is only responsible for the maintenance and synchronization of One-hop Routing Table in DSN VoIP system, its bandwidth consumption is still one of concerned issues. We recorded the bandwidth consumption of SPM under various failure models. Table 3 the bandwidth consumption of SPM +-------------------------------------------------------+ | Failure Model | Bandwidth Consumption (KB/S) | +-------------------------------------------------------+ | Failure1 (1%) | 1.9 | | Failure1 (5%) | 34.8 | | Failure1 (10%) | 75.6 | | Failure2 (50fails) | 93 | | Failure2 (100fails) | 210 | | Failure2 (200fails) | 450 | | Failure2 (500fails) | 820 | | Failure3 | 15.3 | +-------------------------------------------------------+ As can be seen in table 3, in the general failure model such as Failure1 model, the bandwidth consumption of SPM is very tiny, almost can be ignored. Even in the case of Failure2 model (almost 50% of peers in same region failed simultaneously); the bandwidth consumption is just 820KB/s. For such a very low probability of unexpected events, this overhead of SPM is totally acceptable. It is Expires September 9, 2010 [Page 16] Internet-Draft p2psip bcp March 2010 worth mentioning that SPM uses a unified manner to inform the routing table change in Failure3 model, so the bandwidth consumption maintained at a very small value. 4.3.4. others TBD 5. Security Considerations This draft does not introduce any new security issues. 6. IANA Considerations This memo includes no request to IANA. 7. Conclusions Through the measurement for the demo system and simulation experiments, we validate that DSN VoIP system is a potential operationable system, which possesses a certain degree of capability for call transaction, reliability, fault tolerance and load balance. 8. References 8.1. Normative References [1] ITU-T DSN, Proposed scope of DSN, http://www.itu.int/md/meetingdoc.asp?lang=en&parent=T09-SG13- 090112-C&PageLB=50. 8.2. Informative References [2] [draft-zhang-ppsp-dsn-introduction-00]www.ietf.org/internet- draft/draft-zhang-ppsp-dsn-introduction-00.txt [3] [draft-ietf-p2psip-base-07]www.ietf.org/id/draft-ietf-p2psip- base-07.txt [4] Guangyu Shi, Jian Chen, Hao Gong, Lingyuan Fan, Haiqiang Xue, Qingming Lu, and Liang Liang, " SandStone: A DHT based Carrier Grade Distributed Storage System ", Proc. ICPP 2009 Expires September 9, 2010 [Page 17] Internet-Draft p2psip bcp March 2010 9. Acknowledgments Thanks to Haitao Yang,Xiaolin Jiang, Jian Zhang and Jiguang Cao for their hard efforts, comments and suggestions. Special thanks to Guangwu He and Fengbo Wang for very detailed technical support for DSN VoIP system. Expires September 9, 2010 [Page 18] Internet-Draft p2psip bcp March 2010 Authors' Addresses Gang Li China Mobile Beijing 100045, China Phone: 86-13501279120 Email: ligangyf@chinamobile.com Yunfei Zhang China Mobile Beijing 100045, China Phone: 86-13601032119 Email: zhangyunfei@chinamobile.com Baohong He CATR Beijing 100045, China Phone: 86-10-68094120 Email: hebaohong@catr.cn Shihui Duan CATR Beijing 100045, China Phone: 86-10-68094321 Email: duanshihui@catr.cn Cheng Li CATR Beijing 100045, China Phone: 86-10-68094138 Email: licheng@catr.cn Jianxin Xie Huawei Technologies Shenzhen 518129, China Phone: 86-0755-28780808 Email: xiejx@huawei.com Expires September 9, 2010 [Page 19]