Network Working Group Y. Zhang Internet-Draft Y. Cui Intended status: Informational Q. Yin Expires: December 27, 2011 Shanghai Jiaotong University Y. Wang Huawei June 25, 2011 Analysis and Evaluation of Universal Storage Protocol Based on Embedded Linux draft-zhang-decade-transport-comparison-00 Abstract "Traffic Localization" is proposed to release the pressure of backbone mobile networks brought by increasing multimedia applications. A universal storage protocol is needed to serve the purpose on the edge storage nodes. As the three candidates, HTTP, NFS and iSCSI are compared concerning their implementation and performance on embeded clients and servers in this paper. Their differences are discussed based on the experiment, and our suggestion for universal storage protocols on edge storage nodes are put forward. Status of this Memo This Internet-Draft is submitted in full conformance with the provisions of BCP 78 and BCP 79. Internet-Drafts are working documents of the Internet Engineering Task Force (IETF). Note that other groups may also distribute working documents as Internet-Drafts. The list of current Internet- Drafts is at http://datatracker.ietf.org/drafts/current/. Internet-Drafts are draft documents valid for a maximum of six months and may be updated, replaced, or obsoleted by other documents at any time. It is inappropriate to use Internet-Drafts as reference material or to cite them other than as "work in progress." This Internet-Draft will expire on December 27, 2011. Copyright Notice Copyright (c) 2011 IETF Trust and the persons identified as the document authors. All rights reserved. This document is subject to BCP 78 and the IETF Trust's Legal Zhang, et al. Expires December 27, 2011 [Page 1] Internet-Draft transport protocols comparison June 2011 Provisions Relating to IETF Documents (http://trustee.ietf.org/license-info) in effect on the date of publication of this document. Please review these documents carefully, as they describe your rights and restrictions with respect to this document. Code Components extracted from this document must include Simplified BSD License text as described in Section 4.e of the Trust Legal Provisions and are provided without warranty as described in the Simplified BSD License. Table of Contents 1. Introduction . . . . . . . . . . . . . . . . . . . . . . . . . 3 2. Backgrounds . . . . . . . . . . . . . . . . . . . . . . . . . 3 3. TEST APPROACH AND TEST ENVIRONMENT . . . . . . . . . . . . . . 5 3.1. Testbed setting . . . . . . . . . . . . . . . . . . . . . 5 3.2. Test Cases . . . . . . . . . . . . . . . . . . . . . . . . 6 3.3. Implementation of Tests . . . . . . . . . . . . . . . . . 7 4. RESULTS OF PERFORMANCE TESTS . . . . . . . . . . . . . . . . . 7 4.1. Test Results . . . . . . . . . . . . . . . . . . . . . . . 7 4.2. Analysis of Test Results . . . . . . . . . . . . . . . . . 10 5. Conclusion . . . . . . . . . . . . . . . . . . . . . . . . . . 12 6. Security Considerations . . . . . . . . . . . . . . . . . . . 12 7. IANA Considerations . . . . . . . . . . . . . . . . . . . . . 13 8. Informative References . . . . . . . . . . . . . . . . . . . . 13 Authors' Addresses . . . . . . . . . . . . . . . . . . . . . . . . 14 Zhang, et al. Expires December 27, 2011 [Page 2] Internet-Draft transport protocols comparison June 2011 1. Introduction With the rising number of multimedia applications on mobile phones, the mobile backbone network is facing considerable pressure because its limited bandwidth will fail to carry all the multimedia traffic in three or five years. "Traffic Localization", which introduces storage device on the edge network which stores the frequently used resources, is a possible way to solve the problem. Thus, a universal storage protocol is required to carry traffic between end users and the edge storage device. HTTP, NFS and iSCSI are commonly used storage proto- cols presently. The comparison and evaluation of these three protocols offers referential value for the design of universal storage protocols. As a result, the performance tests of the three protocols are conducted on embedded end users and normal servers in this paper. At the end of the paper, our suggestion for universal storage protocols on edge storage nodes are put forward. 2. Backgrounds When we discuss network storage solutions, we frequently refer to protocols. There are numerous protocols which participate in storage networks at various different layers, such as, NFS (Network File System), CIFS (Common Internet File Services), iSCSI (Internet Small Computer System Interface), HTTP (Hypertext Transfer Protocol), etc. In our research, we pay a major concern on the difference among HTTP, NFS, and iSCSI. NFS, short for network file system, allows an end user to access files over networks in a manner similar to that of files on local disks. iSCSI is an abbreviation of Internet Small Computer System Interface carrying SCSI commands over IP networks, which is used to facilitate data transfers over intranets and to manage storage over long distances. Zhang, et al. Expires December 27, 2011 [Page 3] Internet-Draft transport protocols comparison June 2011 +----------------------------------------------------------+ | protocol | HTTP | NFS | iSCSI | |----------------------------------------------------------| | data | html files | all files | blocks | |----------------------------------------------------------| |data sharing |simply |supported | not supported| | |supported | | | |----------------------------------------------------------| | supported OS| Windows,Linux|Linux, Unix |Windows,Linux,| | | , Unix | |Unix | |----------------------------------------------------------| |coding |command field:|XDR | binary | | |base 64; data | | | | |field:optional| | | |----------------------------------------------------------| |file system |at server side|at server side|at user side | |location | | | | |----------------------------------------------------------| |flow control |based on TCP | none | based on TCP | |----------------------------------------------------------| |server level |user mode |kernel mode |kernel mode | | |kernel mode | | | |----------------------------------------------------------| |user level |user mode |kernel mode |kernel mode | |----------------------------------------------------------| |# of packages|3(request, |6(readdir,geta| | |when |response, |ddr,request, | uncertain | |uploading |confirmation) |response) | | |files | | | | |----------------------------------------------------------| |stateful or |stateless | stateless | stateful | |stateless | | | | |----------------------------------------------------------| |system calls |get/put/post/d|file system |SCSI commands | |used |elete |commands | | |----------------------------------------------------------| |scalability |excellent |ordinary | poor | +----------------------------------------------------------+ Table 1 Comparison between HTTP, NFS and iSCSI From the prospective of implementation details, protocol modules, and application situations, we compare HTTP, NFS and iSCSI to give the analysis and evaluation of protocols used for network storage. Zhang, et al. Expires December 27, 2011 [Page 4] Internet-Draft transport protocols comparison June 2011 3. TEST APPROACH AND TEST ENVIRONMENT 3.1. Testbed setting The performance test is divided into two parts: performance test of a embedded client and a normal server. For the client performance test, we choose an embedded development board-ideal6410 as our embedded user which employs arm11 as its CPU. Linux operation system is mi- grated onto the board and transplanted a collection of perfor- mance monitoring tools onto embedded Linux, such as cURL (cURL is a tool for transferring data using HTTP), open- iscsi, portmap, and SYSSTAT - - /----\ +------+ / -\ / \ +-------+ | | / \-- | | | | | | -/ |embeded| |Server|----- / 100Mbit Ethernet\-----|client | | | / \ | | | | / -/ | | +------+ | /----/ +-------+ \--\ -- / \/ \--/ Figure 1. Test Environment for the Client Performance Test At the server side, a mass storage server and several similarly- deployed PCs are prepared which are connected to the Gigabit Ethernet LANs so as to avoid the bottleneck of network bandwidth. - - /----\ +--+ +------+ / -\ / \ +--|PC| | | / \-- | | | | | -/ | +--+ |Server|----- /Gigabit Ethernet\--|--|PC| 7 PCs | | / \ | +--+ | | / -/ | ... +------+ | /----/ | +--+ \--\ -- / \/ +--|PC| \--/ +--+ Figure 2. Test Environment for the Client Performance Test Zhang, et al. Expires December 27, 2011 [Page 5] Internet-Draft transport protocols comparison June 2011 3.2. Test Cases According to the actual network applications, for example download movies, play on-line games, surfing the Internet, publish blogs, we can classify the data transferred into two groups: data-intensive workloads for large files and metadata-intensive workloads for small files. As a matter of fact, we design three test cases to do the simulation: (1)Data-intensive workloads: We download and upload a large file from or to the server both in the embedded environment and in the multiple clients environment.The period of test must be longer than five minutes to ensure the reliability of our test. (2)Metadata-intensive workloads: We upload, download and list 50000 small files (1.1KB for each) from or to the server both in the embedded environment and multiple clients environment. (3)Test of transmission efficiency: Besides the two given cases, the transmission efficiency and number of actual interactive packages are also very important indicators when evaluating protocols. We create a folder of 40MB files, which is composed of 1KB*40960, 4KB*10240, 64KB*640 or 1MB*40 files. We use different protocols to transfer those files to observe transmission efficiency of each protocol. Here we define transmission result as the division of effective data size and total data size transmitted. +-------------------------------------------+ | |throughput for |throughput for | | -- |downloading(MBps)|uploading(MBps) | |-------------------------------------------| |HTTP | 0.75 | 0.84 | |-------------------------------------------| |NFS | 0.71 | 0.84 | |-------------------------------------------| |iSCSI | 0.72 | 0.77 | +-------------------------------------------+ Table 2. Throughput for data-intensive files Zhang, et al. Expires December 27, 2011 [Page 6] Internet-Draft transport protocols comparison June 2011 +-------------------------------------------+ | |fops for |fops for | | -- |downloading(fops)|uploading(fops) | |-------------------------------------------| |HTTP | 83.46 | 58.22 | |-------------------------------------------| |NFS | 106.97 | 36.99 | |-------------------------------------------| |iSCSI | 161.90 | 80.71 | +-------------------------------------------+ Table 3. File operations per second for metadata-intensive files 3.3. Implementation of Tests Actually the existing tools for performance test such as Loadrunneriozone and iometer, can only support one specific protocol. This drawback makes it inconvenient to test and evaluate several protocols at the same time. Therefore, we develop a set of tools which can work over different protocols to solve the problem which include several shell scripts to ge and analyze results and a C program to figure out transmission efficiency. 4. RESULTS OF PERFORMANCE TESTS 4.1. Test Results 1) Performance of Embedded Clients: The throughput for data-intensive workloads, that is the size of data transmitted for second, and file- operations per second for meta-intensive workloads are the two indicators of protocol performance of the embedded client end. Our test results for this part are shown in Table 1 and Table 2: 2) Performance of the Server: The indicator we concern about are similar to those of client performance tests. The number of clients are gradually increased and the performance of the server is monitored accordingly. The test results for server performance are shown in Fig 3 , Fig 4, Table 4 and Table 5: Zhang, et al. Expires December 27, 2011 [Page 7] Internet-Draft transport protocols comparison June 2011 +-------------------------------------------+ | | | | | | | | | | -- | 1 | 2 | 3 | 4 | 5 | 6 | 7 | |-------------------------------------------| |HTTP |33.3|67.1|78.1|82.2|76.4|78.0|71.1| |-------------------------------------------| |NFS |64.2|51.2|39.9|39.8|41.1|38.2|38.1| |-------------------------------------------| |iSCSI |80.5|86.4|72.2|81.1|89.2|84.2|87.1| +-------------------------------------------+ Figure 3. Throughput for downloading large files (this figure will be added soon.) Figure 4 Throughput for uploading large files The test results for transmission efficiency are shown in Fig 5 to Fig 8. +---------------------------------+ |Protocol |HTTP |NFS |iSCSI | |---------------------------------| | 1 |616 | 75 |2246 | |---------------------------------| | 2 |582 |127 |4418 | |---------------------------------| | 3 |270 | 90 | 3965 | |---------------------------------| | 4 |239 | 88 | 5628 | |---------------------------------| | 5 |281 | 93 | 4972 | |---------------------------------| | 6 |345 | 90 |6462 | |---------------------------------| | 7 |350 | 86 |- | +---------------------------------+ Table 4. Fops for downloading files Zhang, et al. Expires December 27, 2011 [Page 8] Internet-Draft transport protocols comparison June 2011 +---------------------------------+ |Protocol |HTTP |NFS |iSCSI | |---------------------------------| | 1 |722 |42 |8960 | |---------------------------------| | 2 |384 |77 |1605 | |---------------------------------| | 3 |483 |112 |1596 | |---------------------------------| | 4 |561 |138 |1883 | |---------------------------------| | 5 |527 |163 |2039 | |---------------------------------| | 6 |576 |196 |2396 | |---------------------------------| | 7 |548 |184 |- | +---------------------------------+ Table 5. Fops for uploading files +-----------------------------+ |1K | 4K | 64K | 1M | +---------------------------------------| | http |66.73 | 48.08 | 42.73 |41.82| |---------------------------------------| | nfs |79.70 | 51.74 | 43.06 |42.13| |---------------------------------------| | iscsi |186.76 | 46.60 | 46.18 |42.40| +---------------------------------------+ Figure 5. Size of actual data transmitted for downloading +-----------------------------+ |1K | 4K | 64K | 1M | +---------------------------------------| | http |67.33 | 49.57 | 44.30 |42.51| |---------------------------------------| | nfs |78.68 | 52.64 | 44.29 |43.80| |---------------------------------------| | iscsi |185.15 | 46.06 | 47.10 |42.30| +---------------------------------------+ Figure 6. Size of actual data transmitted for uploading Zhang, et al. Expires December 27, 2011 [Page 9] Internet-Draft transport protocols comparison June 2011 +-----------------------------+ |1K | 4K | 64K | 1M | +---------------------------------------| | http | 0.59 | 0.81 | 0.90 | 0.94| |---------------------------------------| | nfs | 0.51 | 0.76 | 0.90 | 0.91| |---------------------------------------| | iscsi | 0.22 | 0.87 | 0.85 | 0.95| +---------------------------------------+ Figure 7. Transmission efficiency for downloading (this figure is to be added soon) Figure 8. Transmission efficiency for uploading 4.2. Analysis of Test Results According to the test results under the above circumstances, the three protocols shows comparable performance for data intensive workloads. However, it presents terrible performance on both client and server for meta-data intensive workloads. Besides, the client cpu utilization is much lower for NFS than for the other two protocols when small files are operated. With the increasing number of clients, the performance of NFS server is much poorer than those of HTTP and iSCSI, especially for meta-data intensive workloads. The single embedded client uses similar time to finish the large file upload or download with the three protocols. Theoretically, the performance of NFS ought to be worse than iSCSI for write workloads because NFS employs pseudo- synchronous write policy. NFS specifies a limit on the number of pending writes in the cache. Once this limit is exceeded, the write-back caches degenerates to a write- through cache and application writes see a pseudo-synchronous behavior. However, it appears no significant difference for data- intensive write workloads in our tests. It is noticed that the throughput of the client are all far below the average bandwidth in the lab environment while the client cpu utility is below fifty percent. Thus it can be concluded that the bottleneck of the data intensive workloads in our test lies in the I/O rate of the SD card which covers the negative effect brought by the pseudo-asynchronous write policy. To simulate real mobile phones, SD card is the storage medium in our embedded client. The drawback of SD cards is that its I/O rate is considerable low compared with normal hard disks, which requires the client cpu to wait longer time for the I/O process. The meta-data performance of the NFS protocol suffers primarily because it was designed for sharing of files across clients. Thus, Zhang, et al. Expires December 27, 2011 [Page 10] Internet-Draft transport protocols comparison June 2011 when used in an environment where files are not shared, the protocol pays the penalty of features designed to enable sharing. The unsatisfactory performance of NFS for meta-data intensive application can be attributed to two factors. First, NFS requires clients to update meta- data synchronously to the server which disables update aggregation. Second, although NFS client can also cache meta-data, NFS clients need to perform periodic consistency checks with the server to provide weak consistency guarantees across client machines that share the same NFS namespace. The low client cpu utility for NFS with meta-data intensive application can be attributed to the file system location. In NFS, the file system resides at the server side however in iSCSI it locates at the client side. It is not surprising that the bulk of the meta-data processing is done at the server in the case of NFS while the reserve is true in the case of the iSCSI protocol. As to HTTP which runs at the user space, it consumes more cpu time to copy data between the user space and the kernel space. Thus its client cpu utility is higher than NFS. The server performance for HTTP and iSCSI outperforms that of NFS with the rising number of client ends. The performance for NFS reads is even unsatisfactory. Apart from the conservative meta-data cache policy, the NFS protocol also suffers from the flow-control-free RPC protocol. When the number of packets increases, the competition for the server becomes more fierce and the network becomes more congested, which leads to the increasing packet loss rate on the server as well as the clients. Without flow-control mechanism in RPC, the two parties need to resend the former packets immediately, adding heavier burden on the already- busy network. Thus the server takes longer and longer time to process all the requests from multiple clients. The performance of the NFS server for reading workloads degrades more severely than writing workloads. This may result from the different disk scheduling policy between read and write. When multiple clients download data from the server disk at the same time, the read operations on the server disk are quite random since these files do not reside on neighbouring blocks. In the contrary, when multiple clients upload data onto the server disk, these write operations within a tiny interval may write data onto adjacent disk blocks. As a result, the upload jobs becomes quite I/O sequential. The cpu takes more time to wait random I/O operations than sequential I/O. Thus it takes less time for uploading files than downloading them. Such disk scheduling policy also affects the read and write operations for HTTP and iSCSI. Zhang, et al. Expires December 27, 2011 [Page 11] Internet-Draft transport protocols comparison June 2011 5. Conclusion In the embedded Linux environment, the performance of protocols is almost similar. In the multiple clients environment, HTTP and iSCSI kept up with a much more stable status of throughput as well as less packages overtake than NFS because of the TCP flow control and less data integrity request.HTTP outperformed the other two for transferring big files, and iSCSI is more suitable for meta-data intensive transaction. NFS has more functions like a whole file system. According to the analysis of our test results, the performance of the NFS protocol is affected by the following three factors: (1) to keep data consistency among the server and clients, the cached meta-data must do persistent check every three seconds; (2) NFS disables meta- data update aggregation and requires clients to update meta-data synchronously to the server; (3) NFS is based on the RPC protocol bearing no mechanism to do flow control and congestion handling, which leads to considerable high packet loss rate and re-transmission rate. Influenced by these features, the performance of NFS on both server and client for meta-data intensive workloads is extremely poor. It is concluded that NFS protocol is not suitable to perform as a file storage protocol on local file servers in the 3G edge network proposed in the beginning of this paper due to its bad performance for meta-data intensive operations and terrible support for multiple clients. Through the comparison among NFS, HTTP and iSCSI, it is clear that iSCSI fails to serve the purpose of file sharing since its block-level protocol nature decides its exclusive domination of a certain volume on the disk. HTTP works in the user mode which necessitates data copy between the user mode an the kernel mode and may increase cpu utility. In spite of this, it is designed for simple file sharing with weak demands on data consistency, which makes it more scalable. Besides, HTTP packets run over TCP and uses persistent connection, leading to stable, reliable and less traffic on the network. HTTP is used in wide range of scenarios and is easy to be deployed in physical environment. HTTP has the advantage of accessing to large files and general performance of accessing to small files compared with NFS, iSCSI. Besides, more useful features are expected to be extended to HTTP. Thus, HTTP protocol is promising to function on local file servers in the proposed 3G edge network to relieve the pressure on backbone network. 6. Security Considerations This document does not introduce any new security threats. Zhang, et al. Expires December 27, 2011 [Page 12] Internet-Draft transport protocols comparison June 2011 7. IANA Considerations There is no IANA consideration in this document. 8. Informative References [refs.NFS_Backgrounder] Mckusick, M., Bostic, K., Karels, M., and J. Quarterman, "NFS Backgrounder", Network Appliance. Inc, 1996. [refs.NFS_iSCSI] Radkov, P., Yin, L., Goyal , P., Sarkar, P., and P. Shenoy, "A Performance Comparison of NFS and iSCSI for IP Networked Storage", Proceedings of the 3rd USENIX Conference on File and Storage Technologies, 2004. [refs.SUN_NFS] Sandberg, R., Goldberg, D., Kleiman, S., Walsh, D., and B. Lyon, "Design and Implementation of the Sun Network File System", Proceedings of the Summer 1985 USENIX Conference, 1985. [RFC1813] Callaghan, B., Pawlowski, B., and P. Staubach, "NFS Version 3 Protocol Specification", RFC 1813, June 1995. [RFC2616] Fielding, R., Gettys, J., Mogul, J., Frystyk, H., Masinter, L., Leach, P., and T. Berners-Lee, "Hypertext Transfer Protocol -- HTTP/1.1", RFC 2616, June 1999. [RFC3720] Satran, J., Meth, K., Sapuntzakis, C., Chadalapaka, M., and E. Zeidner, "Internet Small Computer Systems Interface (iSCSI)", RFC 3720, April 2004. [refs.NFS_illustrated] Callaghan, B., "NFS illustrated", Addison-Wesley Longman Ltd., 2000. [refs.Multi] Watson, A., Benn, P., and A. Yoder, "MULTIPROTOCOLDATA ACCESS: NFS,CIDS, AND HTTP", Network Appliance, Inc.. [refs.Comp_Block_File] Rdokov, P., "An Experimental Comparison of Block- and File-Access Protocols for IP-Networked Storage", Proceedings of the Usenix Conference on File and Storage Technologies, 2004. Zhang, et al. Expires December 27, 2011 [Page 13] Internet-Draft transport protocols comparison June 2011 [refs.FS_SB] TRAEGER, A. and E. ZADOK, "A Nine Year Study of File System and Storage Benchmarking", ACM Transactions on Storage, 2008. Authors' Addresses Yuchao Zhang Shanghai Jiaotong University Email: Yao Cui Shanghai Jiaotong University Qianwen Yin Shanghai Jiaotong University Yong Wang Huawei Email: wangyong1@huawei.com Zhang, et al. Expires December 27, 2011 [Page 14]