INTERNET-DRAFT Motoyuki Ooki Satoshi Kamei Intended Status: Informational NTT Communications Expires: January 5, 2014 July 4, 2014 Internet Measurement System draft-ooki-lmap-internet-measurement-system-00 Abstract This document describes an experience of Japanese Internet measurement system to measure Internet performance. We have developed the system toward the enhancement of the network performance in an ISP since October 2013. The considerations about the Internet measurement are introduced along with our current status. This document is expected to be useful for the standardization of Internet measurements. Status of this Memo This Internet-Draft is submitted to IETF in full conformance with the provisions of BCP 78 and BCP 79. Internet-Drafts are working documents of the Internet Engineering Task Force (IETF), its areas, and its working groups. Note that other groups may also distribute working documents as Internet-Drafts. Internet-Drafts are draft documents valid for a maximum of six months and may be updated, replaced, or obsoleted by other documents at any time. It is inappropriate to use Internet-Drafts as reference material or to cite them other than as "work in progress." The list of current Internet-Drafts can be accessed at http://www.ietf.org/1id-abstracts.html The list of Internet-Draft Shadow Directories can be accessed at http://www.ietf.org/shadow.html Copyright and License Notice Copyright (c) 2014 IETF Trust and the persons identified as the document authors. All rights reserved. This document is subject to BCP 78 and the IETF Trust's Legal Ooki, et al. Expires January 5, 2014 [Page 1] INTERNET DRAFT Internet Measurement System July 4, 2014 Provisions Relating to IETF Documents (http://trustee.ietf.org/license-info) in effect on the date of publication of this document. Please review these documents carefully, as they describe your rights and restrictions with respect to this document. Code Components extracted from this document must include Simplified BSD License text as described in Section 4.e of the Trust Legal Provisions and are provided without warranty as described in the Simplified BSD License. Table of Contents 1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . 3 2. Motivation of Internet Measurement . . . . . . . . . . . . . . 3 3. The framework of Internet Measurement System . . . . . . . . . 4 3.1 Measurement Agent . . . . . . . . . . . . . . . . . . . . . 4 3.2 Master Server . . . . . . . . . . . . . . . . . . . . . . . 4 3.3 Architecture . . . . . . . . . . . . . . . . . . . . . . . 5 3.4 Performance Metrics . . . . . . . . . . . . . . . . . . . . 6 4. The Operation of Internet Measurement System . . . . . . . . . 7 4.1 Measurement Target Contents . . . . . . . . . . . . . . . . 7 4.2 Measurement Schedule . . . . . . . . . . . . . . . . . . . 7 4.3 Application of Measurement . . . . . . . . . . . . . . . . 8 5. The problems of Internet Measurement System . . . . . . . . . 8 6. Security Considerations . . . . . . . . . . . . . . . . . . . 9 7. IANA Considerations . . . . . . . . . . . . . . . . . . . . . 9 8. References . . . . . . . . . . . . . . . . . . . . . . . . . . 9 Authors' Addresses . . . . . . . . . . . . . . . . . . . . . . . . 10 Ooki, et al. Expires January 5, 2014 [Page 2] INTERNET DRAFT Internet Measurement System July 4, 2014 1 Introduction In Japan, it is common to use a high speed Internet such as 100Mbps and 1Gbps as an ISP's customer connection. Users only know the maximum bandwidth of the last one mile for the ISP connection. The maximum bandwidth value is ranging from 100Mbps to 2Gbps in ISP's price plan as a FTTH connection. Of course the performance of actual Internet connection is below the bandwidth value. Internet users can obtain actual performance depends on various ISP conditions such as congestions. Internet users don't know the performance of the actual network. On the other hand, ISPs also don't know the quality that Internet users experience. For the ISP's point of view, it is important to understand the service quality for its customers in order to design its network properly. For this reason, it is necessary to measure the actual performance of typical Internet users. The Large-Scale Measurement of Broadband Performance (LMAP) working group is formed to standardize a large scale measurement system to measure broadband network performance. The LMAP WG does not focus on the measurement of global Internet at the moment. However, we believe that either way someday it will be necessary to establish a method for Internet measurement and the standardization of the end to end performance measurement, that is not closed to a certain ISP. This document describes the system and our current status for Internet measurement. We have measured the Internet performance by using Internet measurement system we have been operating since October 2013. We expect the experience of our case can contribute to the standardizations in LMAP. 2. Motivation of Internet Measurement There are two reasons that ISPs want to measure the actual performance of its access services. First, ISPs want to keep the customer satisfaction. Typically ISPs provide the list of maximum bandwidth and the service prices, such as the estimated total fee and the discount rate after the result of a cash back campaign. Japanese users select a ISP based on only those information without knowing the actual Internet performance results. The poor performance causes the lower customer satisfaction. Second, contents providers are beyond the control of ISPs. The traffic volume of Contents Delivery Network (CDN) providers such as Ooki, et al. Expires January 5, 2014 [Page 3] INTERNET DRAFT Internet Measurement System July 4, 2014 AKAMAI and LEVEL3 is increasing in the Internet in recent years. How much users are connecting to which contents providers impact the actual performance. ISPs need to understand their behavior to decide ISP's strategies and operation. Hence, ISPs should measure Internet performances from Internet users to multiple content providers. Then, ISPs can show a performance of the actual network to build brand value compared with other providers. Sandvine and Google corporation presented the reports about Internet traffic and ISP performance based on each criterion recently [Sandvine Report][Google Report]. The Google report presents the results of multiple ISPs measured in for each locations in USA. These reports will have huge impacts on user's choice of the ISP selection. It is better for an ISP to investigate and comprehend the status of Internet performance between ISPs. 3. The framework of Internet Measurement System We introduce the framework of Internet Measurement System in this section. The words, such as Measurement Agent, Controller, and Collector conforms to the glossary of the LMAP document (cite draft- ietf-lmap-framework). 3.1 Measurement Agent The MA has the functions that receive instructions from the Master Server (described below), perform measurement tasks, and send the measurement results to the Master Server. We used a Japanese product, called OpenBlocks [OpenBlocks], which is the Linux box with Dual Core Marvell ARMADA XP 1.33GHz, 1GB SDRAM memory. We selected the box as the MA because of the affordable price, software stability, small form factor, flexible functionally, and extendability. The MA needs some CPU power in order to connect PPPoE access line. Using the box we can perform regular measurements, and can configure tasks and tools easily. We have distributed MAs on many places all over Japan. The number of locations is approximately 130 in June 2014. MAs are located in houses where the residents can respond our requests (e.g., not turning off the power to constantly perform the measurement) to manipulate the device. 3.2 Master Server The Master Server is a Linux server. The Master Server has the Ooki, et al. Expires January 5, 2014 [Page 4] INTERNET DRAFT Internet Measurement System July 4, 2014 functions to instruct the MAs. The Master Server connects to a MA via a VPN connection. If we want to add a particular measurement task to a MA, we can access through VPN and configure the measurement task. Upon completion of the measurement task, the MA returns the results to the Master Server. The Master Server in turn sends the results to the database server at regular intervals. 3.3 Architecture The architecture of the measurement system is composed of Linux box MAs and the Linux Master Server. +-------------+ +------------+ +------------+ | Master | Sync | | | | | | -----> | Database | -----> | Analyzer | | Server | | | | | +-------------+ +------------+ +------------+ ^ | | | Instruction |Send Data |-------++-----------------++--------------------++ || || || |v |v |v +-------------+ +-------------+ +------------+ | Measurement | | Measurement | | Measurement| | Agents | | Agents | | Agents | +-------------+ +-------------+ +------------+ | ^ | ^ | ^ | | Active | | Obtain | | | | Measurement | | Results | | v | v | v | +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~+ | | | FTTH Access Line | | | +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~+ | ^ | | | | v | +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~+ | | | ISP network | | | +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~+ Ooki, et al. Expires January 5, 2014 [Page 5] INTERNET DRAFT Internet Measurement System July 4, 2014 | ^ | | | | v | +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~+ | | | Internet | | | +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~+ | ^ | ^ | | | | | | | | v | v | +--------------------------+ +--------------------------+ | Target Contents | | Target Contents | +--------------------------+ +--------------------------+ Figure 1: Architecture of the Internet Measurement System When a MA is powered on, it tries to establish the FTTH access PPPoE connection with the ISP. After obtaining an IP address, it automatically sets up a VPN Tunnel to the Master Server. The MA prepare for the measurement tasks that is configured beforehand, performs the tasks for Target Contents actively, and collects the measurement results. After the completion of the measurement tasks, the MA sends the measurement results using SSH protocol through the VPN connection. The measurement tasks are performed for each of multiple ISPs. The list of ISPs is pre-configured in a file (called an ISP file). The MA repeatedly connects to an ISP and performs the measurement, for each ISP. The measurement tasks and the ISP file are script files of Shell and Perl programming language. The script files are set up on crontab. The MAs can autonomously perform active measurement tasks without any control sequence from the Master Server. When the specification of the LMAP WG's protocol and framework is finished, we will deploy the protocol in our measurement system. 3.4 Performance Metrics The MAs perform active measurements for Target Contents at a regular interval. Examples of the Target Contents include Video Streaming Ooki, et al. Expires January 5, 2014 [Page 6] INTERNET DRAFT Internet Measurement System July 4, 2014 files and OS update files. The regular interval is currently thirty minutes. In our measurements, performance metrics are below. Round Trip Time (RTT) : This is the time between the submission of the ICMP echo request packet and the reception of the ICMP echo reply packet. This is measured by the ping command. We took the min/avg/max time and the loss rate. The metric can also be regarded as the network latency. Hop Count : This metric refers to the number of intermediate devices (like routers) through which the data must pass between the MA and the Target Contents. This metric is regarded as the network distance between the MA and the Target Contents. This is measured by the traceroute command. Afterwards, by checking the hop counts, we can find the change of the network routing on the Internet. Throughput : This metric refers to how much data can be transferred from the MA to the Target Contents in a given amount of time. This is measured by the wget command. 4. The Operation of Internet Measurement System We introduce the operation of Internet Measurement System we have been operating since October 2013. in this section. 4.1 Measurement Target Contents The selection of the Target Contents is important for the Internet measurement; the type, the length, and the number of the contents. We need to measure the representative contents on the Internet. In order to find such contents, we select contents based on the volume of transferred data of network traffic in an ISP. We obtained partial traffic data on multiple prefectures in Japan. We selected the Target Contents which were higher in the transferred traffic volume ranking. Our Target Contents are Youtube Video Streaming files and Mac OS update file on AKAMAI currently. 4.2 Measurement Schedule MAs start measurement tasks every thirty minutes. The measurements take some time (about five minutes) because the MA connects to the access line through PPPoE authentication, connects a VPN connection to the Master Server, and performs measurement based on our performance metrics on as many Target Contents as pre-configured. When there are many ISPs pre-configured, it will take longer time. We Ooki, et al. Expires January 5, 2014 [Page 7] INTERNET DRAFT Internet Measurement System July 4, 2014 used 300 seconds for wget time out, and 60 seconds for time out of transferring the results (by scp). When the measurement do not complete before the next scheduled time, the MA kills the measurement process and moves to the next measurement. 4.3 Application of Measurement Using the system we have studied the difference of network performance between Japanese ISPs, based on the combinations of Target Contents, measurement time, and areas. We are going to take the measurement results in consideration to ISP network design and ISP operation as a reference information. We developed the portal site that publishes the analysis of the measurement results. The site is used to improve the quality of daily ISP operation. 5. The problems of Internet Measurement System We introduce the problems of Internet Measurement System we have been operating in this section. - IPv6 Support IPv6 network is constructed totally independently from IPv4 network. Hence, the performance of the IPv6 network is highly likely different from that of the IPv4 network. Although the IPv6 network is not the majority yet, it is growing. NTT EAST and WEST provided only 2.7% in NGN (Next Generation Network) on December 2013. The rate of IPv6 enabled network in Japan is 27% in June 2014[IPv6-Promotion Council]. NTT EAST and WEST presented the IPv6 support in PPPoE connection on March 2014. All CPE devices for NTT access line already support IPv6 tunneling, allowing users to adapt IPv6 easily. In order to achieve the broad applicability of our measurement results, we will need to investigate the IPv6 performance also. - Selection of Measurement Target Contents It is difficult to decide what contents should be measured to present the representative performance. There are many kinds of contents on the Internet. This time we have selected the Target Contents based on the volume of transferred data at some points in an ISP. However, there are more metrics to consider, such as the number of accesses to that contents, Ooki, et al. Expires January 5, 2014 [Page 8] INTERNET DRAFT Internet Measurement System July 4, 2014 rather than the transferred volume. Other metrics are not studied in this document. - Scalability The Master Server is connected from multiple MAs via VPN connections. This means that the Master Server needs to hold as many VPN connections as the number of MAs. The number of MAs can easily grow beyond the number of VPN connections that a Master server can hold. If we place hundreds of MAs all over Japan, we will need to improve the scalability of our system. - Stable Operation We had experiences where the measurement results were not sent immediately, and the measurements for some Target Contents were failed. Although the actual causes of these difficulties vary (e.g., accidentally disconnected LAN cable or power cable), we could easily respond to those issues using informations (e.g., time and place) contained in the centralized logs in the Master Server. Another difficulty is the change in the settings of the contents provider. For example, wget command for a video content has not worked due to a change in a setting in the contents provider. This problem is difficult to tackle and is left for future work. 6. Security Considerations This section describes security consideration for Internet measurement. We placed approximately 130 MAs all over Japan. These MAs may become DDoS attackers by wrong commands from the Master Server. From this reason, the list of commands MAs can perform should be restricted. And also, the MAs must deny illegal accesses and logins. MAs should permit only access through VPN connection from the Master Server. 7. IANA Considerations No need to describe any request regarding number assignment. 8. References [Sandvine Report] https://www.sandvine.com/pr/2014/5/14/sandvine- report-netflix%E2%80%99s-british-invasion.html [Google Report] http://www.google.com/get/videoqualityreport/ [OpenBlocks] http://openblocks.plathome.com/products/ax3/ Ooki, et al. Expires January 5, 2014 [Page 9] INTERNET DRAFT Internet Measurement System July 4, 2014 [IPv6-Promotion Council] http://v6pc.jp/jp/spread/ipv6spread_02.phtml Authors' Addresses Motoyuki Ooki NTT Communications 3-4-1 Shibaura, Minato-ku, Tokyo 108-8118, Japan EMail: m.ooki@ntt.com Satoshi Kamei NTT Communications 3-4-1 Shibaura, Minato-ku, Tokyo 108-8118, Japan EMail: skame@nttv6.jp Ooki, et al. Expires January 5, 2014 [Page 10]