Internet Engineering Task Force M. Ooki Internet-Draft S. Kamei Intended status: Informational NTT Communications Expires: June 23, 2015 December 20, 2014 Internet Measurement System draft-ooki-lmap-internet-measurement-system-01 Abstract This document describes an experience of Japanese Internet measurement system to measure Internet performance. We have developed the system toward the enhancement of the network performance in an ISP since October 2013. The considerations about the Internet measurement are introduced along with our current status. This document is expected to be useful for the standardization of Internet measurements. Status of This Memo This Internet-Draft is submitted in full conformance with the provisions of BCP 78 and BCP 79. Internet-Drafts are working documents of the Internet Engineering Task Force (IETF). Note that other groups may also distribute working documents as Internet-Drafts. The list of current Internet- Drafts is at http://datatracker.ietf.org/drafts/current/. Internet-Drafts are draft documents valid for a maximum of six months and may be updated, replaced, or obsoleted by other documents at any time. It is inappropriate to use Internet-Drafts as reference material or to cite them other than as "work in progress." This Internet-Draft will expire on June 23, 2015. Copyright Notice Copyright (c) 2014 IETF Trust and the persons identified as the document authors. All rights reserved. This document is subject to BCP 78 and the IETF Trust's Legal Provisions Relating to IETF Documents (http://trustee.ietf.org/license-info) in effect on the date of publication of this document. Please review these documents carefully, as they describe your rights and restrictions with respect to this document. Code Components extracted from this document must include Simplified BSD License text as described in Section 4.e of Ooki & Kamei Expires June 23, 2015 [Page 1] Internet-Draft Internet Measurement System December 2014 the Trust Legal Provisions and are provided without warranty as described in the Simplified BSD License. Table of Contents 1. Introduction . . . . . . . . . . . . . . . . . . . . . . . . 2 2. Motivation of Internet Measurement . . . . . . . . . . . . . 3 3. The framework of Internet Measurement System . . . . . . . . 4 3.1. Measurement Agent . . . . . . . . . . . . . . . . . . . . 4 3.1.1. Specification of the MA . . . . . . . . . . . . . . . 4 3.1.2. Configuration . . . . . . . . . . . . . . . . . . . . 4 3.1.3. Location of the MA . . . . . . . . . . . . . . . . . 5 3.2. Controller Server . . . . . . . . . . . . . . . . . . . . 5 3.2.1. The management of MAs . . . . . . . . . . . . . . . . 6 3.2.2. Setting the Measurement Task and Measurement Schedule 6 3.2.3. Receiving the Requests . . . . . . . . . . . . . . . 7 3.3. Collector Server . . . . . . . . . . . . . . . . . . . . 7 3.4. Architecture . . . . . . . . . . . . . . . . . . . . . . 8 4. The operation of Internet Measurement System . . . . . . . . 9 4.1. Measurement Performance Metrics . . . . . . . . . . . . . 10 4.2. Measurement Target Contents . . . . . . . . . . . . . . . 11 4.3. Measurement Schedule . . . . . . . . . . . . . . . . . . 11 4.4. Application of Measurement . . . . . . . . . . . . . . . 12 5. The issues of Internet Measurement System . . . . . . . . . . 12 5.1. Architecture Issue . . . . . . . . . . . . . . . . . . . 12 5.2. Operation Issue . . . . . . . . . . . . . . . . . . . . . 13 6. Security Considerations . . . . . . . . . . . . . . . . . . . 13 7. IANA Considerations . . . . . . . . . . . . . . . . . . . . . 14 8. References . . . . . . . . . . . . . . . . . . . . . . . . . 14 8.1. Normative References . . . . . . . . . . . . . . . . . . 14 8.2. URL References . . . . . . . . . . . . . . . . . . . . . 14 1. Introduction In Japan, it is common to use a high speed Internet such as 100Mbps and 1Gbps as an ISP's customer connection. Users only know the maximum bandwidth of the last one mile for the ISP connection. The maximum bandwidth value is ranging from 100Mbps to 2Gbps in ISP's price plan as a FTTH connection. Of course the performance of actual Internet connection is below the bandwidth value. Internet users can obtain actual performance depends on various ISP conditions such as congestions. Internet users don't know the performance of the actual network. On the other hand, ISPs also don't know the quality that Internet users experience. For the ISP's point of view, it is important to understand the service quality for its customers in order to design Ooki & Kamei Expires June 23, 2015 [Page 2] Internet-Draft Internet Measurement System December 2014 its network properly. For this reason, it is necessary to measure the actual performance of typical Internet users. The Large-Scale Measurement of Broadband Performance (LMAP) working group is formed to standardize a large scale measurement system to measure broadband network performance. The LMAP WG does not focus on the measurement of the global Internet at the moment. However, we believe that either way someday it will be necessary to establish a method for the Internet measurement and the standardization of the end to end performance measurement, that is not closed to a certain ISP. This document describes the system and our issues for the Internet measurement. In addition, we are planning to propose Internet measurement requirements. Our measurement requirements can be useful for LMAP framework. We have measured the Internet performance by using Internet measurement system we have been operating since 2011. We expect the experience of our case can contribute to the standardizations in LMAP WG. 2. Motivation of Internet Measurement There are three reasons that ISPs want to measure the actual performance of its access services. First, ISPs want to keep the customer satisfaction. Typically ISPs provide the list of maximum bandwidth and the service prices, such as the estimated total fee and the discount rate after the result of a cash back campaign. Japanese users select a ISP based on only those information without knowing the actual Internet performance results. The poor performance causes the lower customer satisfaction. Second, contents providers are beyond the control of ISPs. The traffic volume of Contents Delivery Network (CDN) providers such as AKAMAI and LEVEL3 is increasing in the Internet in recent years. How much users are connecting to which contents providers impact the actual performance. ISPs need to understand their behavior to decide ISP's strategies and operation. Third, we would like to support the public evaluations of ISPs. Some contents service provider e.g., Google or Sandvine[google][sandvine], presented the reports about Internet traffic and ISP performance based on each criterion recently. The Google report presents the results of multiple ISPs measured in for each locations in USA. People in the world can browse the reports on the Internet. These reports will have huge impact on user's choice of the ISP selection. We would like to double-check by using our performance data in order to confirm whether the reports can be reliable or not. If we can Ooki & Kamei Expires June 23, 2015 [Page 3] Internet-Draft Internet Measurement System December 2014 find the difference of the performance results between our data and the reports, we might be able to review whether our measurement methods are mistaken or not. It is also better for an ISP to investigate and comprehend the status of Internet performance between ISPs. So, we have to measure the Internet performance by ourself. Hence, ISPs should measure Internet performances from Internet users to multiple content providers. Then, ISPs can show a performance of the actual network to build brand value compared with other providers. 3. The framework of Internet Measurement System We introduce the framework of Internet Measurement System in this section. The words, such as Measurement Agent, Controller, and Collector conforms to the glossary of the LMAP document[I-D.ietf- lmap-framework]. 3.1. Measurement Agent The MA has the functions that receive instructions from Controller Server (described below), performs measurement tasks, and sends the measurement results to the Collector Server. 3.1.1. Specification of the MA We used a Japanese product, called OpenBlocks [plathome], which is the Linux box with Dual Core Marvell ARMADA XP 1.33GHz, 1GB SDRAM memory. We selected the box as the MA because of the affordable price, software stability, small form factor, flexible functionally, and extendability. The MA needs some CPU power in order to connect PPPoE access line and download tens of contents on the Internet. The OpenBlocks stacks CPU enough to archive them. 3.1.2. Configuration We introduce information configured on the MA in this section. o MA's ID We have to setup the MA's ID. The ID has to be a unique among MAs in order for Controller server to distinguish MAs. The information is described in the "/etc/hostname" on Linux File System. The naming rule is based on the location of MA, the types of line, and the plan, etc. o HTTP Get Tasks Ooki & Kamei Expires June 23, 2015 [Page 4] Internet-Draft Internet Measurement System December 2014 MA automatically gets the measurement tasks from Controller Server every five minutes. MA sends the request about the tasks and schedule to Controller Server by HTTP. o Convert measurement results MA automatically converts from raw data to the type of JSON data. The convert process is executed every one minute. The converted JSON data is written in the measurement result files. An example of JSON data type of the results of wget raw data is below. {"host":"tokyo-xXx01","filename":"tokyo-xXx01_ISP_target_wget_2014120 3235011.log.ok","result":"ok","line":"2","message":"2014-12-03 23:50:18 (10.1 MB/s) - `/dev/null' saved [67206439/67206439]"} o Fluentd The fluentd is an open source data collector, which lets you unify the data collection and consumption for a better use and understanding of data. The above measurement result files are submitted to Collector Server by the process as soon as the files are created. An example of the running flutend process is below. /usr/bin/ruby1.9.1 /usr/local/bin/fluentd --daemon /var/lib/fluent/ fluentd.pid --user fluent --group fluent --config /etc/fluent/ fluent.conf --log /var/log/fluent/fluent.log -vv o Self Check The MAs check whether the above processes for measurement service are down or not on the regular interval. If a process is down, the MA transmits the message about it to Controller Server. 3.1.3. Location of the MA We have distributed MAs on many places all over Japan. The number of locations is approximately 150 in December 2014. The number of our MAs will be increasing in approximately 200 by the next year. MAs are located in houses where the residents can respond our requests (e.g., not turning off the power to constantly perform the measurement) to manipulate the device. 3.2. Controller Server The Controller Server is a Linux server. The Controller Server has the functions to instruct the MAs and receives the HTTP GET requests from MAs. The Controller Server has 3 tables implemented by MySQL for instructing MAs. Ooki & Kamei Expires June 23, 2015 [Page 5] Internet-Draft Internet Measurement System December 2014 3.2.1. The management of MAs The Controller Server manage MAs by using two tables (MA Table and GROUP Table). A example of the MA Table is below. MA_ID is a key identifying MA. TYPE is a kind of network type. MODE expresses the type of the measurement.IF MODE is 0, it means MA is the measurement mode. The mode is the status that MAs are performing a measurement task. If MODE is 1, it means MA is the maintenance mode. The mode is the status that MA stop performing a measurement task. In case of MODE 1, the MA automatically connect to Controller Server by using ssh protocol. We can login to the MA of MODE 1 from Controller Server and change the configuration. We can manage the behavior for MAs by switching the MODE. +-------------+-------+-------+---------+------+--------------------+ | MA_ID | TYPE | AREA | OS_TYPE | MODE | GET_SCHEDULE_TIME | +-------------+-------+-------+---------+------+--------------------+ | tokyo-nFh04 | flets | tokyo | OS_TYPE | 1 | 2014-12-08 | | | | | | | 23:21:00 | +-------------+-------+-------+---------+------+--------------------+ Table 1: MA Table A example of the GROUP Table is below. The GROUP_ID is a key record grouping MAs. MAs are sure to belong one group at least. MA, tokyo- nFh04, belongs to the group-id1. The GROUP_INFO is the remarks. We can set the information of the group which MAs belong in the column +-----------+-------------+------------+ | GROUP_ID | MA_ID | GROUP_INFO | +-----------+-------------+------------+ | group-id1 | tokyo-nFh04 | Group 01 | +-----------+-------------+------------+ Table 2: GROUP Table 3.2.2. Setting the Measurement Task and Measurement Schedule We set the measurement tasks to instruct MAs on the Controller Server. MAs automatically download the task from Controller Server by the fixed time. We need to set a measurement schedule with the measurement task at the same time. A example of the MEASUREMENT SCHEDULE Table is below. SCH_ID is a key record within the table. LINE_TYPE is a type of network provided by a network service. ISP is a Internet service provider to perform the measurement tasks. SCRIPT is the script file of the measurement tasks described by some programming languages. PARAM is a parameter file required for performing measurement tasks. START_TIME is the time when MAs start Ooki & Kamei Expires June 23, 2015 [Page 6] Internet-Draft Internet Measurement System December 2014 performing a measurement task. END_TIME is the time when MAs stop performing a measurement task. +------+--------+--------+----+----------+-------+---------+--------+ | SCH_ | GROUP_ | LINE_T | IS | SCRIPT | PARAM | START_T | END_TI | | ID | ID | YPE | P | | | IME | ME | +------+--------+--------+----+----------+-------+---------+--------+ | 1 | group- | flets | IS | measure1 | param | 00:00:0 | 00:00: | | | id1 | | P1 | .sh | 1 | 0 | 00 | | 2 | group- | flets | IS | measure2 | param | 00:00:0 | 00:00: | | | id2 | | P2 | .sh | 2 | 0 | 00 | +------+--------+--------+----+----------+-------+---------+--------+ Table 3: MEASUREMENT SCHEDULE Table 3.2.3. Receiving the Requests On the Controller Server, a httpd program is running as a daemon that executes continuously in the background to handle requests. The Controller Server returns the appropriate measurement tasks and measurement schedules to MAs in response to HTTP GET requests. The MA which complete own measurement task receives a new measurement task continuously. MAs can start performing the next measurement tasks continuously. 3.3. Collector Server The Collector Server receives the measurement results from MAs through fluentd process. The fluentd process is running as a daemon that executes continuously in the background to handle the measurements data. The detail of the measurement results received by fluentd process is below 20141214230628+0900 measure.tokyo-nFh04 {"host":"tokyo- nFh04","filename":"tokyo-nFh04_ISP_DEST_wget_20141214230450.log.ok"," result":"ok","message":"2014-12-14 23:06:18 (745 KB/s) - `/dev/null' saved [67206439/67206439]"} 20141214230902+0900 measure.tokyo-nFh04 {"host":"tokyo- nFh04","filename":"tokyo-nFh04_ISP_DEST_wget_20141214230731.log.ok"," result":"ok","message":"2014-12-14 23:08:52 (811 KB/s) - `/dev/null' saved [67206439/67206439]"} A example of the directory structure of the stored measurement results is below. A MA's measurement result file is created by the day. /data/MA's-ID/measure_result_MA's-ID.DATETIME.log Ooki & Kamei Expires June 23, 2015 [Page 7] Internet-Draft Internet Measurement System December 2014 When the Collector Server receives the measurement results, the server creates the directory of the MA-ID of MA and the measurement result files. The measurement results are stored in the data directory. 3.4. Architecture The architecture of the measurement system is composed of MAs, Controller Server, and, Collector Server. +----------------+ +----------------+ | Controller | | Collector | | | | | | Server | | Server | +----------------+ +----------------+ ^ ^ | | | Get Measurement Sending | | Tasks and Measurement the Results | | Schedules | | +-------------+ | | | Measurement | | +---------- | | -----------+ | Agents | +-------------+ | ^ Perform Only | | Obtain Active Measurement | | Results v | +---------------------------------------------------+ | | | FTTH Access Line | | | +---------------------------------------------------+ | ^ | | | | v | +---------------------------------------------------+ | | | ISP network | | | +---------------------------------------------------+ | ^ | | | | v | +---------------------------------------------------+ Ooki & Kamei Expires June 23, 2015 [Page 8] Internet-Draft Internet Measurement System December 2014 | | | Internet | | | +---------------------------------------------------+ | ^ | | | | v | +----------------------+ | Target Contents | +----------------------+ Figure 1: Architecture of the Internet Measurement System We need to import Controller Server the record of MA's configuration in MA Table, GROUP Table, and MEASUREMENT SCHEDULE Table before the MA is powerd on. When a MA is powered on, it tries to establish the FTTH access PPPoE connection with the ISP. After obtaining an IP address, it automatically connect the Controller Server and gets the configuration by HTTP. If the value of MODE column in MA Table is 1, the MA automatically gets the maintenance mode. IF the value of that is 0, the MA automatically gets the measurement mode and start downloading the measurement tasks that is configured in MEASUREMENT SCHEDULE Table. The MA prepares for the measurement tasks, performs the tasks for Measurement Target Contents actively, and collects the measurement results. After the completion of the measurement tasks, the MA sends the measurement results to Collector Server using fluentd process. and submits the request for downloading the next measurement tasks to Controller Server using HTTP. When the specification of the LMAP WG's protocol and framework is finished, we will deploy the protocol in our measurement system. 4. The operation of Internet Measurement System We introduce the operation of Internet Measurement System we have been operating since October 2013 in this section. Ooki & Kamei Expires June 23, 2015 [Page 9] Internet-Draft Internet Measurement System December 2014 4.1. Measurement Performance Metrics The MAs perform only active measurements for Target Contents. Examples of the Target Contents include Video Streaming files and OS update files. In our measurements, the measurement performance metrics are below. o Round Trip Time (RTT) This is the response time between the submission of the ICMP echo request packet and the reception of the ICMP echo reply packet. The metric can also be regarded as the round-trip delay time. This is measured by the ping command. We take the min/avg/max time and the loss rate based on measuring this metrics one hundred times by the measurement task. An example of the command MAs execute is below. MA-ID $ ping -i 0.05 -c 100 {contents_ip_address} o Hop Count and Network Path This metric refers to the number of intermediate devices (like routers) through which the data must pass between the MA and the Target Contents. This metric is regarded as the network distance between the MA and the Target Contents. This is measured by the traceroute command. Actually, MA submits the ICMP echo requests three times. Afterwards, by checking the hop counts and network path, we can find the change of the network routing on the Internet. An example of the command MAs execute is below. MA-ID $ traceroute -nIq 3 {contents_ip_address} o Throughput This metric refers to how much data can be transferred from the MA to the Target Contents in a given amount of time. This is regarded as the bandwidth. We can understand how fast we can get the contents the on the network. Currently, this is measured by the wget command. A MA receives URL of the measurement targets and start downloading the contents using HTTP GET. When the download is completed, the value divided the contents size by the download complete time is regarded as the performance metrics of throughput. An example of the command MAs execute is below. MA-ID $ wget -T 300 -dvO /dev/null {wgetopts} {contents_url} In addition to above three performance metrics, we are studying the change of destination IP address of the Internet contents distributed Ooki & Kamei Expires June 23, 2015 [Page 10] Internet-Draft Internet Measurement System December 2014 by some contents service providers. It is important for ISPs to know the mechanism of the contents delivery networks. MAs resolve some FQDN and gets the destination IP address. We are studying the mechanism of the contents delivery networks based on the response results. 4.2. Measurement Target Contents The selection of the Target Contents is important for the Internet measurement; the type, the length, and the number of the contents. We need to measure the representative contents on the Internet. In order to find such contents, we have selected contents based on two viewpoints. One is the volume of transferred data of network traffic in an ISP. We obtained partial traffic data on multiple prefectures in Japan. We selected the Target Contents which were higher in the transferred traffic volume ranking. Examples of such target Contents are Youtube Video Streaming files and Mac OS update file on AKAMAI and so on. Second is the number of access to the contents in the Internet. For example, the portal sites such as Google or Yahoo!, etc. and the shopping sites such as Amazon and iTuens, etc. are always the higher in the number of access to the contents in the Internet. In another viewpoint, we need to change the target contents according to the purpose of analysis. If our purpose is to measure an event traffic, e.g., the download traffic concerning iOS update or the access traffic concerning the special winning sale of the professional baseball team, etc., we need to measure the related contents. 4.3. Measurement Schedule On receiving the measurement tasks from Controller Server, MAs start measurement tasks. MAs used to perform the measurement task by thirty minutes. When the measurement completes, MAs wait the next scheduled time and do not perform the next measurement tasks. On the contrary, when the measurement does not complete before the next scheduled time, the MA kills the measurement process and moves to the next measurement. The current system in that point is flexible because the MA can start the next task as soon as a measurement task is completed. We can collect more kinds of data than before. Ooki & Kamei Expires June 23, 2015 [Page 11] Internet-Draft Internet Measurement System December 2014 4.4. Application of Measurement Using the system we have studied the difference of network performance between Japanese ISPs, based on the combinations of Target Contents, measurement time, and areas. We are going to take the measurement results in consideration to ISP network design and ISP operation as a reference information. We developed the portal site that publishes the analysis of the measurement results. The site is used to improve the quality of daily ISP operation. 5. The issues of Internet Measurement System We introduce the issues of Internet Measurement System we have been operating in this section. The issue section is divided into two parts: Architecture Issue and Operation Issue. 5.1. Architecture Issue o Scalability The Controller Server is connected to receive HTTP GET requests from multiple MAs. This means that the Controller Server needs to process as many HTTP GET requests as the number of MAs. The number of MAs can easily grow beyond the number of HTTP GET requests that a Controller Server can process. If we place hundreds of MAs all over Japan, we will need to improve the scalability of our system. o IPv6 Support IPv6 network is constructed totally independently from IPv4 network. Hence, the performance of the IPv6 network is highly likely different from that of the IPv4 network. Although the IPv6 network is not the majority yet, it is growing. NTT EAST and WEST provided only 2.7% in NGN (Next Generation Network) on December 2013. The rate of IPv6 enabled network in Japan is 27% in June 2014[IPv6-Promotion Council]. NTT EAST and WEST presented the IPv6 support in PPPoE connection on March 2014. All CPE devices for NTT access line already support IPv6 tunneling, allowing users to adapt IPv6 easily. In order to achieve the broad applicability of our measurement results, we will need to investigate the IPv6 performance also. o Data Reliability Ooki & Kamei Expires June 23, 2015 [Page 12] Internet-Draft Internet Measurement System December 2014 We need many kinds of data in order to improve the reliability of data analysis. If there are many kinds of data, the reliability of our analysis results will be improved and the analysis results might be statistically significant. We need to develop the architecture to collect as many different types data as possible. When a MA completed all the instructed measurement tasks, we are creating the measurement system that the MA performs other measurement tasks being high priority as soon as possible when a MA completed all the instructed measurement tasks. 5.2. Operation Issue o Selection of Measurement Target Contents It is difficult to decide what contents should be measured to present the representative performance. There are many kinds of contents on the Internet. This time we have selected the Target Contents based on the volume of transferred data at some points in an ISP. However, there are more metrics to consider, such as the number of accesses to that contents, rather than the transferred volume. Other metrics are not studied in this document. o Stable Operation We had experiences where the measurement results were not sent immediately, and the measurements for some Target Contents were failed. Although the actual causes of these difficulties vary (e.g., accidentally disconnected LAN cable or power cable), we could easily respond to those issues using informations (e.g., time and place) contained in the centralized logs in the Collector Server. Another difficulty is the change in the settings of the contents provider. For example, wget command for a video content has not worked due to a change in a setting in the contents provider. This issue is difficult to tackle and is left for future work. 6. Security Considerations This section describes security consideration for Internet measurement. We placed approximately 150 MAs all over Japan. These MAs may become DDoS attackers by wrong commands from the Controller Server. From this reason, the list of commands MAs can perform should be restricted. And also, the MAs must deny illegal accesses and logins. MAs should permit only access through instruction from the Controller Server. Ooki & Kamei Expires June 23, 2015 [Page 13] Internet-Draft Internet Measurement System December 2014 7. IANA Considerations No need to describe any request regarding number assignment. 8. References 8.1. Normative References [I-D.ietf-lmap-framewark] Eardley, P., Morton, A., Bagnulo, M., Burbridge, T., Aitken, P., and A. Akhter, "A framework for large-scale measurement platforms(LMAP),draft-ietf-lmap-framework-09 (work in progress), December 2014", . 8.2. URL References [sandvine] Sandvine Report, "https://www.sandvine.com/pr/2014/5/14/ sandvine- report-netflix%E2%80%99s-british-invasion.html", . [google] Google Report, "http://www.google.com/get/videoqualityreport/", . [plathome] OpenBlocks, "http://openblocks.plathome.com/products/ax3/", . [IPv6-Promotion-Council] Japanese IPv6-Promotion, "http://v6pc.jp/jp/spread/ipv6spread_02.phtml", . Authors' Addresses Motoyuki Ooki NTT Communications GranPark 16F 3-4-1 Shibaura, Minato-ku, Tokyo 108-8118,Japan EMail: m.ooki@ntt.com Ooki & Kamei Expires June 23, 2015 [Page 14] Internet-Draft Internet Measurement System December 2014 Satoshi Kamei NTT Communications GranPark 16F 3-4-1 Shibaura, Minato-ku, Tokyo 108-8118,Japan EMail: skame@nttv6.jp Ooki & Kamei Expires June 23, 2015 [Page 15]