Internet DRAFT - draft-white-httpbis-spdy-analysis
draft-white-httpbis-spdy-analysis
Network Working Group G. White
Internet-Draft J-F. Mule
Intended status: Informational D. Rice
Expires: January 7, 2013 CableLabs
July 6, 2012
Analysis of SPDY and TCP Initcwnd
draft-white-httpbis-spdy-analysis-00
Abstract
Making the Internet faster is the goal of many product and service
companies. Many strategies exist, from data caching to packet
switching, performance improvements in Internet browser clients and
server software, without forgetting the transport network scaling
from the fiber core to the access edges.
This report investigates two proposed protocol enhancements aimed at
making the web browsing user experience better by optimizing the
transport of web objects and thereby reducing the page load time.
The two enhancements are SPDY, a replacement of the HyperText
Transfer Protocol (HTTP) requiring client and server changes, and a
TCP tune-up, an increase in the TCP initial congestion window
accomplished by a setting change on the web servers. Both show
promise, but there are some caveats, particularly with SPDY. SPDY is
a candidate for standardization as HTTP/2.0.
In addition to a discussion of the two enhancements, this report
provides the results of laboratory testing on SPDY version 2 and the
proposed increase in the TCP initial congestion window in a variety
of simulated conditions, comparing the page load time when using the
proposed enhancement to the page load time for the default case of
HTTPS and current initial congestion window settings.
The proposed enhancements generate mixed results: web page load times
were reduced in some scenarios but increased significantly in others.
The performance improvement (or degradation) varied depending on the
number of servers, configuration of the initial TCP congestion
window, and especially any network packet loss. The following
results were obtained across all scenarios comparing SPDY and
congestion window enhancements to standard HTTPS.
o Average reduction in page load time was 29%
o Best improvement was over 78% reduction in page load time
White, et al. Expires January 7, 2013 [Page 1]
Internet-Draft spdy-analysis July 2012
o Worst cases showed a negative impact, resulting in a 3.3x increase
in page load time
These results lead us to the following conclusions:
o The SPDY protocol is currently a moving target, and thus it would
be challenging to realize a return on investment for general-
purpose usage in web servers.
o Protocol improvements, standardization in the IETF and wider
adoption by the client/server software may warrant a second look
at SPDY.
o Some applications in controlled environments may gain by
leveraging SPDY. SPDY might be a valuable tool where the a single
entity provides the servers, the client software, and the web
content.
o If SPDY were adopted very widely it may have some secondary
benefits for network operators through improved infrastructure
scalability due to a significant reduction in concurrent TCP
sessions, as well as a reduction in Packets Per Second.
o The proposed increase in the TCP initial congestion window is
straightforward, requires no client modifications, and on its own
provides consistent (albeit modest) performance gains.
This report is available in a somewhat more detailed form in
[SPDY-ANALYSIS].
Status of this Memo
This Internet-Draft is submitted in full conformance with the
provisions of BCP 78 and BCP 79.
Internet-Drafts are working documents of the Internet Engineering
Task Force (IETF). Note that other groups may also distribute
working documents as Internet-Drafts. The list of current Internet-
Drafts is at http://datatracker.ietf.org/drafts/current/.
Internet-Drafts are draft documents valid for a maximum of six months
and may be updated, replaced, or obsoleted by other documents at any
time. It is inappropriate to use Internet-Drafts as reference
material or to cite them other than as "work in progress."
This Internet-Draft will expire on January 7, 2013.
Copyright Notice
White, et al. Expires January 7, 2013 [Page 2]
Internet-Draft spdy-analysis July 2012
Copyright (c) 2012 IETF Trust and the persons identified as the
document authors. All rights reserved.
This document is subject to BCP 78 and the IETF Trust's Legal
Provisions Relating to IETF Documents
(http://trustee.ietf.org/license-info) in effect on the date of
publication of this document. Please review these documents
carefully, as they describe your rights and restrictions with respect
to this document. Code Components extracted from this document must
include Simplified BSD License text as described in Section 4.e of
the Trust Legal Provisions and are provided without warranty as
described in the Simplified BSD License.
White, et al. Expires January 7, 2013 [Page 3]
Internet-Draft spdy-analysis July 2012
Table of Contents
1. Introduction . . . . . . . . . . . . . . . . . . . . . . . . . 5
2. SPDY . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 6
2.1. The SPDY Protocol . . . . . . . . . . . . . . . . . . . . 7
2.2. SPDY Server Implementations . . . . . . . . . . . . . . . 9
2.3. SPDY Client Implementations . . . . . . . . . . . . . . . 9
2.4. SPDY in the Wild . . . . . . . . . . . . . . . . . . . . . 9
2.5. SPDY Protocol Development and IETF Standardization . . . . 9
3. TCP Initial Congestion Window . . . . . . . . . . . . . . . . 10
3.1. Historical Settings for initcwnd . . . . . . . . . . . . . 10
4. The Current Web Environment . . . . . . . . . . . . . . . . . 11
4.1. Content Environment . . . . . . . . . . . . . . . . . . . 11
4.2. Network Environment . . . . . . . . . . . . . . . . . . . 13
4.2.1. Round-Trip Time (RTT) . . . . . . . . . . . . . . . . 13
4.2.2. Access Network Data Rate . . . . . . . . . . . . . . . 15
4.2.3. Packet Loss . . . . . . . . . . . . . . . . . . . . . 15
5. Laboratory Testing and Results . . . . . . . . . . . . . . . . 16
5.1. Goals . . . . . . . . . . . . . . . . . . . . . . . . . . 16
5.2. Laboratory Environment . . . . . . . . . . . . . . . . . . 16
5.2.1. Web Server Configuration . . . . . . . . . . . . . . . 16
5.2.2. Client Configurations . . . . . . . . . . . . . . . . 17
5.3. Test Conditions . . . . . . . . . . . . . . . . . . . . . 17
5.3.1. Protocol Options . . . . . . . . . . . . . . . . . . . 17
5.3.2. Web Sites . . . . . . . . . . . . . . . . . . . . . . 18
5.4. Test Results . . . . . . . . . . . . . . . . . . . . . . . 20
5.4.1. CASE 1: SPDY vs. HTTPS . . . . . . . . . . . . . . . 21
5.4.2. CASE 2: initcwnd=10 vs. initcwnd=3 . . . . . . . . . 24
5.4.3. CASE 3: SPDY+initcwnd=10 vs. HTTPS+initcwnd=3 . . . . 27
5.4.4. Results Summary . . . . . . . . . . . . . . . . . . . 29
6. Recommendations and Conclusions . . . . . . . . . . . . . . . 30
7. IANA Considerations . . . . . . . . . . . . . . . . . . . . . 31
8. Security Considerations . . . . . . . . . . . . . . . . . . . 31
9. Acknowledgements . . . . . . . . . . . . . . . . . . . . . . . 31
10. References . . . . . . . . . . . . . . . . . . . . . . . . . . 32
Authors' Addresses . . . . . . . . . . . . . . . . . . . . . . . . 33
White, et al. Expires January 7, 2013 [Page 4]
Internet-Draft spdy-analysis July 2012
1. Introduction
The current method of transporting web objects from a server to a
client utilizes Hypertext Transfer Protocol version 1.1 (HTTP/1.1),
running atop the Transport Control Protocol (TCP) [RFC0793].
HTTP/1.1 was published as [RFC2616] in 1999, and has since become the
most widely used application-layer protocol on the Internet. TCP
pre-dates HTTP/1.1 by about 18 years, and even though TCP has evolved
over time, it is fundamentally unchanged, and runs atop IPv4 and
IPv6.
Since the advent of the World Wide Web 15+ years ago, access network
bandwidths as well as server and client CPU horsepower have increased
by a factor of approximately 1.5x year-over-year, yet the apparent
speed of the web (from the web browsing user's perspective) has grown
much more slowly. In part, this can be explained by a concomitant
increase in web page complexity, as measured by any number of
attributes including total page size, number of linked resources,
amount of server-side code, and lines of JavaScript. However, there
is a view that the underlying protocols used to transport web
resources are becoming increasingly out-of-date with the network and
computing resources that are in use today, and that significant
improvements in performance (generally page load time) can be
achieved by revisiting these fundamental technologies.
Bolstering this view is the fact that while network bandwidths have
been improving rapidly, network latencies have not, and there is
little hope of achieving significant reductions in network latency,
as it is dominated by propagation delay. The result is that a
fundamental network parameter, the Bandwidth-Delay Product (BDP), is
much larger today than in the early days of the Internet. It is
becoming clear that HTTP/1.1 and TCP are not particularly optimized
for networks with high bandwidth-delay product.
Furthermore, in order to maximize performance using the existing
tools of HTTP/1.1 and TCP, web architects (browser, server and site
developers) have employed work-arounds that have some negative
implications. The most significant implications result from the use
of multiple simultaneous TCP connections. Modern browsers will open
up to six simultaneous TCP connections to each server from which they
need to retrieve resources, and optimized websites will compound this
by spreading content over multiple servers, a practice known as
"domain sharding". The result is that a single browser may have 20
or more simultaneous TCP connections to the hosts of a particular
site while it is downloading a single web page. The parallel nature
of this approach is intended to reduce the page download time
(compared to the alternative of a single, non-pipelined TCP
connection), and in many cases it succeeds. However, a web browser
White, et al. Expires January 7, 2013 [Page 5]
Internet-Draft spdy-analysis July 2012
with 20 or more simultaneous TCP sessions can drown out other
applications that use a small number of TCP sessions (e.g., one).
This is due to the fact that the TCP congestion avoidance algorithm
provides approximately fair sharing of the bandwidth on the
bottleneck link on a per-TCP-session basis. Additionally, some
browsers do not reuse TCP connections, so each web resource is
retrieved using a separate TCP connection. The result is that the
vast majority of sessions never reach the congestion avoidance phase,
and network resources can therefore either be underutilized or
excessively congested. The network overhead for establishing and
tearing down all of the TCP connections can also be a concern.
A number of efforts have sprung up to develop ways to improve page
download time and consequently improve the web browsing user
experience. This paper discusses two proposed enhancements, both
originally proposed by Google as part of their "Let's Make the Web
Faster" project [Faster]. The first is a replacement for HTTP 1.1,
called SPDY [SPDY], which aims to make more efficient use of the
network in common web browsing scenarios. The second is an
incremental enhancement to the TCP protocol in the form of an
increase in the server's initial congestion window (initcwnd).
In addition to a discussion of the two proposed enhancements, this
paper presents the results of laboratory testing conducted by
CableLabs on both technologies independently, and on the combination
of the two technologies.
2. SPDY
SPDY [SPDY] is an experimental protocol developed by Mike Belshe and
Roberto Peon of Google in late 2009 to replace HTTP/1.1 communication
for transporting web content between a client and a server. It is
deployed in production on many Google servers but requires a
compatible browser such as Google Chrome. The stated goals of SPDY
are:
o 50% reduction in page load time
o Minimize deployment complexity
o Avoid changes to content by web authors
o Solve this collaboratively via open-source software
At the time of this study, the current implemented version of SPDY is
version 2 (SPDY/2). Version 3 is currently being developed.
White, et al. Expires January 7, 2013 [Page 6]
Internet-Draft spdy-analysis July 2012
2.1. The SPDY Protocol
SPDY/2 retains the HTTP/1.1 metadata, but replaces the transport
aspects of HTTP with a more streamlined approach. SPDY has three
basic features that are used to accelerate the loading of a page:
o Multiplexed streams - The fundamental enhancement of SPDY is that
multiple resources can be retrieved via a single TCP connection.
The expectation (and current implementation in Chrome) is for the
client to open a single TCP connection to each server, and to
request all resources of the server over that single connection.
The use of a single TCP connection in this way allows the
congestion avoidance algorithm in TCP to more effectively manage
data flow across the network. Also, the client can include
multiple requests in a single message, thereby reducing overhead
and the number of round-trip times necessary to begin file
transfer for requests beyond the first. While this is similar to
HTTP pipelining [HTTP_Pipelining], the difference introduced by
SPDY is that the server can transfer all of the resources in
parallel (multiplexed) without the "head-of-line blocking" problem
that can occur with HTTP pipelining.
o Request prioritization - While SPDY may serve to decrease total
page load time, one side-effect of stream multiplexing is that a
critical page resource might be delivered more slowly due to the
concurrent delivery of a less critical resource. To prevent this,
SPDY provides a way for the client to indicate relative priorities
for the requested resources. For example, if a JavaScript library
is required in order to generate URLs for additional page
resources, the client can request that the library be delivered
with higher priority so that it isn't holding up those subsequent
requests.
o HTTP header compression - SPDY mandates that HTTP headers be
compressed to reduce the number of bytes transferred. HTTP Header
compression has been an Internet standard but it is not widely
used. Google SPDY makes it mandatory. This might be a fairly
small gain on the response headers, but it can provide a
significant benefit on the requests (which in many cases are
entirely headers). For example, each request from a particular
client to a particular server includes an identical user-agent
string (which can be in excess of 100 bytes), and in some cases
the same cookie is sent in each request. Both Chromium and
Firefox developers have reported approximately 90% header
compression using the proposed zlib compression. This could allow
a lot more requests to be packed into each request packet.
SPDY also has two advanced features that can be used in certain cases
White, et al. Expires January 7, 2013 [Page 7]
Internet-Draft spdy-analysis July 2012
to further accelerate the web user experience:
o Server Push - SPDY allows the server to create a stream to the
client, and via this stream send web resources that were not
explicitly requested by the client. The expectation is that the
client will cache the pushed resource, and then upon needing it,
will retrieve it from local cache. This function could
potentially be used by the server to push objects (of which the
client isn't yet aware) that are necessary for rendering the
current page. Additionally, this function could be used to push
resources for related pages that the user may be likely to
request. The heuristics used by the server to decide when/if to
push objects and what those objects are is left to the server
implementer. This feature is somewhat controversial, but the
authors defend it by pointing out that it is better than the
practice of "in-lining" resources into the html page, since it
allows the pushed resources to be cached for multiple future
usages.
o Server Hint - SPDY defines a new header that the server can use to
suggest additional resources that the client should request. This
is a less forceful approach than the Server Push, and allows the
client to be involved in the decision whether or not a particular
resource is delivered. The client might, for example, examine its
own cache and only request resources that are not resident in
local cache. On the other hand, Server Hint theoretically
requires one additional round trip that would be eliminated by
Server Push.
While it isn't fundamentally required for the protocol to function,
in practice SPDY/2 runs only over an encrypted (TLS) connection. The
rationale for this is three-fold:
o TLS involves a client-server handshake to negotiate cipher-suite.
The authors of SPDY extend this handshake to negotiate whether
SPDY can be used. This Next Protocol Negotiation (NPN)
[I-D.agl-tls-nextprotoneg] allows the use of the https:// URI,
rather than a new spdy:// URI, and as a result a single html page
can work for clients that support SPDY and clients that don't.
o TLS passes through firewalls and bypasses intermediaries. Many
HTTP requests today are processed (and modified) by transparent
proxies without the knowledge of the end-user. It would create a
tremendous barrier to adoption of SPDY if it were necessary for
intermediaries to be upgraded to handle the new protocol.
o Encryption is good. The authors of SPDY state a philosophical
belief that all HTTP traffic should be encrypted. They cite the
White, et al. Expires January 7, 2013 [Page 8]
Internet-Draft spdy-analysis July 2012
relative ease by which traffic (particularly Wi-Fi traffic) can be
snooped.
2.2. SPDY Server Implementations
There are a number of SPDY server implementations that are in various
stages of development as open source projects. The Chromium project
maintains a list of the implementations available [SPDY]. At the
time this study was initiated, many of these implementations
supported a subset of SPDY functionality and/or supported SPDY
version 1. At the time this study began, only two implementations
appeared to support SPDY/2 in a reliable way. The first is the
Chromium "FLIP Server" (FLIP was an early internal code-name for SPDY
within Google). This server is built off of the immense Chromium
source tree. The second is a Javascript implementation called "node-
spdy" which is built on the node.js server framework. There is also
an Apache module for SPDY, but at the time this study was initiated
it was incomplete.
2.3. SPDY Client Implementations
On the client side, both the Chrome browser and the Mozilla Firefox
browser support SPDY/2. Chrome also includes a built-in diagnostics
function that allows the user to examine the active SPDY sessions
<chrome://net-internals/#spdy>. Finally, the Silk browser in the
Amazon Kindle Fire tablet computer purportedly utilizes SPDY/2 for
its connection to the Amazon EC2 cloud when performing web
acceleration.
2.4. SPDY in the Wild
In terms of live web content, the most significant, publicly
available sites that serve content via SPDY are run by Google. Many
of the Google properties have SPDY/2 enabled, including Gmail, Google
Docs, Picasa, Google+, and Google Encrypted Search. All of these
sites utilize only the basic SPDY features; there are no known live
instances of Server Push or Server Hint. In addition, the Twitter
and Wordpress websites utilize SPDY, and the web acceleration
companies Cotendo (acquired by Akamai in Dec. 2011) and Strangeloop
indicate that they have deployed SPDY in some capacity.
2.5. SPDY Protocol Development and IETF Standardization
Google has actively solicited input on enhancements to the SPDY/2
protocol. Up until very recently, development and discussion of
SPDY/3 has taken place on an open, Google-managed forum. However,
the SPDY/3 draft was submitted to the IETF HTTPbis working group on
February 23, 2012, for comments [I-D.mbelshe-httpbis-spdy].
White, et al. Expires January 7, 2013 [Page 9]
Internet-Draft spdy-analysis July 2012
3. TCP Initial Congestion Window
The TCP initial congestion window (initcwnd) is used at the start of
a TCP connection. In the context of an HTTP session, the server's
initcwnd setting controls how many data packets will be sent in the
first burst of data from the server. It is a standard protocol
parameter that can be changed on Linux servers via a simple command
line.
Absent packet loss or receiver window limits, the TCP slow start
operation looks like:
Table 1. TCP Slow Start Operation
+--------------+---------------------------+-------------------------+
| Round-Trip # | Client | Server |
+--------------+---------------------------+-------------------------+
| 1 | TCP SYN | TCP SYN/ACK |
+--------------+---------------------------+-------------------------+
| 2 | TCP ACK and HTTP GET <url>| initcwnd data packets |
+--------------+---------------------------+-------------------------+
| 3 | TCP ACKs | 2*initcwnd data packets |
+--------------+---------------------------+-------------------------+
| 4 | TCP ACKs | 4*initcwnd data packets |
+--------------+---------------------------+-------------------------+
| | | and so on.... |
+--------------+---------------------------+-------------------------+
3.1. Historical Settings for initcwnd
A larger value for initcwnd will clearly result in fewer Round-Trip
Times (RTTs) to deliver a file. However, the downside to an
excessively large initcwnd is that there is an increased risk of
overflowing a router buffer on an intermediate hop, resulting in an
increase in latency from packet loss and retransmissions. A value of
initcwnd that is greater than the Bandwidth-Delay Product (BDP) of
the network path between the server and client has an increased
likelihood of causing packet loss that may lead to poor performance.
As network bandwidths have increased over time, the BDP has
increased, and as a result the defined value for initcwnd has been
adjusted. The early specifications for TCP ([RFC1122], [RFC2001])
required that a TCP implementation set initcwnd to 1 packet.
Starting in 2002 ([RFC2414], [RFC3390]), the initcwnd value was
raised to 4000 bytes (effectively 3 packets in most networks).
Google has submitted an IETF draft [I-D.ietf-tcpm-initcwnd] proposing
that initcwnd now be increased to at least 10 packets, based on a
series of tests performed using production Google servers
White, et al. Expires January 7, 2013 [Page 10]
Internet-Draft spdy-analysis July 2012
[INITCWND-PAPER]. In Google's tests, a value of 10-16 packets
resulted in the minimum average latency for delivery of web
resources.
In November 2011, CDN Planet [CDNPLANET] performed an assessment of
CDNs to see what value is currently used for initcwnd. Many CDNs
have already increased their initcwnd beyond 3 packets, some as high
as 10-16 packets.
This increase from a value of 3 to a value of 10 will result in the
elimination of up to 2 RTTs for each file transfer. The maximum gain
will be seen for files that take 9 packets to deliver (i.e., files
around 11 kB in size). Files of that size would take 4 RTTs to
deliver using the default initcwnd, but can be delivered in 2 RTTs
with the proposed increase, a 50% improvement. Files smaller than
about 3 kB will not experience any acceleration.
4. The Current Web Environment
4.1. Content Environment
The construction of web page content plays a very important role in
determining the page load time. In order to validate the two
proposed enhancements, we will create a test bed that consists of web
servers, web clients and web content. We performed a survey of some
popular (and resource heavy) websites in order to understand the
content environment. The sites used in the survey were selected as
examples of sites that would stand to benefit from an acceleration
technology such as SPDY. Many of these sites are composed of a
fairly large number of resources, the content changes frequently
(reducing local cache acceleration), and they would presumably be
interested in page load performance.
The results shown in Table 2 indicate, for each site's home page, the
number of servers from which content was drawn, the number of
resources requested, the total transferred size of all page
resources, and the page load time. The reported page load time is
the total time to download all of the resources for the page. It is
possible, and even likely, that one or more of these pages are
largely complete (perhaps only missing some non-critical resources)
in significantly less time than is reported here.
Additionally, for each webpage we evaluated the percentage of total
resources that were requested from the top N servers for the page,
for values of N between 1 and 5. For example, Engadget receives 34%
of the web objects from the most used server, 82% of the objects from
3 most used servers and 87% of them come from the 5 servers with the
White, et al. Expires January 7, 2013 [Page 11]
Internet-Draft spdy-analysis July 2012
highest count. These servers may include the host servers for the
website, CDN servers for images and media content, advertising
servers for ad content, and analytics collection, among other types
of resources.
Table 2. Website Survey
+-------------+-------+-----+------+-------+------------------------+
| | | |total | | % of GETs from |
| | | |page | total | top N servers |
| | | |size | time +----+----+----+----+----+
|Page |servers|GETs |(KB) | (s) | N=1| N=2| N=3| N=4| N=5|
+-------------+-------+-----+------+-------+----+----+----+----+----+
|Engadget | 26 | 278 | 1500 | 23.0 | 34%| 67%| 82%| 85%| 87%|
+-------------+-------+-----+------+-------+----+----+----+----+----+
|NY Times | 27 | 148 | 1500 | 13.6 | 33%| 49%| 59%| 67%| 73%|
+-------------+-------+-----+------+-------+----+----+----+----+----+
|Gizmodo | 25 | 116 | 3900 | 13.2 | 34%| 60%| 69%| 76%| 78%|
+-------------+-------+-----+------+-------+----+----+----+----+----+
|CNN | 22 | 158 | 1180 | 12.6 | 59%| 70%| 75%| 80%| 83%|
+-------------+-------+-----+------+-------+----+----+----+----+----+
|comcast.net | 12 | 55 | 606 | 12.0 | 51%| 67%| 75%| 80%| 84%|
+-------------+-------+-----+------+-------+----+----+----+----+----+
|ESPN | 26 | 144 | 5400 | 11.6 | 40%| 51%| 60%| 67%| 70%|
+-------------+-------+-----+------+-------+----+----+----+----+----+
|Amazon | 15 | 139 | 1100 | 11.0 | 34%| 58%| 81%| 86%| 90%|
+-------------+-------+-----+------+-------+----+----+----+----+----+
|RoadRunner | 30 | 128 | 85 | 9.50 | 27%| 48%| 61%| 65%| 68%|
+-------------+-------+-----+------+-------+----+----+----+----+----+
|Wired | 32 | 130 | 1200 | 8.90 | 48%| 56%| 61%| 65%| 68%|
+-------------+-------+-----+------+-------+----+----+----+----+----+
|eBay | 14 | 53 | 520 | 8.60 | 23%| 40%| 55%| 68%| 75%|
+-------------+-------+-----+------+-------+----+----+----+----+----+
|LinkedIn | 9 | 75 | 457 | 7.17 | 48%| 80%| 88%| 92%| 95%|
+-------------+-------+-----+------+-------+----+----+----+----+----+
|Hulu | 12 | 193 | 2000 | 4.96 | 60%| 79%| 87%| 95%| 96%|
+-------------+-------+-----+------+-------+----+----+----+----+----+
|Yahoo | 6 | 51 | 423 | 4.60 | 88%| 92%| 94%| 96%| 98%|
+-------------+-------+-----+------+-------+----+----+----+----+----+
|YouTube | 6 | 37 | 2400 | 4.16 | 41%| 68%| 81%| 92%| 97%|
+-------------+-------+-----+------+-------+----+----+----+----+----+
|Google | 3 | 20 | 73 | 2.45 | 45%| 90%|100%|100%|100%|
| shopping | | | | | | | | | |
+-------------+-------+-----+------+-------+----+----+----+----+----+
|Average | 18 | 115 | 1490 | 9.82 | 44%| 65%| 75%| 81%| 84%|
+-------------+-------+-----+------+-------+----+----+----+----+----+
White, et al. Expires January 7, 2013 [Page 12]
Internet-Draft spdy-analysis July 2012
4.2. Network Environment
The network environment between the servers and the client plays an
equally large role in determining page load time. The most important
parameters that drive performance are the round-trip time, the
bottleneck link bandwidth, and the packet loss rate.
4.2.1. Round-Trip Time (RTT)
In the absence of congestion, typical Internet round-trip times can
range from as low as 10 milliseconds when accessing content served in
close proximity, to as high as 500 milliseconds or more when
accessing servers across continents. For wireline broadband
customers in the U.S., the most popular websites can generally be
reached with RTTs ranging from 15 ms to 150 ms. For the websites
included in the survey in Section 4.1, a sample TCP RTT to each of
the top-five servers (in terms of number of resources requested) from
CableLabs headquarters in Louisville, Colorado, is shown (in
milliseconds) in Table 3 along with a summary of the minimum, mean,
and max RTT of those top-five servers. The RTT was calculated by
doing a TCP "ACK RTT" analysis of a packet capture of each page load
using the Wireshark tool. The ISP connection used during this
testing consisted of two load-balanced 100 Mbps duplex links with a
mean RTT of less than 2 ms.
White, et al. Expires January 7, 2013 [Page 13]
Internet-Draft spdy-analysis July 2012
Table 3. Round-Trip Times to Website Servers in Milliseconds
+-----------+------+------+------+------+------+-----+-------+-----+
|Page |server|server|server|server|server| | | |
| | 1 | 2 | 3 | 4 | 5 | min |average| max |
+-----------+------+------+------+------+------+-----+-------+-----+
|Engadget | 63* | 62* | 64* | 62* | 38 | 38 | 57.8 | 64 |
+-----------+------+------+------+------+------+-----+-------+-----+
|NY Times | 64* | 64* | 62* | 64* | 64* | 62 | 63.6 | 64 |
+-----------+------+------+------+------+------+-----+-------+-----+
|Gizmodo | 37* | 39* | 134 | 25 | 22 | 22 | 51.4 | 134 |
+-----------+------+------+------+------+------+-----+-------+-----+
|CNN | 46* | 53 | 69* | 32 | 45 | 32 | 49.0 | 69 |
+-----------+------+------+------+------+------+-----+-------+-----+
|Comcast.net| 65* | 69* | 94 | 28 | 71 | 28 | 65.4 | 94 |
+-----------+------+------+------+------+------+-----+-------+-----+
|ESPN | 64* | 33 | 65* | 64* | 64* | 33 | 58.0 | 65 |
+-----------+------+------+------+------+------+-----+-------+-----+
|Amazon | 35 | 72* | 16* | 98 | 33 | 16 | 50.8 | 98 |
+-----------+------+------+------+------+------+-----+-------+-----+
|RoadRunner | 56 | 58 | 29* | 52* | 61* | 29 | 51.2 | 61 |
+-----------+------+------+------+------+------+-----+-------+-----+
|Wired | 70* | 114 | 66* | 24* | 85 | 24 | 71.8 | 114 |
+-----------+------+------+------+------+------+-----+-------+-----+
|eBay | 62* | 75 | 64* | 62* | 62 | 62 | 65.0 | 75 |
+-----------+------+------+------+------+------+-----+-------+-----+
|LinkedIn | 50* | 72* | 78 | 54 | 63* | 50 | 63.4 | 78 |
+-----------+------+------+------+------+------+-----+-------+-----+
|Hulu | 65* | 67* | 66* | 64* | 22 | 22 | 56.8 | 67 |
+-----------+------+------+------+------+------+-----+-------+-----+
|Yahoo | 28 | 27 | 64 | 29 | 61 | 27 | 41.8 | 64 |
+-----------+------+------+------+------+------+-----+-------+-----+
|YouTube | 23 | 25 | 23 | 22 | 22 | 22 | 23.0 | 25 |
+-----------+------+------+------+------+------+-----+-------+-----+
|Google | 34 | 42 | 43 | 22 | - | 22 | 35.3 | 43 |
| shopping | | | | | | | | |
+-----------+------+------+------+------+------+-----+-------+-----+
* denotes a host run by a CDN provider.
When network links are congested, a phenomenon referred to as
"bufferbloat" can result in an increase in RTT on the order of
hundreds of milliseconds. Bufferbloat arises due to the fact that
many network elements support much more buffering than is needed to
maintain full link utilization. These oversized buffers will be kept
full by the TCP (particularly when there are multiple simultaneous
sessions), thereby causing an increase in latency. In recent updates
to the DOCSIS 3.0 cable modem specification, a new feature has been
White, et al. Expires January 7, 2013 [Page 14]
Internet-Draft spdy-analysis July 2012
added to mitigate this effect by proper sizing of the cable modem's
upstream buffer. At this time, 14 cable modem models from 10
manufacturers have been certified as supporting this feature. For
more detail on bufferbloat and the DOCSIS Buffer Control feature,
see: [BUFFERCONTROL].
4.2.2. Access Network Data Rate
Residential access network data rates have steadily risen over time,
with a 10 Mbps downstream rate and 1 Mbps upstream rate being fairly
common in North America, and 20 Mbps x 2 Mbps rates becoming
increasingly available.
As a result, for purposes of our investigation of the two proposed
technologies, we will utilize the 10x1 and 20x2 data rate
configurations.
Some access networks, in particular cable modem access networks, are
able to provide higher burst data rates that go well beyond the
sustained traffic rate for the connection. In this report we will
not attempt to study the impact that these "Powerboost" type features
have on the two proposed technologies. In addition, many operators
offer higher speed tiers than those mentioned above, with speeds in
Europe up to 360 Mbps on the very high end, but they currently have
fairly low take rates. Finally, some access networks may have lower
speed limitations placed on the services such as public and outdoor
Wi-Fi networks.
4.2.3. Packet Loss
It is important to note that packet loss due to router (or switch)
buffer overflow is expected behavior in networks, particularly when
TCP is utilized. In fact, packet loss due to buffer overflow is
fundamental to the congestion avoidance algorithm in TCP; it is the
only signal that the TCP has (absent the Explicit Congestion
Notification field in IP packets) that it has saturated the link and
needs to reduce its transmission rate. As a result, it is not a
concern in this testing. The concern here is random packet loss due
to noise, interference or network faults that will effectively send
TCP an erroneous signal to reduce its transmission rate.
In wired networks, packet loss due to noise is fairly uncommon, for
example with packet loss rates of approximately 10^-5 being typical
for DOCSIS cable modem networks. In wireless networks, packet loss
can vary dramatically as a result of interference and fluctuations in
carrier-to-noise ratio.
White, et al. Expires January 7, 2013 [Page 15]
Internet-Draft spdy-analysis July 2012
5. Laboratory Testing and Results
5.1. Goals
The goals of the testing are to investigate the page load performance
impacts of the two proposed technologies. In particular, we examine
Google SPDY with an eye toward evaluating SPDY's ability to achieve
the stated performance goal and to understand what aspects of SPDY
provide acceleration and in which scenarios. Testing is limited to
the SPDY "basic" features; no testing of server-push or server-hint
is included in this study, primarily due to the fact that the
performance enhancement gained by using them would be heavily
dependent on how they are configured and utilized by the website
administrator. Some benchmarks of server-push and server-hint have
been published by external sources; see [GOOGLE-BENCHMARKS] . The
results have not shown a statistically significant gain.
As stated previously, the TCP initcwnd increase is tested both
independently (using HTTPS) and in conjunction with SPDY.
5.2. Laboratory Environment
The entire test network was virtualized on a dedicated VMWare ESX
Virtual Server. The test network consisted of 7 virtual hosts, each
having two network interfaces. A single network interface on each
host was connected to a virtual LAN within the ESX environment. The
second network interface on each host was connected to the corporate
network. During the experiments, the CPU load on the virtualization
server was checked to ensure that it was not impacting the results.
5.2.1. Web Server Configuration
Four of the hosts were configured identically as web servers, with
the following software:
o OS: Ubuntu 11.04
o Web Server: "Express-SPDY" written in JavaScript for node.js
The "Express-SPDY" server is the melding of a compact and efficient
open-source HTTPS server (known as "Express") with an open-source
SPDY/2 implementation called "node-spdy". The implementation is
written in JavaScript using the node.js server environment. This
implementation was chosen over the Chromium package published by
Google due to the fact that it was lightweight and could be deployed
relatively easily. Unfortunately, the version of Express-SPDY
available at the time of this testing was found to not be stable, and
required some work to debug and fix prior to beginning any testing.
White, et al. Expires January 7, 2013 [Page 16]
Internet-Draft spdy-analysis July 2012
Since our testing commenced, the author of node-spdy rewrote a large
part of the package.
5.2.2. Client Configurations
The remaining three hosts were configured as client machines. The
primary client ran Windows 7, and the other two clients (used for
spot testing) ran Windows XP and Ubuntu 11.04. All clients used the
Chrome 16 browser, with the SPDY Benchmarking plugin, and were
installed with Wireshark to capture network traffic. The Dummynet
[DUMMYNET] network simulator was used to simulate the various network
conditions.
For each test condition, Dummynet was configured with the desired
upstream and downstream rates, with the desired RTT split equally
between the two links. In cases where packet loss was being tested,
the same packet loss rate was configured on both links. Both links
were left with the default buffering configuration of 50 packets.
Note that two of the hosts (Win7 and Ubuntu) support the TCP Window
Scaling Option by default, while the third (WinXP) does not.
5.3. Test Conditions
5.3.1. Protocol Options
As previously stated, the testing is intended to compare the SPDY
protocol to the HTTPS protocol, and to investigate the use of the
increased TCP Initial Congestion Window (initcwnd) (both with HTTPS
and in conjunction with SPDY). As a result, we measured the time it
took to load a web page using four different protocol options as
shown in Table 4. One option defines the baseline scenario or "Base
Case": with HTTPS for transport and the default initcwnd setting of
3. The results for the three cases involving a proposed enhancement
are then compared to the "Base Case".
Table 4. Matrix of Protocol Options
+-------------+---------+
| HTTPS | SPDY |
+------------------+-------------+---------+
| initcwnd = 3 | Base Case | CASE 1 |
+------------------+-------------+---------+
| initcwnd = 10 | CASE 2 | CASE 3 |
+------------------+-------------+---------+
White, et al. Expires January 7, 2013 [Page 17]
Internet-Draft spdy-analysis July 2012
5.3.2. Web Sites
Nine different "websites" were used to evaluate the impact that the
content and server configuration would have on the test results. The
nine websites were formed by the 3x3 matrix of three different server
configurations and three different web pages as follows.
5.3.2.1. Server Configurations
The server configuration parameter determined how all of the web
resources for a page were served.
o Server Case 1, all resources were served by a single server.
o Server Case 2, the resources were divided equally across two
servers.
o Server Case 3, the resources were divided equally across four
servers.
5.3.2.1.1. Web Pages
Three web pages were used in testing. These pages were chosen to
span a range of possible page configurations. In particular, Page B
was modeled based on the data collected in the website survey
(Section 4.1), while Pages A and C represent more extreme situations.
Page A is the smallest of the three pages, and is populated by a
number of small image files. Page C is the largest, and is populated
by a small number of very large image files. It could be argued that
a page like Page C would be fairly rare on the Internet compared to
pages like Page A or Page B.
Page A consisted of 102 total resources, with a total page size of
369 KB. It was composed of the following resources:
o 1 HTML file: 12.8 KB
o 1 JavaScript Library (jquery.js): 94.5 KB
o 100 JPEG images: min/mean/max/stddev size = 1.3/2.6/5.5/1.2 KB
Page B consisted of 101 total resources, with a total page size of
1.4 MB. It was composed of the following resources:
o 1 HTML file: 8.2 KB
o 100 JPEG/PNG/GIF images:
White, et al. Expires January 7, 2013 [Page 18]
Internet-Draft spdy-analysis July 2012
* min/mean/max/stddev size = 46B/14KB/103KB/24KB
* Approximately log-uniform size distribution
Page C consisted of 11 total resources, with a total page size of 3.0
MB. It was composed of the following resources:
o 1 HTML file: 0.8 KB
o 10 JPEG images: min/mean/max/stddev size = 298/302/310/4 KB
5.3.2.2. Channel Conditions
Eight different channel conditions were used in the testing. These
eight cases were formed from the 2x2x2 matrix of the following
parameters:
Configured Data Rate:
10 Mbps downstream, 1 Mbps upstream
20 Mbps downstream, 2 Mbps upstream
Round Trip Time:
20 ms
100 ms
Packet Loss Rate:
0%
1%
The two Configured Data Rates were chosen to match two commonly used
residential high-speed data service configurations. The Round Trip
Times correspond to a fairly short RTT that models a user accessing a
site that is located fairly close to them (or is accelerated by a
CDN), as well as a longer RTT which models a U.S. coast-to-coast page
load or a case where there is increased upstream latency due to
bufferbloat. Performance in higher RTT cases, such as an overseas
connection, is not considered in this study. The two packet loss
rates correspond to an idealistic case that might be approached by an
end-to-end wired network connection and a case with packet loss that
might be more typical of a network connection that includes a Wi-Fi
link.
White, et al. Expires January 7, 2013 [Page 19]
Internet-Draft spdy-analysis July 2012
5.3.2.3. Test Execution
The test execution will be performed in the following fashion:
o Run all 288 test conditions on Win 7
o Run spot tests on WinXP and Ubuntu
o For each test condition, perform 20 page loads (clearing cache and
resetting connections before each load) and calculate median load
time.
5.4. Test Results
The result of the spot tests on WinXP and Ubuntu were largely
consistent with the results for the Win7 testing. However, there was
a subset of the test cases with WinXP where SPDY showed degraded
performance compared to the other OSes. This occurred with the cases
that used a single server, 100 ms RTT, and 0% packet loss. This is
most likely a result of the lack of TCP Window Scaling in the WinXP
client, which limits the data rate of each TCP connection to 5 Mbps
in these conditions. Since SPDY is using a single TCP connection
(compared to six for HTTP), SPDY will see a performance hit due to
its inability to use the entire 10 Mbps or 20 Mbps pipe.
The raw data for all test cases (including the WinXP and Ubuntu test
cases) are provided in [SPDY-ANALYSIS]. The remainder of this report
will focus on the Win7 test results.
As described in Section 5.3.1, the 2x2 matrix of protocol options
will be presented as three different "cases", where each case
represents the gain achieved by using one of the three proposed
protocol enhancement options compared to the base case. This section
is broken into three subsections that correspond to the three cases.
For each case, the results comprise a five-dimensional matrix of test
conditions. In order to present the results in a compact way, each
subsection below will first examine the impact that website
configuration has on the achieved gain, then second will examine the
impact of the channel conditions.
The results provide a comparison between the median web page download
time achieved using the proposed protocol enhancement option and that
which is achieved in the base case.
White, et al. Expires January 7, 2013 [Page 20]
Internet-Draft spdy-analysis July 2012
5.4.1. CASE 1: SPDY vs. HTTPS
Case 1 compares the median web page download time achieved using
SPDY/2 to the median time achieved using HTTPS. The comparison is
made for each website configuration and channel condition
combination. The comparison is reported as a "gain", which is
calculated as the ratio of median page load time using HTTPS to the
median page load time using SPDY. As a result, gain values greater
than 1 indicate that SPDY/2 provides an advantage over HTTPS. As an
additional point of reference, a gain of 2 corresponds to a 50%
reduction in page load time, the goal to which SPDY aspires.
As stated above, the impact that the website configuration has on the
achieved gain will be examined first. Table 5 shows the 3x3 matrix
of website configurations. For each website, the maximum, minimum
and mean gain values (across all channel conditions) are provided, in
the manner shown below.
+-------------+
| max |
| mean |
| min |
+-------------+
Additionally, row-wise, column-wise, and overall statistics are
calculated and shown around the periphery of the 3x3 matrix.
Table 5. Website Impact on SPDY Gain
| 1 server | 2 servers | 4 servers | average |
---------+-------------+-------------+-------------+-------------+
| 4.1 | 1.8 | 1.8 | 4.1 |
Page A | 2.4 | 1.3 | 1.3 | 1.7 |
| 1.6 | 1.1 | 1.1 | 1.1 |
---------+-------------+-------------+-------------+-------------+
| 3.3 | 1.4 | 1.4 | 3.3 |
Page B | 1.7 | 1.0 | 1.0 | 1.3 |
| 0.8 | 0.6 | 0.7 | 0.6 |
---------+-------------+-------------+-------------+-------------+
| 1.0 | 1.0 | 1.1 | 1.1 |
Page C | 0.5 | 0.7 | 0.9 | 0.7 |
| 0.3 | 0.4 | 0.5 | 0.3 |
---------+-------------+-------------+-------------+-------------+
| 4.1 | 1.8 | 1.8 | 4.1 |
average | 1.6 | 1.0 | 1.1 | 1.2 |
| 0.3 | 0.4 | 0.5 | 0.3 |
---------+-------------+-------------+-------------+-------------+
White, et al. Expires January 7, 2013 [Page 21]
Internet-Draft spdy-analysis July 2012
These results show that SPDY worked well for Page A (many smaller
images), showing gains across all test conditions and an average gain
of 1.7. For Page B (many images more typical size), the results were
a bit more hit-or-miss, with SPDY not always resulting in improved
performance. Nonetheless, on average a gain of 1.3 was achieved.
Page C (small number of large images), on the other hand, resulted in
almost universally worse performance with SPDY than with HTTPS. In
the worst case, SPDY resulted in a 3.3x increase in page load time
(gain of 0.3). On average SPDY took 1.4x longer (gain of 0.7) to
download Page C than traditional HTTPS.
In general, the SPDY impact (positive or negative) diminished as more
servers were utilized. This is the result of the increased
parallelism and decreased number of objects requested per SPDY
session, which if taken to the extreme would result in a pattern of
requests that is similar to the HTTPS case.
The results shown in Table 6 examine the impact that channel
conditions have on SPDY performance (relative to HTTPS). Since the
channel conditions were formed from a 2x2x2 matrix of data-rate, RTT,
and packet-loss rate (PLR), the results are depicted as two 2x2
matrices of data-rate and RTT, one corresponding to the 0% PLR test
cases, and the other corresponding to the 1% PLR test cases. For
each channel condition, the results for all nine websites are
summarized via the maximum, mean, and minimum gain achieved. Similar
to the row and column statistics provided in the website impact
analysis in Table 5, summary statistics for all three dimensions are
provided around the periphery of the 2x2x2 cube (i.e., in the right-
most column, the last row, and the third table).
Table 6a. Channel Impact on SPDY Gain (0% PLR)
--------+-------------+-------------+-------------+
0% PLR | 20ms | 100ms | average |
--------+-------------+-------------+-------------+
| 2.1 | 4.1 | 4.1 |
10/1 | 1.4 | 1.7 | 1.6 |
| 0.7 | 0.8 | 0.7 |
--------+-------------+-------------+-------------+
| 1.6 | 3.4 | 3.4 |
20/2 | 1.2 | 1.6 | 1.4 |
| 0.5 | 0.7 | 0.5 |
--------+-------------+-------------+-------------+
| 2.1 | 4.1 | 4.1 |
average | 1.3 | 1.6 | 1.5 |
| 0.5 | 0.7 | 0.5 |
--------+-------------+-------------+-------------+
White, et al. Expires January 7, 2013 [Page 22]
Internet-Draft spdy-analysis July 2012
Table 6b. Channel Impact on SPDY Gain (1% PLR)
--------+-------------+-------------+-------------+
1% PLR | 20ms | 100ms | average |
---------------------------------------------------
| 2.3 | 2.2 | 2.3 |
10/1 | 1.1 | 0.9 | 1.0 |
| 0.4 | 0.3 | 0.3 |
--------+-------------+-------------+-------------+
| 2.0 | 1.7 | 2.0 |
20/2 | 1.0 | 0.8 | 0.9 |
| 0.3 | 0.3 | 0.3 |
--------+-------------+-------------+-------------+
| 2.3 | 2.2 | 2.3 |
average | 1.0 | 0.8 | 0.9 |
| 0.3 | 0.3 | 0.3 |
--------+-------------+-------------+-------------+
Table 6c. Channel Impact on SPDY Gain (all PLR)
--------+-------------+-------------+-------------+
all PLR | 20ms | 100ms | average |
--------+-------------+-------------+-------------+
| 2.3 | 4.1 | 4.1 |
10/1 | 1.3 | 1.3 | 1.3 |
| 0.4 | 0.3 | 0.3 |
--------+-------------+-------------+-------------+
| 2.0 | 3.4 | 3.4 |
20/2 | 1.1 | 1.2 | 1.1 |
| 0.3 | 0.3 | 0.3 |
--------+-------------+-------------+-------------+
| 2.3 | 4.1 | 4.1 |
average | 1.2 | 1.2 | 1.2 |
| 0.3 | 0.3 | 0.3 |
--------+-------------+-------------+-------------+
The most interesting observation here is the comparison between the
two PLR tests. SPDY provided significant gains when the PLR was 0%
(average gain 1.5), but showed worse average performance than HTTPS
when packet loss was 1% (average gain 0.9). This effect was more
pronounced in the large RTT cases as compared to the small RTT. This
result points to a weakness in the way that SPDY utilizes TCP. With
a typical HTTPS download of a web page, the browser will open a large
number of simultaneous TCP connections. In this case, a random
packet loss will cause the one affected connection to temporarily
reduce its congestion window (and hence the effective data rate), but
the other connections will be unaffected, and in fact may be able to
opportunistically make use of the bandwidth made available by the
White, et al. Expires January 7, 2013 [Page 23]
Internet-Draft spdy-analysis July 2012
affected connection. The result for HTTPS is that random packet loss
has only a minor impact on page download time. In the case of SPDY,
the number of parallel TCP connections is dramatically reduced (by as
much as a factor of six), so that random packet loss has a much
bigger impact on the overall throughput.
In the absence of packet loss, SPDY provides better gains as the RTT
increases. This is the result of reducing the number of round trips
needed to fetch all of the page resources. Nonetheless, there were
test cases with 0 packet loss where SPDY performed worse than HTTPS.
This only occurred with Page C; SPDY always provided a benefit for
Pages A and B when there was no packet loss. The root cause of the
degradation is unknown, but it may be the result of the buffering
configuration of the network simulator Dummynet. As stated
previously, testing was performed with the default configuration of
50 packet buffers. This configuration is not reflective of real
networks (which in some cases may have more than double that amount
of buffering), and could bias the results somewhat against
applications (such as SPDY) that use a small number of TCP sessions.
Additionally, SPDY generally shows more gain in the lower Data Rate
cases.
The overall average gain of 1.2 seen in our experiments aligns well
with the performance gains that Google is seeing in their live
deployments. In results presented in a December 8, 2011, Google
TechTalk [TECHTALK], they report a 15.4% improvement in page load
time (equivalent to a gain of 1.18).
5.4.2. CASE 2: initcwnd=10 vs. initcwnd=3
The initcwnd increase provides very modest gain across the majority
of test cases. In a few cases there was a marginal degradation of
performance compared to the default initcwnd case. Across all test
conditions an average gain of 1.1 (or 9% reduction in page load time)
was seen, as indicated in Table 7 and Table 8.
White, et al. Expires January 7, 2013 [Page 24]
Internet-Draft spdy-analysis July 2012
Table 7. Website Impact on Initcwnd Gain
+-------------+-------------+-------------+-------------+
| 1 server | 2 servers | 4 servers | average |
---------+-------------+-------------+-------------+-------------+
| 1.1 | 1.1 | 1.1 | 1.1 |
Page A | 1.0 | 1.0 | 1.0 | 1.0 |
| 0.9 | 1.0 | 0.9 | 0.9 |
---------+-------------+-------------+-------------+-------------+
| 1.1 | 1.3 | 1.2 | 1.3 |
Page B | 1.0 | 1.1 | 1.0 | 1.0 |
| 1.0 | 1.0 | 0.9 | 0.9 |
---------+-------------+-------------+-------------+-------------+
| 1.2 | 1.3 | 1.5 | 1.5 |
Page C | 1.1 | 1.1 | 1.2 | 1.1 |
| 1.0 | 1.0 | 1.0 | 1.0 |
---------+-------------+-------------+-------------+-------------+
| 1.2 | 1.3 | 1.5 | 1.5 |
average | 1.1 | 1.1 | 1.1 | 1.1 |
| 0.9 | 1.0 | 0.9 | 0.9 |
---------+-------------+-------------+-------------+-------------+
Overall the Page C test cases showed a slightly higher gain over the
other two pages. Since Page C involves large files, it isn't
surprising that the effect is slightly more pronounced than in Page
A. In Page A, the majority of resources can be delivered in 3 packets
or less anyway, so the increase in initcwnd doesn't reduce the number
of round-trips.
Table 8a. Channel Impact on Initcwnd Gain (0% PLR)
---------+-------------+-------------+-------------+
0% PLR | 20ms | 100ms | average |
---------+-------------+-------------+-------------+
| 1.1 | 1.2 | 1.2 |
10/1 | 1.0 | 1.1 | 1.0 |
| 1.0 | 1.0 | 1.0 |
---------+-------------+-------------+-------------+
| 1.1 | 1.1 | 1.1 |
20/2 | 1.0 | 1.0 | 1.0 |
| 1.0 | 0.9 | 0.9 |
---------+-------------+-------------+-------------+
| 1.1 | 1.2 | 1.2 |
average | 1.0 | 1.0 | 1.0 |
| 1.0 | 0.9 | 0.9 |
---------+-------------+-------------+-------------+
White, et al. Expires January 7, 2013 [Page 25]
Internet-Draft spdy-analysis July 2012
Table 8b. Channel Impact on Initcwnd Gain (1% PLR)
---------+-------------+-------------+-------------+
1% PLR | 20ms | 100ms | average |
---------+-------------+-------------+-------------+
| 1.5 | 1.2 | 1.5 |
10/1 | 1.1 | 1.1 | 1.1 |
| 0.9 | 1.0 | 0.9 |
---------+-------------+-------------+-------------+
| 1.4 | 1.3 | 1.4 |
20/2 | 1.1 | 1.1 | 1.1 |
| 0.9 | 0.9 | 0.9 |
---------+-------------+-------------+-------------+
| 1.5 | 1.3 | 1.5 |
average | 1.1 | 1.1 | 1.1 |
| 0.9 | 0.9 | 0.9 |
---------+-------------+-------------+-------------+
Table 8c. Channel Impact on Initcwnd Gain (all PLR)
---------+-------------+-------------+-------------+
all PLR | 20ms | 100ms | average |
---------+-------------+-------------+-------------+
| 1.5 | 1.2 | 1.5 |
10/1 | 1.1 | 1.1 | 1.1 |
| 0.9 | 1.0 | 0.9 |
---------+-------------+-------------+-------------+
| 1.4 | 1.3 | 1.4 |
20/2 | 1.1 | 1.1 | 1.1 |
| 0.9 | 0.9 | 0.9 |
---------+-------------+-------------+-------------+
| 1.5 | 1.3 | 1.5 |
average | 1.1 | 1.1 | 1.1 |
| 0.9 | 0.9 | 0.9 |
---------+-------------+-------------+-------------+
When viewing the results broken down by channel condition, we don't
see a significant difference in gain from one channel condition to
the next. Notably, even in the high RTT cases we don't see much
change in the gain. This is likely due to the fact that the majority
of resources had sizes that either fell below the 3 packet default
initcwnd, or were much larger (e.g., over 210 packets each for the
images in Page C) so that the initcwnd change might save a single RTT
for a transfer that took 9 RTTs using the default initcwnd.
White, et al. Expires January 7, 2013 [Page 26]
Internet-Draft spdy-analysis July 2012
5.4.3. CASE 3: SPDY+initcwnd=10 vs. HTTPS+initcwnd=3
The combination of SPDY and the initcwnd increase resulted in a
benefit that was more than the product of the two individual gains.
This is a result of the fact that, due to SPDY, all of the TCP
sessions involved multiple round-trips, so the acceleration of
initial data rate provided by the initcwnd increase resulted in
tangible benefits. In fact, one case (Page A, single server) saw an
astounding gain of 4.7. By examining the raw data in
[SPDY-ANALYSIS], it can be seen that this combination reduced the
median page load time of 8.7 seconds down to 1.9 seconds.
Unfortunately, this result is not typical, and Page C again
experienced worse performance when the new protocol options are used.
The overall average gain seen was 1.4 across all test cases as shown
in Table 9 and Table 10.
Table 9. Website Impact on SPDY+Initcwnd Gain
--------+-------------+-------------+-------------+-------------+
| 1 server | 2 servers | 4 servers | average |
--------+-------------+-------------+-------------+-------------+
| 4.7 | 2.3 | 1.5 | 4.7 |
Page A | 3.2 | 1.5 | 1.4 | 2.0 |
| 2.2 | 1.2 | 1.2 | 1.2 |
--------+-------------+-------------+-------------+-------------+
| 3.3 | 1.6 | 1.5 | 3.3 |
Page B | 1.9 | 1.1 | 1.1 | 1.4 |
| 0.9 | 0.6 | 0.7 | 0.6 |
--------+-------------+-------------+-------------+-------------+
| 1.0 | 1.0 | 1.1 | 1.1 |
Page C | 0.6 | 0.7 | 0.9 | 0.7 |
| 0.3 | 0.4 | 0.6 | 0.3 |
--------+-------------+-------------+-------------+-------------+
| 4.7 | 2.3 | 1.5 | 4.7 |
average | 1.9 | 1.1 | 1.1 | 1.4 |
| 0.3 | 0.4 | 0.6 | 0.3 |
--------+-------------+-------------+-------------+-------------+
White, et al. Expires January 7, 2013 [Page 27]
Internet-Draft spdy-analysis July 2012
Table 10a. Channel Impact on SPDY+Initcwnd Gain (0% PLR)
---------+-------------+-------------+-------------+
0% PLR | 20ms | 100ms | average |
---------+-------------+-------------+-------------+
| 3.2 | 4.7 | 4.7 |
10/1 | 1.7 | 1.8 | 1.8 |
| 1.0 | 0.9 | 0.9 |
---------+-------------+-------------+-------------+
| 3.1 | 3.9 | 3.9 |
20/2 | 1.6 | 1.7 | 1.6 |
| 0.7 | 0.7 | 0.7 |
---------+-------------+-------------+-------------+
| 3.2 | 4.7 | 4.7 |
average | 1.7 | 1.7 | 1.7 |
| 0.7 | 0.7 | 0.7 |
---------+-------------+-------------+-------------+
Table 10b. Channel Impact on SPDY+Initcwnd Gain (1% PLR)
---------+-------------+-------------+-------------+
1% PLR | 20ms | 100ms | average |
---------+-------------+-------------+-------------+
| 3.1 | 2.4 | 3.1 |
10/1 | 1.2 | 0.9 | 1.1 |
| 0.4 | 0.3 | 0.3 |
---------+-------------+-------------+-------------+
| 2.8 | 2.2 | 2.8 |
20/2 | 1.2 | 1.0 | 1.1 |
| 0.3 | 0.3 | 0.3 |
---------+-------------+-------------+-------------+
| 3.1 | 2.4 | 3.1 |
average | 1.2 | 1.0 | 1.1 |
| 0.3 | 0.3 | 0.3 |
---------+-------------+-------------+-------------+
White, et al. Expires January 7, 2013 [Page 28]
Internet-Draft spdy-analysis July 2012
Table 10c. Channel Impact on SPDY+Initcwnd Gain (all PLR)
---------+-------------+-------------+-------------+
all PLR | 20ms | 100ms | average |
---------+-------------+-------------+-------------+
| 3.2 | 4.7 | 4.7 |
10/1 | 1.5 | 1.4 | 1.4 |
| 0.4 | 0.3 | 0.3 |
---------+-------------+-------------+-------------+
| 3.1 | 3.9 | 3.9 |
20/2 | 1.4 | 1.3 | 1.3 |
| 0.3 | 0.3 | 0.3 |
---------+-------------+-------------+-------------+
| 3.2 | 4.7 | 4.7 |
average | 1.4 | 1.3 | 1.4 |
| 0.3 | 0.3 | 0.3 |
---------+-------------+-------------+-------------+
In the packet loss test cases, there was a fairly significant
performance loss with SPDY, as reported in earlier test cases. This
is particularly true in the high RTT cases.
5.4.4. Results Summary
The results are summarized further in Table 11. A caveat on the
average results presented here is worth noting: the 72 test cases
were selected to test the protocol changes in a fairly wide range of
conditions in order to understand the impact that individual factors
have on the performance gains. As a result, the 72 test cases likely
do not comprise a statistically accurate sampling of real-world
sites, and so the average results presented here may not accurately
reflect the average performance gain that would be seen in the real
world. However, as noted previously, the average results do appear
to align well with the average gains seen by Google via their live
deployments with users utilizing the Chrome browser. In addition, it
is interesting to examine the average gain achieved in a particular
subset of the test conditions. For that we select the subset
consisting of pages A and B operating on a wireline network (PLR=0%).
For that subset, we see that the combination of SPDY/2 and the
initcwnd increase achieves an average gain of 2.1 (52% reduction in
page load time).
White, et al. Expires January 7, 2013 [Page 29]
Internet-Draft spdy-analysis July 2012
Table 11. Summary of Results
+--------------+--------------+--------------+
| Case 1 | Case 2 | Case 3 |
| SPDY | initcwnd |SPDY+initcwnd |
+----------------------+--------------+--------------+--------------+
|Best Gain | 4.1 | 1.5 | 4.7 |
|(all test conditions) |(A,1,10,100,0)|(C,4,10,20,1) |(A,1,10,100,0)|
+----------------------+--------------+--------------+--------------+
|Average Gain | | | |
|(all test conditions) | 1.2 | 1.1 | 1.4 |
+----------------------+--------------+--------------+--------------+
|Worst Gain | 0.3 | 0.9 | 0.3 |
|(all test conditions) | (C,1,x,x,1) |(A,1,20,100,0)|(C,1,20,100,1)|
+----------------------+--------------+--------------+--------------+
|# test cases achieving| 40 of 72 | 58 of 72 | 44 of 72 |
| gain >= 1 | (56%) | (81%) | (61%) |
+----------------------+--------------+--------------+--------------+
|# test cases achieving| 8 of 72 | 0 of 72 | 14 of 72 |
| gain >= 2 | (11%) | (0%) | (19%) |
+----------------------+--------------+--------------+--------------+
|Average Gain | | | |
|(Pages A and B | 1.8 | 1.0 | 2.1 |
| with PLR=0%) | | | |
+----------------------+--------------+--------------+--------------+
6. Recommendations and Conclusions
The results presented here indicate that SPDY/2 in conjunction with
the TCP Initial Congestion Window increase has the potential to
improve or impair web page download performance depending on a number
of factors.
Deployment of SPDY/2 for general-purpose web servers should be
considered in light of the following concerns:
o While the Chrome and Firefox browsers are important (approximately
29% and 22% market share respectively as of June 2012), client
support isn't ubiquitous.
o Some web page downloads were significantly impaired by SPDY.
o Work is underway on a revision to the protocol, and with
standardization in the IETF a possibility in the near future,
waiting may be worthwhile to see what develops in this space.
On the other hand, for a controlled environment where a single entity
White, et al. Expires January 7, 2013 [Page 30]
Internet-Draft spdy-analysis July 2012
provides the servers, the client software, and the web content, SPDY
might be a valuable tool. An example application might be a remote
user interface (RUI) that is delivered via HTTP to a CPE device.
In addition to potential uses for SPDY by network operators, another
area of interest is the impact that SPDY would have on network
traffic if it were to be widely adopted. In general the story here
is good, by reducing the number of simultaneous TCP sessions and
extending the duration of many of the sessions, other applications
could see improved performance as a result of TCP congestion
avoidance being invoked far more often. Secondly, the expectation
would be to see a slight reduction in data usage, due to the greater
efficiency that SPDY provides (fewer TCP control packets), as well as
a slight increase in average packet size, due to the multiplexing of
HTTP responses. Both of these factors will serve to reduce the
packets per second rate of the network, which improves scalability of
CMTSs, routers, DPI boxes, etc. Finally, equipment implementing
Carrier Grade NAT (which will be a key element of the IPv6 transition
in some networks) could see improved scalability as the number of
simultaneous TCP connections is reduced.
In regards to the increase in the TCP Initial Congestion Window,
while we see only marginal gains resulting from this enhancement, the
change can be made simply by changing a server operating system
parameter. It requires no client modifications. As a result, we see
no reason not to set the server initcwnd at the proposed value of 10.
7. IANA Considerations
This document makes no request of IANA.
Note to RFC Editor: this section may be removed on publication as an
RFC.
8. Security Considerations
See [I-D.mbelshe-httpbis-spdy] and [I-D.ietf-tcpm-initcwnd] for
Security Considerations.
9. Acknowledgements
The authors would like to thank Marc Weaver, Darshak Thakore, Eric
Winkelman, Robin Sam Ku, Scott Maize, and Rob Moon for their
assistance in developing and configuring the test bed, and collecting
the data presented in this report. The authors would also like to
White, et al. Expires January 7, 2013 [Page 31]
Internet-Draft spdy-analysis July 2012
gratefully express their appreciation to Chris Donley, Joan Strosin,
Ken Barringer and Christie Poland for manuscript preparation
assistance, and to Mike Belshe and Roberto Peon for their review and
very insightful comments.
10. References
[BUFFERCONTROL]
CableLabs, "Cable Modem Buffer Control", <http://
www.cablelabs.com/specifications/
CM-GL-Buffer-V01-110915.pdf>.
[CDNPLANET]
"www.cdnplanet.com", <http://www.cdnplanet.com>.
[DUMMYNET]
"Dummynet", <http://info.iet.unipi.it/~luigi/dummynet/>.
[Faster] Google, "Let's Make the Web Faster",
<http://code.google.com/speed>.
[GOOGLE-BENCHMARKS]
Lloyd , M., "SPDY and Server Push: Analysis and
Experiments", June 2010,
<https://docs.google.com/View?id=d446246_0cc6c6dkr>.
[HTTP_Pipelining]
Wikipedia, "HTTP Pipelining",
<http://en.wikipedia.org/wiki/HTTP_pipelining>.
[I-D.agl-tls-nextprotoneg]
Langley, A., "Transport Layer Security (TLS) Next Protocol
Negotiation Extension", draft-agl-tls-nextprotoneg-03
(work in progress), April 2012.
[I-D.ietf-tcpm-initcwnd]
Chu, J., Dukkipati, N., Cheng, Y., and M. Mathis,
"Increasing TCP's Initial Window",
draft-ietf-tcpm-initcwnd-03 (work in progress),
February 2012.
[I-D.mbelshe-httpbis-spdy]
Belshe, M. and R. Peon, "SPDY Protocol",
draft-mbelshe-httpbis-spdy-00 (work in progress),
February 2012.
[INITCWND-PAPER]
White, et al. Expires January 7, 2013 [Page 32]
Internet-Draft spdy-analysis July 2012
Dukkipati , N., Refice, T., Cheng, Y., Chu, J., Sutin, N.,
Agarwal, A., Herbert, T., and A. Jain, "An Argument for
Increasing TCP's Initial Congestion Window", <http://
code.google.com/speed/articles/tcp_initcwnd_paper.pdf>.
[RFC0793] Postel, J., "Transmission Control Protocol", STD 7,
RFC 793, September 1981.
[RFC1122] Braden, R., "Requirements for Internet Hosts -
Communication Layers", STD 3, RFC 1122, October 1989.
[RFC2001] Stevens, W., "TCP Slow Start, Congestion Avoidance, Fast
Retransmit, and Fast Recovery Algorithms", RFC 2001,
January 1997.
[RFC2414] Allman, M., Floyd, S., and C. Partridge, "Increasing TCP's
Initial Window", RFC 2414, September 1998.
[RFC2616] Fielding, R., Gettys, J., Mogul, J., Frystyk, H.,
Masinter, L., Leach, P., and T. Berners-Lee, "Hypertext
Transfer Protocol -- HTTP/1.1", RFC 2616, June 1999.
[RFC3390] Allman, M., Floyd, S., and C. Partridge, "Increasing TCP's
Initial Window", RFC 3390, October 2002.
[SPDY] "SPDY - The Chromium Projects",
<http://www.chromium.org/spdy>.
[SPDY-ANALYSIS]
White, G., "Analysis of Google Spdy and TCP initcwnd",
May 2012, <http://www.cablelabs.com/downloads/pubs/
Analysis_of_Google_SPDY_TCP.pdf>.
[TECHTALK]
"http://www.cnx-software.com/2012/01/27/
spdy-aims-to-make-the-web-faster-and-replace-http/",
<http://www.cnx-software.com/2012/01/page/2/ >.
White, et al. Expires January 7, 2013 [Page 33]
Internet-Draft spdy-analysis July 2012
Authors' Addresses
Greg White
CableLabs
858 Coal Creek Circle
Louisville, CO 80027-9750
USA
Email: g.white@cablelabs.com
Jean-Francois Mule
CableLabs
180 Montgomery St
Suite 2480
San Francisco, CA 94104-4203
USA
Email: jf.mule@cablelabs.com
Dan Rice
CableLabs
858 Coal Creek Circle
Louisville, CO 80027-9750
USA
Email: d.rice@cablelabs.com
White, et al. Expires January 7, 2013 [Page 34]
