Internet Engineering Task Force T. Ahuja
Internet-Draft Cisco Systems, Inc.
Intended status: Informational T. Alexander
Expires: July 5, 2008 VeriWave, Inc.
S. Bradner
Harvard University
S. Hooda
Cisco Systems, Inc.
J. Perser
VeriWave, Inc.
M. Sambi
Cisco Systems, Inc.
January 2, 2008
Benchmarking Methodology for Wireless LAN Switching Systems
draft-alexander-bmwg-wlan-switch-meth-01
Status of this Memo
By submitting this Internet-Draft, each author represents that any
applicable patent or other IPR claims of which he or she is aware
have been or will be disclosed, and any of which he or she becomes
aware will be disclosed, in accordance with Section 6 of BCP 79.
Internet-Drafts are working documents of the Internet Engineering
Task Force (IETF), its areas, and its working groups. Note that
other groups may also distribute working documents as Internet-
Drafts.
Internet-Drafts are draft documents valid for a maximum of six months
and may be updated, replaced, or obsoleted by other documents at any
time. It is inappropriate to use Internet-Drafts as reference
material or to cite them other than as "work in progress."
The list of current Internet-Drafts can be accessed at
http://www.ietf.org/ietf/1id-abstracts.txt.
The list of Internet-Draft Shadow Directories can be accessed at
http://www.ietf.org/shadow.html.
This Internet-Draft will expire on July 5, 2008.
Copyright Notice
Copyright (C) The IETF Trust (2008).
Ahuja, et al. Expires July 5, 2008 [Page 1]
Internet-Draft WLAN Switch Benchmarking Methodology January 2008
Abstract
This document provides a framework and methodology for performing
performance test and benchmarking of wireless LAN (WLAN) switches and
controllers, including systems comprising groups of controllers and
WTPs. This document defines and discusses a number of tests and
associated test conditions that may be used to characterize the
performance of such systems, and also supplies the methods used to
calculate the expected results of these tests. Specific formats for
reporting the results of the tests are also provided, where
applicable. The tests described in this document extend the
methodology defined for benchmarking network interconnect devices in
RFC 2544, and LAN switches in RFC 2889, to WLAN switch controller
systems. The methodology herein is to be used together with the
companion terminology document.
Ahuja, et al. Expires July 5, 2008 [Page 2]
Internet-Draft WLAN Switch Benchmarking Methodology January 2008
Table of Contents
1. Introduction . . . . . . . . . . . . . . . . . . . . . . . . . 5
2. Existing definitions and requirements . . . . . . . . . . . . 6
3. General description and test setups . . . . . . . . . . . . . 6
3.1. Tester functional model . . . . . . . . . . . . . . . . . 6
3.2. Test setups . . . . . . . . . . . . . . . . . . . . . . . 7
3.3. Configuration parameters . . . . . . . . . . . . . . . . . 10
3.3.1. WTP setup . . . . . . . . . . . . . . . . . . . . . . 10
3.3.2. Service priority . . . . . . . . . . . . . . . . . . . 10
3.3.3. Test conditions . . . . . . . . . . . . . . . . . . . 11
4. Interpreting and reporting test results . . . . . . . . . . . 21
5. Benchmarking tests . . . . . . . . . . . . . . . . . . . . . . 21
5.1. Data plane tests . . . . . . . . . . . . . . . . . . . . . 22
5.1.1. Unicast throughput . . . . . . . . . . . . . . . . . . 24
5.1.2. Unicast maximum forwarding rate and frame loss
ratio . . . . . . . . . . . . . . . . . . . . . . . . 25
5.1.3. Multicast forwarding rate . . . . . . . . . . . . . . 27
5.1.4. Latency and jitter . . . . . . . . . . . . . . . . . . 29
5.1.5. QoS differentiation . . . . . . . . . . . . . . . . . 31
5.1.6. Power-save throughput . . . . . . . . . . . . . . . . 33
5.2. Control plane tests . . . . . . . . . . . . . . . . . . . 35
5.2.1. Endstation roaming delay . . . . . . . . . . . . . . . 36
5.2.2. Endstation roaming rate . . . . . . . . . . . . . . . 39
5.2.3. Endstation association rate . . . . . . . . . . . . . 41
5.2.4. Endstation capacity . . . . . . . . . . . . . . . . . 43
5.2.5. WTP capacity . . . . . . . . . . . . . . . . . . . . . 46
5.2.6. Reset recovery time . . . . . . . . . . . . . . . . . 48
5.2.7. Failover recovery time . . . . . . . . . . . . . . . . 50
6. Security Considerations . . . . . . . . . . . . . . . . . . . 52
7. IANA Considerations . . . . . . . . . . . . . . . . . . . . . 52
8. References . . . . . . . . . . . . . . . . . . . . . . . . . . 52
8.1. Normative References . . . . . . . . . . . . . . . . . . . 52
8.2. Informative References . . . . . . . . . . . . . . . . . . 53
Appendix A. Intended load computations . . . . . . . . . . . . . 53
A.1. Calculating theoretical maximum media capacity . . . . . . 54
A.2. Calculating constant intended load . . . . . . . . . . . . 55
A.3. Calculating burst intended load . . . . . . . . . . . . . 56
Authors' Addresses . . . . . . . . . . . . . . . . . . . . . . . . 56
Ahuja, et al. Expires July 5, 2008 [Page 3]
Internet-Draft WLAN Switch Benchmarking Methodology January 2008
Intellectual Property and Copyright Statements . . . . . . . . . . 59
Ahuja, et al. Expires July 5, 2008 [Page 4]
Internet-Draft WLAN Switch Benchmarking Methodology January 2008
1. Introduction
Wireless LANs (WLANs) are deployed on a large scale in traditional
enterprises, in commercial service offerings such as coffee shops,
and in vertical applications such as inventory management. Large
deployments of WLANs, however, introduce several issues: an increased
administrative burden due to the use of IP-addressable Wireless
Termination Points (WTPs - i.e., Access Points); the need to ensure
consistency of configuration across all WTPs; the need to deal with
the dynamic nature of the WLAN medium, and to combat interference;
and the increased need for securing the network against unauthorized
intrusion or access.
To address the above problems, vendors offer solutions that combine
aspects of LAN switching, centralized control, and distributed
wireless access in an architecture comprising a set of relatively
simple Wireless termination points (WTPs) coupled to one or more
Access controllers (ACs). The use of centralized control and
monitoring simplifies many of the management and security issues
noted above, as the WTPs can be configured and controlled as a group
by the ACs, security policies can be administered on a WLAN-wide
basis, and the RF domain can be monitored and controlled from a
central location.
Each vendor offering such a system needs a protocol between ACs and
WTPs to support both centralized management and data transport
functions. The general practice has been for vendors to use a
proprietary protocol; however, the CAPWAP (Control and Provisioning
of Wireless Access Points) protocol is being standardized by the IETF
to provide a multi-vendor interoperable interface between WTPs and
ACs. The CAPWAP protocol also defines a standardized WLAN
architecture and mandatory functions (such as discovery) to enable a
common functional model to be adopted across the vendor base.
The ACs may perform both control plane and data plane functions
within a WLAN. It is therefore of significant interest to benchmark
their performance, as they have a material impact on the performance
and perceived end-user experience of WLANs built around them. ACs
may be benchmarked either as stand-alone entities, or in conjunction
with the WTPs to which they connect. When ACs are benchmarked in
conjunction with WTPs, the CAPWAP architectural model is used as a
reference.
This document defines and describes a test methodology that may be
used by vendors and users of IEEE 802.11 Wireless LAN (WLAN) [802.11]
switch controllers to measure and report performance characteristics
of such devices. It extends the methodology that was originally
defined for benchmarking network interconnecting devices in RFC 2544
Ahuja, et al. Expires July 5, 2008 [Page 5]
Internet-Draft WLAN Switch Benchmarking Methodology January 2008
[RFC2544], and then subsequently extended to other types of devices
(such as LAN switching devices in RFC 2889 [RFC2889]), to cover IEEE
802.11 WLAN devices.
Note that this document does not specify RF-related tests, or
performance benchmarks that pertain to the IEEE 802.11 link layer.
Instead, this document focuses on datapath and control path related
measurements performed above the link layer on ACs, or combinations
of ACs and WTPs. For RF-related tests, link layer metrics and tests
on individual WTPs, the reader is referred to the IEEE 802.11.2 draft
Recommended Practice on Wireless Performance [802.11.2], which
describes such tests.
2. Existing definitions and requirements
RFC 2544, "Benchmarking Methodology for Network Interconnect Devices"
[RFC2544] and RFC 2889, "Benchmarking Methodology for LAN Switching
Devices" [RFC2889], provide useful background information and
context, and SHOULD be reviewed before conducting tests based on this
document. WLAN-specific terms and definitions in this document are
described in Clauses 3 and 4 of the IEEE 802.11 standard [802.11].
For the sake of clarity and continuity this RFC adopts the general
template for benchmarking tests set out in Section 26 of RFC 2544.
The key words "MUST", "MUST NOT", "REQUIRED", "SHALL", "SHALL NOT",
"SHOULD", "SHOULD NOT", "RECOMMENDED", "MAY", and "OPTIONAL" in this
document are to be interpreted as described in RFC 2119 [RFC2119].
3. General description and test setups
A common set of test setup and measurement conditions is used across
all of the tests described in this document. Exceptions to these
conditions are noted, if necessary, in the descriptions of the
individual tests.
3.1. Tester functional model
For the purposes of this document, the tester is defined as a
separate device that is used to transmit controlled test traffic to
the physical ports of the device under test (DUT) or system under
test (SUT), as well as to receive and measure test traffic from the
physical ports of the DUT or SUT. The tester MUST NOT be a part of
the DUT or SUT, nor can the DUT or SUT provide any portion of the
reported test results.
Ahuja, et al. Expires July 5, 2008 [Page 6]
Internet-Draft WLAN Switch Benchmarking Methodology January 2008
The tester MUST transmit conformant traffic to the DUT or SUT during
the tests described herein, and MUST follow the rules of the relevant
protocol with respect to media access and frame exchanges. It MAY be
configured to transmit non-conformant traffic for special purposes
(e.g., for debug), but this is outside the scope of this document.
The tester MUST support some means of distinguishing test traffic
(either injected into or emitted by the DUT or SUT) from normal data,
control and management frames that are generated by the DUT or SUT
itself. The tester SHOULD further support means of unambiguously
determining frame loss and frame duplication (e.g., by the use of
sequence numbers), as well as time-stamping transmitted and received
frames.
No constraints are placed by this document on the specific
implementation of the tester or test system, provided that it is
capable of measuring DUT or SUT responses to the required degree of
accuracy, establishing the required test conditions at the physical
interfaces of the DUT or SUT, and generating test traffic with the
relevant parameters. These parameters include frame sizes, offered
load, burst sizes and inter-burst gap, signal output level, RTS/CTS
setting, and fragmentation setting.
3.2. Test setups
The general test setup comprises a DUT or SUT and a tester, as shown
in the figure below. The tester has at least one wired (Ethernet)
interface to the DUT or SUT; the other interface(s) may be either
wired or wireless (i.e., Ethernet or 802.11). In most cases, the DUT
or SUT has multiple interfaces that must be driven, and so the tester
likewise will have multiple wired and/or wireless interfaces.
802.11-Side Interfaces Ethernet-Side Interfaces
+---------------------------+
+-------->| |<----------+
| .......>| DUT or SUT |<......... |
| : .....>| |<...... : |
| : : +---------------------------+ : : |
| : : : : |
| : : +-------------------+ : : |
| : :........>| |<.........: : |
| :..........>| Tester |<............: |
+------------>| |<--------------+
+-------------------+
Figure 1
The DUT or SUT may be comprised of two different arrangements,
Ahuja, et al. Expires July 5, 2008 [Page 7]
Internet-Draft WLAN Switch Benchmarking Methodology January 2008
leading to two variations on the above general test setup. One
arrangement has only one or more ACs, but no WTPs; in this case, the
tester must emulate the WTPs as well as the endstations behind them,
and present the aggregate to the AC(s). In the second arrangement,
the device is actually a SUT comprising both WTPs and ACs (treated as
a single system); in this case, the tester only emulates the
endstations and presents them to the WTPs.
The following figure shows the first setup (DUT), with a single
tester that interfaces to a collection of one or more ACs. Note that
the ACs may be physically distinct (i.e., implemented in separate
chassis) or logically distinct (i.e., implemented in a single
chassis), but in either case is expected to form a logical whole for
the purposes of management, addressing, endstation association, and
traffic handling.
+-----+
Set of 1 or +-----+ |<-----------+
more ACs +------+ |-+ |
+---------------->| AC |-+ |
| +------+ |
| |
| |
| +---------------------------+ |
| | | |
+------->| Tester |<---+
| |
+---------------------------+
Figure 2
The figure below shows the second setup (SUT), again with a single
tester that interfaces to a set of WTPs connected to one or more ACs.
As before, the ACs may be physically or logically distinct.
Ahuja, et al. Expires July 5, 2008 [Page 8]
Internet-Draft WLAN Switch Benchmarking Methodology January 2008
+------+
+----->| WTP1 |----+
| +------+ |
| | +------+
| | +------+ |<----+
| +------+ | +------+ |-+ |
| +--->| WTP2 |----|---------| AC |-+ |
| | +------+ | +------+ |
| | . | |
| | . | |
| | +------+ | |
| | +->| WTPn |----+ |
| | | +------+ |
| | | |
| | | |
| | | +---------------------------+ |
| | +----->| | |
| +------->| Tester |<-----+
+--------->| |
+---------------------------+
Figure 3
A logical diagram of the test setup usually entails multiple VLANs
configured on the Ethernet side of the DUT or SUT, and multiple WLANs
configured on the wireless side. This is represented in the figure
below, which shows a SUT comprising several WTPs interfaced to an AC,
with three WLANs on the wireless side that are logically connected to
three VLANs on the Ethernet side. Note that a one-to-one
correspondence of WLANs to VLANs is usual but not required.
WLAN1,ES1---+-----+
WLAN2,ES2---| WTP |--+ +-----+
WLAN3,ES3---+-----+ | +--| H1 |--VLAN1
| | +-----+
| |
WLAN1,ES1---+-----+ | +-----+ | +-----+
WLAN2,ES2---| WTP |--+---| AC |---+--| H2 |--VLAN2
WLAN3,ES3---+-----+ | +-----+ | +-----+
| |
| |
WLAN1,ES1---+-----+ | | +-----+
WLAN2,ES2---| WTP |--+ +--| H3 |--VLAN3
WLAN3,ES3---+-----+ +-----+
Figure 4
Ahuja, et al. Expires July 5, 2008 [Page 9]
Internet-Draft WLAN Switch Benchmarking Methodology January 2008
3.3. Configuration parameters
The general DUT or SUT setup MUST follow the requirements described
in Section 7 of RFC 2544 [RFC2544].
The specific software or firmware version being used in the DUT or
the individual devices that make up the SUT, as well as the exact
device configuration(s) (including any functions that have been
disabled) MUST be reported together with the results.
3.3.1. WTP setup
This section is applicable only if the SUT includes WTPs (i.e., APs).
The WTPs in the SUT MUST be configured to use only the subset of
wireless channels available to a normal user at the location where
the system is intended to be used. For example, if the test is run
in the U.S., then standard U.S. wireless channels are used. The
channels used MUST be reported with the test results.
The 802.11 protocol supports the use of a Request To Send (RTS) /
Clear To Send (CTS) handshake prior to data transfer, as a means for
interfaces to seize and reserve the medium before actually
transferring data. For SUTs or WTPs with adjustable RTS thresholds,
tests MAY be run at different RTS thresholds, although a full suite
of tests MUST be run at the highest RTS threshold supported by the
SUT. The RTS threshold used MUST be reported with the test results.
The 802.11 protocol supports fragmentation and reassembly at the link
layer, in order to decrease retransmission overhead under high error
rates that may prevail in a radio frequency (RF) environment. For
SUTs or WTPs with adjustable fragmentation thresholds, tests MAY be
run at different fragmentation thresholds, although a full suite of
tests MUST be run at the highest fragmentation threshold supported by
the SUT. The fragmentation threshold used MUST be reported with the
test results.
Note that RTS/CTS and fragmentation are not used when transferring
multicast frames; they apply only to unicast frames.
3.3.2. Service priority
WLAN ACs frequently include the capability to classify traffic flows
and assign them to different service levels or service priorities.
For example, a voice flow may be classified as real-time traffic and
assigned a high level of service (e.g., with assured limits on delay
and loss), while an HTTP stream may be assigned a lower service
priority. Classification may also be made on the basis of DiffServ
Ahuja, et al. Expires July 5, 2008 [Page 10]
Internet-Draft WLAN Switch Benchmarking Methodology January 2008
code points (DSCP) or even 802.1D user priority values.
For DUTs or SUTs supporting multiple service priorities (QoS levels),
tests MAY be run at different service priorities, although a full
suite of tests SHOULD be run at least one service priority. For such
DUTs or SUTs, the service priority used in each test MUST be reported
with the test results.
Throughput and latency tests on WTPs involving traffic traversing
wired interfaces can be affected by QoS settings on these wired
interfaces. In such situations, the QoS settings assigned to the
wired interfaces of WTPs MUST be reported with the test results.
3.3.3. Test conditions
Test conditions for measurements on WLAN devices are covered in this
section. The complexity of the wireless LAN media and protocol
necessitate special attention to specifying and setting up these
conditions in order to obtain consistent results.
3.3.3.1. Test environment
This section is only applicable to SUTs containing WTPs.
Wireless LAN test environments may be divided into two general
categories: shielded environments and open-air (unshielded)
environments.
Shielded environments use cabling and/or RF shielding techniques to
significantly attenuate external signals and noise. The WTP(s) that
are part of the SUT are enclosed within an RF-tight chamber and
cabled to the tester as well as the AC or ACs. The tester is also
placed in an RF-tight chamber, or has an RF-tight enclosure. Such
environments are well known to provide the highest level of
repeatability and reproducibility, with the minimum amount of
complicated RF management and setup issues. Different types of
enclosures (shielded enclosures, screened rooms, and anechoic
chambers) may be employed to provide RF shielding.
Open-air environments mimic the actual use model of a WLAN DUT or
SUT. In this case, the WTP(s) that are part of the SUT are placed at
a specific location within some moderately controlled (or at least
well characterized) indoor or outdoor environment, and antennas are
used for coupling between equipment.
To assure the maximum level of repeatability and reproducibility
without complicated environment setup and characterization needs, the
tests described in this document MUST be carried out in a fully
Ahuja, et al. Expires July 5, 2008 [Page 11]
Internet-Draft WLAN Switch Benchmarking Methodology January 2008
shielded (conducted) test environment. The use of a fully shielded
environment eliminates the possibility of interference from
surrounding networks or devices emitting RF energy. The enclosures
and cabling used MUST provide a minimum of 80 dB of RF isolation.
Provided that the power levels are set as described below, the test
results are materially independent of the exact type of shielded test
environment and the properties of the enclosures used.
3.3.3.2. Power levels
This section is only applicable to SUTs containing WTPs.
Power levels are generally set by measuring and controlling the
Received Signal Strength Indication (RSSI) at the RF receivers within
the setup. The RSSI measured at the tester's receiver SHOULD be in
the range of -25 dBm to -35 dBm. Similarly, the RSSI measured at
each WTP's receiver SHOULD be in the range of -25 dBm to -35 dBm.
Passive attenuators SHOULD be used to control and set the RSSI within
these limits. The RSSI MUST be measured and reported with the test
results. The actual choice of WTP power level should not materially
affect the results of the tests described herein, provided that the
RSSI values fall within these limits.
If these power level settings are not used, then the tester MUST
ensure that the RF power levels (at the receivers of the WTP(s) and
tester) are at least 20 dB above the minimum and 10 dB below the
maximum levels specified for a 10% Frame Error Ratio (FER). Further,
the tester MUST ensure that the absolute signal level transmitted to
the DUT or SUT is held constant to within +/- 3 dB over the duration
of the trial.
3.3.3.3. Data plane test frame sizes
All of the data plane tests SHOULD be performed using several fixed
sizes of test data frames. Regardless of the interface type, frame
sizes MUST be calculated from the first byte of the MAC header to the
last byte of the FCS. The test results MUST list the frame sizes
used for test data frames.
MAC frame sizes change drastically when traffic moves from the 802.11
(WLAN) side to the 802.3 (Ethernet) side, and vice versa. Further,
the 802.11 MAC protocol specifies various mode-specific
encapsulations; for instance, the four currently defined encryption
modes (none, WEP, TKIP and AES-CCMP) result in four different 802.11
MAC frame sizes for the same size Ethernet frame, and enabling QoS on
the WLAN links adds another 2 bytes to the WLAN MAC header. Use of
CAPWAP or similar protocols on the wireless-facing side of an AC adds
an encapsulation header to the 802.11 MAC frame. This makes frame
Ahuja, et al. Expires July 5, 2008 [Page 12]
Internet-Draft WLAN Switch Benchmarking Methodology January 2008
size calculations and reporting of results quite challenging.
To avoid confusion, therefore, frame sizes reported with the test
results MUST be referenced to the Ethernet side of the DUT or SUT.
Further, the DUT or SUT configuration parameters such as encryption
mode and QoS, necessary to enable the equivalent WLAN-side frame size
to be calculated from the Ethernet side frame size, MUST be reported
as well.
The change in frame size also leads to confusion when interpreting
results expressed in bits/second or megabits/second. Test results
SHOULD therefore be expressed, where applicable, in frames/second of
a particular size on the Ethernet. The equivalent bits/second or
megabits/second values MAY be provided as well, as long as the test
reports include at least the values for the Ethernet interface and
clearly identify the interface(s) to which these values apply.
Note that all test data frames crossing the WLAN/Ethernet boundary
contain IP packets; in fact, due to the change in MAC encapsulation
when traversing this boundary, only the IP information will be
transported intact. Therefore, test results MAY also be expressed in
terms of IP packets/second of a particular size. As far as the test
data traffic is concerned, the number of IP packets per second on any
path traversing the DUT or SUT is exactly equal to the number of link
layer frames/second measured at each of the interfaces traversed by
that path.
Referenced to the Ethernet side, the MAC frame sizes that MUST be
used during data plane tests are:
88, 128, 256, 512, 1024, 1280, 1518
Note that the smallest frame size is 88 bytes, instead of the 64
bytes more commonly encountered in standard wired LAN benchmarks. A
larger frame size is selected in order to provide enough room in the
payload for an internetwork layer header, a transport layer header,
and a tag or signature field that SHOULD be inserted by the test
equipment to track the data traffic packets. An 88 byte MAC frame
size allows at least 26 bytes of payload for a signature field.
If the test equipment is capable of properly marking and tracking 64-
byte Ethernet frames (with or without 802.1Q VLAN tags present), then
64-byte MAC frames MUST be used during tests, in addition to the
frame sizes listed above.
If the wired interfaces of the DUT or SUT can support jumbo frames
(i.e., Ethernet frame sizes larger than 1518 bytes), then the
following MAC frame sizes MAY additionally be used:
Ahuja, et al. Expires July 5, 2008 [Page 13]
Internet-Draft WLAN Switch Benchmarking Methodology January 2008
2030, 2322
The maximum Ethernet MAC frame size of 2322 bytes ensures that the
re-encapsulated payload on the WLAN side does not exceed the maximum
IEEE 802.11 MAC payload size of 2304 bytes.
These frame sizes apply to Ethernet MAC frames irrespective of
whether they contain VLAN tags or not. If 802.1Q VLAN tagging is
used on the Ethernet side, then this MUST be reported with the test
results, as the consequence is to make the 802.11-side MAC frames 4
bytes smaller.
3.3.3.4. Control plane test frame sizes
The control plane tests described in this document also involve test
traffic data flows generated by the tester as part of the measurement
process. Unless otherwise specified, a frame size of 256 bytes
(referenced to the Ethernet side) MUST be used for these test data
flows. (This value is arbitrarily selected, but specified to ensure
reproducibility of tests.) If 802.1Q VLAN tagging is used, this will
cause the WLAN-side frame size to be smaller, and thus this MUST be
noted in the test results. It is not necessary to repeat control
plane tests with other MAC frame sizes, though this MAY be done if
desired.
Frame sizes of 802.11 management and control frames generated during
the test MUST conform to those required by the 802.11 standard
[802.11].
3.3.3.5. Frame formats and verification
The frame formats used for test data frames (with the exception of
null 802.11 frames) SHOULD follow the recommendations in Appendix C
of RFC 2544 [RFC2544]. LLC/SNAP encapsulation as per RFC 1042
[RFC1042] MUST be used on the WLAN side of the DUT or SUT, and Type-
encoded encapsulation as per IEEE 802.3 [802.3] MUST be used on the
Ethernet side.
In all cases, the test frame format MUST contain some means (such as
a unique signature field, as described in Section 4 of RFC 2544
[RFC2544]) that will enable the tester to filter out frames that are
not part of the offered load, or are duplicated by the DUT. In tests
on DUTs/SUTs involving multiple virtual or physical test endstations,
the test frame format MAY also support means for distinguishing
between frames originating from different endstations.
The provisions for verifying received frames in Section 10 of RFC
2544 [RFC2544] SHOULD be followed as well. This is particularly
Ahuja, et al. Expires July 5, 2008 [Page 14]
Internet-Draft WLAN Switch Benchmarking Methodology January 2008
significant for test setups where the tester connects to an 802.11
wireless interface, as 802.11 implements retransmission at the link
layer. The verification of received frames SHOULD be independent of
the facilities provided by the MAC, IP and TCP/UDP layers. In the
case of data plane tests, the tester MAY support an independent means
of verifying the absence of end-to-end payload corruption, such as a
checksum or CRC calculated over the payload portion of the data
frames.
3.3.3.6. Addressing
To ensure independence of test results relative to address patterns,
the test setup SHOULD follow the recommendations in RFC 4814
[RFC4814], to create pseudorandom MAC and IP address patterns.
However, as noted in Section 4.2.1 of RFC 4814, ACs, particularly
those implementing ARP proxies and spoofing protection, can reject
endstations that change their MAC and/or IP address mappings from
trial to trial. To avoid this, the address mappings used MUST NOT
change between trials or between consecutive tests that are performed
without sufficient time for the context entries in the AC to age out.
DHCP MAY be used to provide IP addresses to endstations in data plane
tests. If this is done, then the test results MUST note that DHCP
was being used. DHCP MAY NOT be used in control plane tests unless
specifically required as one of the test conditions. Note that many
ACs require DHCP to be used to provide IP addresses to WTPs; in this
situation, a DHCP server is needed for this purpose. If all of the
WTPs (and/or all of the endstations) cannot be located within one
subnet, then DHCP helper addresses MUST be configured to allow the
use of a single DHCP server.
In multicast tests, multicast IP addresses MUST be drawn from the
Class D address pool and multicast MAC addresses MUST correspond to
the Class D IP addresses, as described in RFC 3918 [RFC3918] and RFC
1112 [RFC1112]. IGMP MUST be active, and the IGMP version being used
MUST be reported with the test results.
3.3.3.7. Traffic flows and topologies
Traffic flows used in the data and control plane tests transit the AC
within the DUT or SUT; that is, they are either from the Ethernet
side to the wireless side, from the wireless side to the Ethernet
side, or in both directions. Traffic flows that are purely Ethernet
or purely wireless-side are outside the scope of this document. Note
that wireless-to-wireless test metrics and procedures are already
covered in IEEE 802.11.2.
Fully-meshed traffic topologies, per section 3.3.3 of RFC 2285
Ahuja, et al. Expires July 5, 2008 [Page 15]
Internet-Draft WLAN Switch Benchmarking Methodology January 2008
[RFC2285] are not applicable to wireless testing; virtually no
unicast traffic is sent directly between wireless devices. Instead,
one-to-many or many-to-one topologies (per section 3.3.2 of RFC 2285
[RFC2285]) are used. In addition, the partially-meshed one-to-many/
many-to-one topology is commonly used for forwarding, latency, and
QoS tests.
In the case of multicast traffic flows, the most common case is a
traffic flow direction from Ethernet to wireless (i.e., downstream
relative to the endstations) using a one-to-many topology. New
applications such as push-to-talk for VoIP over WLAN handsets also
require many-to-many traffic flow topologies; however, the tests
described in this document do not yet cover such situations.
3.3.3.8. Half-duplex effects
WLAN WTPs and endstations perform medium access in half-duplex mode,
which can cause the actual offered load to be less than the intended
load imposed by the tester. The tester MUST therefore adjust the
inter-frame spacing according to the target intended load (i.e., to
achieve the desired rate of frame transmission), and then MUST
measure and report the actual offered load at the end of the trial.
Appendix A of this document provides some notes about generating the
intended load for tests described herein. Either the frame-based or
the time-based method described in Appendix B of RFC 2889 [RFC2889]
MAY be used, but, in either case, the method used MUST be reported
with the results. Most of the tests in this document use a constant
(non-bursty) load, and the Iload calculations in Section A.2 apply.
Burst loads use the calculations in Section A.3.
The tester SHOULD note attempts by the DUT or SUT to violate the
timing requirements of the 802.11 protocol by not conforming to the
backoff rules, or by reducing its inter-frame spacing to less than
the legal minimum.
3.3.3.9. Wireless physical layer (PHY) settings
This section is only applicable to SUTs containing WTPs.
The physical layer of the 802.11 WLAN protocol supports data transfer
(PHY data rate) at a number of different bit rates. For instance,
the 2.4 GHz OFDM PHY layer (formerly referred to as 802.11g) uses bit
rates of 1, 2, 5.5, 6, 9, 11, 12, 18, 24, 36, 48 and 54 Mb/s. These
data rates are achieved with different modulation formats, generally
resulting in different frame error ratios and/or signal-to-noise
ratios at the receiver. Tests performed at one data rate may not
correlate with tests performed at another rate. The test results
Ahuja, et al. Expires July 5, 2008 [Page 16]
Internet-Draft WLAN Switch Benchmarking Methodology January 2008
MUST therefore record the data rate used by both the DUT or SUT and
the tester. At least one trial SHOULD be performed at the highest
PHY data rate supported by the DUT or SUT.
Rate adaptation is supported by 802.11 devices in order to transfer
unicast data under changing signal conditions. (Multicast frames are
transmitted by each WTP at a fixed default rate, which is usually the
highest basic rate configured.) Devices can automatically fall back
to lower PHY data rates to cope with decreasing signal-to-noise
ratios. This can make test results impossible to reproduce or
interpret in the case of measurements needing constant PHY data
rates. A given trial MUST maintain a constant PHY data rate for all
test data packets presented on the wireless side of the DUT or SUT,
unless otherwise specified by the test.
The 802.11 WLAN protocol uses retransmissions to compensate for the
relatively high frame error ratio on the wireless medium. This
effectively trades goodput for reliable data transfer. Frame errors
rise sharply as the wireless signal-to-noise ratio degrades. Hence
the performance of the 802.11 wireless link is materially affected by
the signal levels at the receivers of both the DUT or SUT and the
tester. The test results therefore MUST report the average RSSI
(Received Signal Strength Indication) measured at the wireless
interfaces of the tester during the test.
3.3.3.10. Security settings
WLAN DUTs/SUTs typically implement a wide variety of authentication
and encryption protocols, and it is of interest to characterize their
performance when using these protocols. Examples of authentication
protocols include preshared key (PSK), EAP-TLS, PEAP, LEAP, EAP-MD5,
EAP-MSCHAP and EAP-TTLS. Examples of encryption protocols include
WEP-40, WEP-128, TKIP and AES-CCMP.
Transfer of data between stations in 802.11 is permitted to begin
only after a successful connection setup, including capabilities
negotiation, user authentication, and (usually) installation of
encryption keys. Also, as per the IEEE 802.11 standard, data must
not be transmitted after a connection has been terminated.
For data plane tests, the tester MUST authenticate the virtual or
physical endstations used for each test with the DUT or SUT prior to
the start of each trial, using the appropriate security method. It
SHOULD perform a endstation connection handshake with the DUT or SUT
at the start of each trial, and a deauthentication handshake at the
conclusion. However, it MAY elect to remain connected with the DUT
across multiple trials, in which case the deauthentication handshake
can be performed at the end of the test.
Ahuja, et al. Expires July 5, 2008 [Page 17]
Internet-Draft WLAN Switch Benchmarking Methodology January 2008
Disassociation of a virtual or physical endstation by the DUT or SUT
during the test data transfer portion of a trial MUST be reported and
SHOULD cause the trial to be terminated. Further, the tester MUST
NOT count as valid any unicast data frame from the DUT or SUT that
arrives for a virtual or physical endstation when the latter is not
completely authenticated.
For a number of control plane tests, the connection handshake forms
an integral part of the test procedure, and the requirements for
authentication and connection setup are described in the specific
tests.
To provide a baseline, a full suite of tests SHOULD be run with no
encryption and no authentication enabled.
3.3.3.11. Multiple endstations
To characterize the context-handling capabilities of the DUT or SUT
datapaths, measurements of frame loss, throughput, forwarding rate,
latency and burst capacity SHOULD be performed with multiple
endstations on either the wireless side, the wired side, or both. If
used, the multiple endstations MUST be concurrently transmitting
traffic.
This simulates the situation in an actual network, where multiple
endstations and servers are connected to a single DUT or SUT and
contribute to the total offered load. Each endstation is represented
by a different MAC/IP address combination. The MAC and IP addresses
of the different endstations SHOULD be chosen according to the
preceding section on addressing.
The number of endstations used in the test MUST be reported. For
throughput, frame loss, forwarding rate and burst capacity tests, the
aggregate results for all of the endstations combined MUST be
reported. For latency tests, the worst-case latency and smoothed
interarrival jitter among all of the endstations MUST be reported.
To provide a baseline, a full suite of tests SHOULD be run with a
single endstation on the wireless side and a single endstation on the
Ethernet side.
3.3.3.12. Multiple virtual WLANs
Virtually all enterprise WLAN infrastructure equipment allows the
configuration of multiple overlay (or "virtual") WLANs on the
wireless topology. These virtual WLANs are distinguished by
different SSIDs being supported on each WTP and AC; on any given WTP,
each SSID corresponds to a separate BSSID and has a separate beacon
Ahuja, et al. Expires July 5, 2008 [Page 18]
Internet-Draft WLAN Switch Benchmarking Methodology January 2008
stream associated with it. The virtual WLANs serve the same purpose
as VLANs in Ethernet networks, namely, segregation, security and
management. Each virtual WLAN usually (but not necessarily) maps to
a separate VLAN or subnet on the Ethernet side of the DUT or SUT.
The different virtual WLANs may have different security, QoS,
endstation capacity, radio characteristics, bandwidth limits, and
other parameters. The virtual WLAN configuration of the DUT or SUT
therefore has a material impact on the performance results obtained.
In all cases, the baseline measurement MUST be made with a single
virtual WLAN (i.e., a single SSID) configured on the DUT or SUT.
Once the baseline has been measured, additional measurements MAY be
carried out with more than one virtual WLAN being configured. The
virtual WLAN setup (including security and QoS parameters) MUST be
detailed in the test results.
3.3.3.13. Mobility
The common case of mobility (roaming) is one or more endstations that
move between WTPs within the same IP subnet, thus keeping the roaming
process transparent to the IP protocol layer and above. However,
many DUTs/SUTs allow WLAN endstations to roam between WTPs that are
physically located on different subnets, without requiring the
roaming endstation to obtain a new IP address via DHCP or other
means. This 'inter-subnet roaming' is actually carried out by a
proxying process; the WTP to which the endstation roams acts as a
'remote agent' for the WTP from which the endstation has roamed, with
tunnels being used to transport the endstation's data packets back to
the subnet to which it belongs.
Tests of mobility SHOULD therefore include both inter-subnet roaming
as well as intra-subnet roaming. The two cases MUST be tested
separately, and the results reported separately.
In addition, roaming may also be divided into intra-AC and inter-AC
roaming scenarios (as described in the accompanying terminology
document), provided that the DUT or SUT contains multiple ACs. There
are hence four possible roaming test scenarios:
o intra-subnet, intra-AC
o intra-subnet, inter-AC
o inter-subnet, intra-AC
o inter-subnet, inter-AC
Ahuja, et al. Expires July 5, 2008 [Page 19]
Internet-Draft WLAN Switch Benchmarking Methodology January 2008
Of these four scenarios, the first (intra-subnet, intra-AC) MUST be
tested as the baseline case. The other three scenarios SHOULD also
be tested if the DUT or SUT supports the relevant functions. The
results for each scenario MUST be reported separately.
Data plane tests MUST NOT involve roaming behavior (i.e., endstations
moving between WTPs) because roaming causes irreproducible traffic
delays and interruptions. Roaming behavior MUST occur only in the
tests specifically indicated as roaming tests.
When conducting roaming tests, the tester MUST distinguish between
the tester-imposed delay and the DUT or SUT imposed delay. For
example, the time delay between the receipt of a connection handshake
frame by the tester during the reassociation process, and the
transmission of the corresponding response frame, is tester-imposed
delay. The two kinds of delays MUST be reported separately, and the
tester-imposed delay SHOULD be subtracted from the overall roaming
delay when reporting the overall roaming delay of the DUT or SUT.
The tester MUST transmit learning frames after each roam event by an
endstation. Learning frames are necessary to update the end-to- end
datapath and cause data traffic to resume expeditiously. The time
delay between the completion of the connection handshake and the
first learning frame MUST NOT be counted as part of the roaming delay
attributed to the DUT or SUT.
The 802.11 protocol contains numerous shortcuts and enhancements
(e.g., preauthentication, PMKID caching, and opportunistic key
caching) that are designed to speed up the roaming process. Most of
these features are optional. The tester SHOULD implement as many of
these optional features as possible, to enable the roaming
performance of the DUT or SUT to be measured under different
scenarios. Any optional speed-up features that have been used in a
test MUST be reported with the results.
3.3.3.14. Trial duration
The duration of each trial SHOULD be selected using the guidelines of
Section 24 of RFC 2544 [RFC2544]. Further, it SHOULD be long enough
to minimize any connection setup and startup effects that can affect
the test results. In the case of tests involving WTPs (i.e., APs),
the trial duration SHOULD also be long enough to make the random
fluctuations of the CSMA/CA access method statistically
insignificant.
The recommended duration of each trial is 60 seconds. The trial
duration MAY be adjustable between 30 seconds and 300 seconds. The
tester MUST transmit all test data frames within the trial duration.
Ahuja, et al. Expires July 5, 2008 [Page 20]
Internet-Draft WLAN Switch Benchmarking Methodology January 2008
To eliminate the case where a device possessing large frame buffers
can appear to be faster than it actually is, the tester MUST NOT
accept received test data frames for more than 2 seconds beyond the
end of the trial duration. (Thus a 60 second trial duration causes
the tester to receive frames for no more than 62 seconds, starting
from the beginning of the trial.) Frames received outside of these
limits MUST NOT be counted as part of the results.
The trial duration MUST NOT include the time taken for initial
connection setup and system state stabilization, unless this is
specifically part of the test. For example, authentication and
association of wireless clients can take a relatively long time; if
this is included within the trial duration of a throughput test, the
results will be significantly affected. This also applies to
spanning tree convergence and other connection state settling delays.
4. Interpreting and reporting test results
Test results SHOULD be reported in a common format to aid the reader
in interpreting results and comparing them across DUTs. Results from
a set of trials involving the variation of one or more test
parameters described in Section 3 above SHOULD be presented as
graphs, with the x coordinate being the parameter value and the y
coordinate being the result. Detailed results SHOULD be presented in
tabular format to simplify analysis.
The following test conditions MUST be reported with the results of
each trial:
Tester and WTP signal level, PHY bit rate, PHY layer options, and
channel used (if WTPs form part of the SUT).
Security modes (encryption and authentication).
Trial duration.
Number of endstations used in the test.
Frame size and offered load.
5. Benchmarking tests
The following tests are divided into two categories: data plane tests
and control plane tests.
Data plane tests relate to the performance of the traffic handling
Ahuja, et al. Expires July 5, 2008 [Page 21]
Internet-Draft WLAN Switch Benchmarking Methodology January 2008
functions of the DUT or SUT; the results of such tests are mainly
governed by the packet forwarding hardware and software. Control
plane tests, on the other hand, stress the connection setup and
context management capabilities of the DUT or SUT, and the results of
these tests are dictated by the performance of the CPU(s) and front-
end traffic classification and exception handling mechanisms.
The correlation between system performance and data plane metrics
such as throughput and latency is well known, of course. The
performance of the management and security protocols in a WLAN,
however, are also key determinants of the perceived user experience.
For example, support of 802.11-based VoIP handsets is of significant
interest in an enterprise environment; however, poor roaming
performance can make such handsets nearly unusable except in a fixed
usage model. Thus it is important to quantify control plane metrics
as well.
This section treats data plane and control plane tests separately.
Objectives, test parameters, procedures and reporting formats are
described for each test.
5.1. Data plane tests
Data plane tests comprise the following:
Unicast and multicast throughput and forwarding rate
Latency and jitter
QoS differentiation
Burst capacity
Power-save throughput
Throughput and forwarding rate tests have an obvious correlation to
end-user perceptions of the speed and capacity of the wireless LAN.
For throughput and forwarding rate tests, either the Frame Based or
Time Based modes of testing may be used, as described in Appendix B
of RFC 2889 [RFC2889]. The DUT or SUT is initially set up according
to the baseline configuration, using a starting combination of test
parameters. Packets are then sent to the DUT or SUT by the tester at
a specific offered load for the duration of the trial, and the number
of frames received from the DUT or SUT are counted. The process MUST
be iterated at different offered loads, using a search algorithm,
until the desired measurement (throughput or maximum forwarding rate)
has been made. Additional trials are then performed in the same
manner using different DUT or SUT configurations until all
Ahuja, et al. Expires July 5, 2008 [Page 22]
Internet-Draft WLAN Switch Benchmarking Methodology January 2008
configurations have been exhausted.
Latency and jitter tests are principally of interest in delay
sensitive applications such as voice and video. Jitter tests use the
smoothed interarrival jitter calculation method described in RFC 3550
[RFC3550]. A means of timestamping transmitted frames is required in
order to calculate both latency and jitter.
QoS differentiation tests seek to quantify the performance of the DUT
or SUT when handling traffic such as VoIP that requires preferential
treatment over best-effort data traffic. This is particularly
important in WLANs, not only because of the growing use of Voice over
WLAN (VoWLAN) handsets, but also because of the limited bandwidth
available over the wireless medium. (Wired enterprise LANs have
heretofore dealt with the QoS problem by throwing bandwidth at the
problem, as QoS is mainly an issue in oversubscribed links; due to
spectrum limitations this pleasantly simple approach is not simple or
practicable in a wireless context.)
Burst capacity and power-save throughput deal with the unique need
for WLAN infrastructure devices to buffer considerable amounts of
data, in order to compensate for unpredictable variations in the
links to wireless endstations. The wireless media may experience
rapid variations in capacity due to rate adaptation, requiring the
infrastructure devices to buffer packets until higher- layer
protocols such as TCP can 'catch up'. In addition, wireless
endstations themselves may elect to enter a low-power mode in order
to extend battery life, again requiring the WLAN infrastructure to
buffer packets until the endstation wakes up and calls for the data.
As the loss of even a few packets has a substantial impact on upper
layer protocols such as TCP, the burst capacity and power-save
throughput of the DUT or SUT can significantly affect the end-user
experience.
In all of the data plane tests, the tester MUST count as valid
received test frames only those which it receives without error
within the testing time window, with the proper signature, and
correctly directed (i.e., having the right combination of source and
destination addresses, frame length and payload). All other frames
(including management/control frames) MUST NOT be included when
computing the test results.
For the purposes of computing the actual offered load, the tester
MUST count as valid transmitted frames only those test frames that
were acknowledged by the DUT on the wireless medium (i.e., with an
802.11 ACK frame), or transferred to the DUT or SUT without a locally
detected error on the wired medium within the testing time window.
All other frames MUST NOT be counted as part of the offered load.
Ahuja, et al. Expires July 5, 2008 [Page 23]
Internet-Draft WLAN Switch Benchmarking Methodology January 2008
Note that this is only applicable to tests involving WTPs.
In addition, in tests involving WTPs the tester must only count as
valid those unique data frames for which it sent an 802.11 ACK frame
to the DUT or SUT in response. It MUST NOT count duplicate frames,
frames originating from the DUT, data frames that it did not
acknowledge, or management and control frames as part of the
measurements. Such frames MAY be counted separately for diagnostic
purposes, or not counted at all.
5.1.1. Unicast throughput
5.1.1.1. Objective
To determine the throughput of the DUT or SUT when forwarding unicast
data frames between the wireless and the wired sides of the DUT or
SUT. The results of this test can be used to determine the ability
of an AC to support multiple wireless endstations transferring data
to a wired LAN segment.
Note that while the IEEE 802.11 wireless medium has a high frame
error ratio relative to wired media, the low-level acknowledgement
and retransmission protocol implemented by the 802.11 MAC effectively
results in nearly zero loss as seen by the IP layer. In fact the
loss ratio at the MAC service interface of an 802.11 device is no
worse than the loss ratio for a wired Ethernet device. (Obviously a
high frame error ratio on the wireless medium will then manifest
itself as a reduced throughput at the MAC service interface, but the
signal level precautions described in section 3.4.8 will ensure that
the maximum possible throughput is obtained.)
The general setup for the test comprises one or more virtual or
physical endstations on the wireless side of the DUT or SUT that
transfer data to or from one or more virtual endstations on the wired
side.
5.1.1.2. Test parameters
The following configuration parameters MUST be established prior to
each trial, as per the requirements in section 3 above:
Frame size, Flow direction, Number of wireless and Ethernet
endstations, Fragmentation, RTS/CTS usage, Security mode and
Number of virtual WLANs.
The baseline DUT or SUT configuration for performing this test
consists of: frame sizes as per section 3.4.2, downstream transfer
direction, a single wireless endstation, a single Ethernet
Ahuja, et al. Expires July 5, 2008 [Page 24]
Internet-Draft WLAN Switch Benchmarking Methodology January 2008
endstation, fragmentation off, RTS/CTS disabled, security not used,
and a single virtual WLAN.
5.1.1.3. Procedure
The DUT or SUT is first set up according to the baseline
configuration, using the initial setting of frame size. The initial
offered load is computed per Appendix A to equal the aggregate
theoretical maximum capacity of all the wireless-side links. For
bidirectional tests involving WTPs, the tester MUST follow the half-
duplex test conditions described in Section 3.4.7. The throughput is
then measured as described below. The measurement is repeated for
each value of frame size.
The tester MUST send learning frames (after endstation connection
setup as applicable) to allow the DUT or SUT to update its address
tables properly. A search algorithm is used to determine the
throughput.
After the baseline configuration has been tested, the tester MAY
repeat the process with a new configuration, until the desired number
of different configurations have been exercised.
In tests involving multiple endstations (either on the wireless or
the wired side, or both), the tester MUST ensure a uniform
distribution of frames from each source endstation. In addition, all
permissible combinations of source and destination addresses
(consistent with the traffic direction setting) MUST be represented
equally within each trial. This distributes the load of transmission
and reception uniformly among the endstations. Note that this
corresponds closely to the partially-meshed one-to-many/many-to-one
topology described in RFC 2889 [RFC2889].
5.1.1.4. Analysis and reporting
The throughput of the DUT or SUT is computed and reported (per
Section 26.1 of RFC 2544 [RFC2544]) as the maximum offered load, in
frames per second, resulting in zero frame loss rate [RFC1242].
The test results SHOULD be reported as graphs of throughput versus
frame size. Separate results MUST be reported per configuration.
5.1.2. Unicast maximum forwarding rate and frame loss ratio
5.1.2.1. Objective
To determine the maximum rate at which the DUT or SUT can forward
unicast data frames between the wireless and the wired sides of the
Ahuja, et al. Expires July 5, 2008 [Page 25]
Internet-Draft WLAN Switch Benchmarking Methodology January 2008
DUT or SUT, irrespective of frame loss. This is a measure of the
peak capacity of the DUT or SUT datapath, and is especially useful in
conjunction with traffic such as UDP. The frame loss ratio is also
measured under the same conditions.
The general setup for the test comprises one or more virtual or
physical endstations on the wireless side of the DUT or SUT that
transfer data to or from one or more virtual endstations on the wired
side.
5.1.2.2. Test parameters
The following parameters MUST be configured prior to each trial as
specified in Section 3.5.14:
Frame size, Flow direction, Number of wireless and Ethernet
endstations, Fragmentation, RTS/CTS usage, Security mode and
Number of virtual WLANs.
The baseline DUT or SUT configuration for performing this test
consists of: frame sizes as per section 3.4.2, downstream transfer
direction, a single wireless endstation, a single Ethernet
endstation, fragmentation off, RTS/CTS disabled, security not used,
and a single virtual WLAN.
5.1.2.3. Procedure
The DUT or SUT is first set up according to the baseline
configuration, using an initial frame size. The starting offered
load is computed per Appendix A. The maximum forwarding rate and
frame loss ratio are then measured as described below. The
measurements are repeated for each value of frame size.
The tester MUST send learning frames (after endstation connection
setup as applicable) to allow the DUT or SUT to update its address
tables properly. A search algorithm SHOULD be used to determine the
maximum forwarding rate.
After the baseline configuration has been tested, the tester MAY
repeat the process with a new configuration, until the desired number
of different configurations have been exercised.
In tests involving multiple endstations (either on the wireless or
wired side, or both), the tester MUST ensure a uniform distribution
of frames from each source endstation. In addition, all permissible
combinations of source and destination addresses (consistent with the
traffic direction setting) MUST be represented equally within each
trial. This distributes the load of transmission and reception
Ahuja, et al. Expires July 5, 2008 [Page 26]
Internet-Draft WLAN Switch Benchmarking Methodology January 2008
uniformly among the endstations. Note that this corresponds closely
to the partially-meshed one-to-many/many-to-one topology described in
RFC 2889 [RFC2889].
5.1.2.4. Analysis and reporting
The maximum forwarding rate of the DUT or SUT is computed and
reported as the maximum number of test frames per second that the DUT
or SUT is observed to successfully forward, irrespective of frame
loss, at some value of offered load. The offered load applied to the
DUT or SUT at the maximum forwarding rate MUST be reported as well.
The frame loss ratio MUST be reported with the maximum forwarding
rate, as the percentage of frames that were successfully injected
into the DUT or SUT by the tester but not forwarded by the DUT or SUT
to the tester for any reason.
The test results SHOULD be reported as a graph of maximum forwarding
rate versus frame size. Separate results MUST be reported per
configuration.
If the maximum forwarding rate of a SUT (containing WTPs) exceeds the
theoretical maximum medium capacity of the wireless LAN medium, then
the SUT is departing from the DCF contention behavior specified by
the IEEE 802.11 MAC protocol. In this case, Forward Pressure (as
defined in 3.7.2 of RFC 2285 [RFC2285]) has been detected, and MUST
be highlighted in the test results. The calculation of theoretical
maximum medium capacity MUST account for the effects of QoS settings,
if QoS is enabled.
Note that the wired-side interfaces of the DUT or SUT are often
capable of much higher link rates than the wireless side, potentially
leading to extremely high frame loss rates when oversubscription
occurs on the wired interfaces. Care should be taken to allow enough
time for SUTs that include WTPs to recover and return to a normal
state between trials.
5.1.3. Multicast forwarding rate
5.1.3.1. Objective
To determine the maximum rate at which the DUT or SUT can forward
multicast data frames. As multicast (or broadcast) traffic is dealt
with differently from unicast traffic by the 802.11 protocol, this
test therefore determines the ability of the DUT or SUT to handle
such traffic.
This test is run only in a downstream (wired to wireless) direction,
Ahuja, et al. Expires July 5, 2008 [Page 27]
Internet-Draft WLAN Switch Benchmarking Methodology January 2008
as wireless endstations do not normally generate high volumes of
multicast data. The general setup comprises at least one source
(virtual or physical endstation) on the wired side of the DUT or SUT
that injects multicast data destined for the wireless side, as well
as at least one virtual or physical endstation on the wireless side
that acts as a recipient for multicast traffic.
Note that the 802.11 protocol does not make special provisions for
multicast versus broadcast traffic. A single test is thus sufficient
to measure the ability of DUTs/SUTs to handle both types of data.
5.1.3.2. Test parameters
The following parameters MUST be configured prior to each trial as
specified in Section 3.5.14:
Frame size, Number of wireless endstations (multicast sinks),
Number of Ethernet endstations (multicast sources), Security mode
and Number of virtual WLANs.
The baseline DUT or SUT configuration for performing this test
consists of: frame sizes as per 3.4.2, a single wireless endstation,
a single Ethernet endstation, fragmentation off, RTS/CTS disabled,
security not used, and a single virtual WLAN.
5.1.3.3. Procedure
The DUT or SUT is first set up according to the baseline
configuration, using an initial value of frame size. The starting
offered load is computed per Appendix A to equal the theoretical
maximum capacity of the wireless-side link with the lowest PHY bit
rate. The maximum multicast forwarding rate is then measured as
described below. The measurements are repeated for each value of
frame size.
A trial is considered to be successful if each target (wireless)
endstation receives at least 50% of the multicast frames expected to
be received during that trial. For example, if 1000 multicast frames
are transmitted during a trial, and there are 8 wireless endstations
configured on 8 WTPs, then each wireless endstation must receive at
least 500 multicast frames for the trial to report that the injected
frames were successfully forwarded.
After the baseline configuration has been tested, the tester MAY
repeat the process with a new configuration, until the desired number
of different configurations have been exercised.
The target multicast addresses (MAC and IP) MUST be configured as
Ahuja, et al. Expires July 5, 2008 [Page 28]
Internet-Draft WLAN Switch Benchmarking Methodology January 2008
described in section 3.4.5 above. The addresses used MUST be
reported with the results.
5.1.3.4. Analysis and reporting
The maximum multicast forwarding rate of the DUT or SUT is computed
and reported as the maximum number of test frames per second that the
DUT or SUT is observed to successfully forward, irrespective of frame
loss, at some value of offered load. The offered load applied to the
DUT or SUT at the maximum forwarding rate MUST be reported as well.
A search algorithm SHOULD be used to determine the maximum multicast
forwarding rate.
The worst-case frame loss ratio MUST be reported along with the
maximum multicast forwarding rate, as the percentage of frames that
were injected into the DUT or SUT by the tester, but not forwarded by
the DUT or SUT to the tester for any reason. The worst-case frame
loss ratio is obtained by calculating the frame loss ratio at each
target wireless endstation, and taking the maximum.
The test results SHOULD be reported as a graph of maximum forwarding
rate versus multicast frame size. Separate results MUST be reported
per configuration.
Note that the wired interfaces of the DUT or SUT are often capable of
much higher link rates than the wireless interfaces, potentially
leading to extremely high frame loss rates when transferring
multicast frames to the wireless media. Care should be taken to
allow enough time for the DUT or SUT to recover and return to a
normal state between trials.
5.1.4. Latency and jitter
5.1.4.1. Objective
To determine the latency and latency variation (also known as jitter)
exhibited by the DUT or SUT when forwarding unicast data frames
between the wired and wireless sides of the DUT or SUT. The results
of this test can be used to estimate the impact of the DUT or SUT on
delay-sensitive traffic to/from a wireless endstation such as a VoIP
handset.
The general setup for the test comprises one or more virtual or
physical endstations on the wireless side of the DUT or SUT that
transfer data to/from one or more virtual endstations on the wired
side.
Ahuja, et al. Expires July 5, 2008 [Page 29]
Internet-Draft WLAN Switch Benchmarking Methodology January 2008
5.1.4.2. Test parameters
The following parameters MUST be configured prior to each trial as
specified in Section 3.5.14:
Frame size, Flow direction, Number of wireless and Ethernet
endstations, Fragmentation, RTS/CTS usage, Security mode and
Number of virtual WLANs.
The baseline DUT or SUT configuration for performing this test
consists of: frame sizes as per 3.4.2, downstream transfer direction,
a single wireless endstation, a single Ethernet endstation,
fragmentation off, RTS/CTS disabled, security not used, and a single
virtual WLAN.
5.1.4.3. Procedure
The DUT or SUT is initially set up according to the baseline
configuration, and data are transmitted to it by the tester at a
constant load for the duration of the trial. The offered load
presented to the DUT or SUT MUST be less than or equal to the
measured throughput of the DUT or SUT, and SHOULD be set to 90% of
its unicast throughput as measured under the same test conditions and
with the same configuration parameters.
The latency and jitter introduced by the DUT or SUT are measured over
the entire trial duration, as described below. An identifying tag or
signature MUST be placed in each data frame sent to the DUT or SUT
during the measurement interval, so that it can be correlated with
the frames received from the DUT. The measurements are repeated for
each value of frame size.
The tester MUST send learning frames (after endstation connection
setup as applicable) to allow the DUT or SUT to update its address
tables properly. If multiple endstations are used, the traffic
topology MUST be of the partially meshed one-to-many/many-to-one
type.
After the baseline configuration has been tested, the tester MAY
repeat the process with a new configuration, until the desired number
of different configurations have been exercised.
When testing with multiple endstations on the wireless side and/or
Ethernet side, consecutive frames transmitted by the tester to the
DUT or SUT MUST have different combinations of source and destination
addresses, and all possible such combinations of addresses MUST be
represented equally within each trial. This distributes the delay
impact and traffic load uniformly among the endstations. Failure to
Ahuja, et al. Expires July 5, 2008 [Page 30]
Internet-Draft WLAN Switch Benchmarking Methodology January 2008
ensure this can lead to inconsistent results.
5.1.4.4. Analysis and reporting
The instantaneous latency of the DUT or SUT is measured (per section
26.2 of RFC 2544 [RFC2544]) as the difference, in seconds, between
the timestamps assigned to a frame transmitted to the DUT or SUT and
the corresponding frame received from the DUT. The minimum, maximum
and mean of these differences in timestamps over all the data frames
received from the DUT or SUT during the trial duration are computed
and reported as the average latency introduced by the DUT.
The smoothed interarrival jitter introduced by the DUT or SUT is
calculated over the entire trial duration according to the algorithm
in section 6.4.1 of RFC 3550 [RFC3550]. (See Appendix A.8 of this
document for an example.)
The offered load over the trial duration MUST be reported as well.
The test results SHOULD be reported as graphs of latency and jitter
versus frame size. Separate results MUST be reported per
configuration.
5.1.5. QoS differentiation
5.1.5.1. Objective
The limited capacity of the wireless medium makes it imperative to
implement effective QoS schemes in order to successfully support
delay and loss sensitive traffic such as voice and video. Many ACs
and WTPs implement extensive classification and prioritization
functions to ensure that high levels of best-effort data traffic do
not adversely impact the delay, jitter and packet loss of higher
priority real-time traffic.
This test therefore seeks to quantify the level to which the DUT or
SUT isolates real-time traffic from best-effort data traffic that is
sharing the same channel. This is done by injecting progressively
higher levels of best-effort traffic into the DUT or SUT until the
QoS requirements of a previously established real-time stream are no
longer met. Ideally, the DUT or SUT will not permit the best-effort
traffic stream to affect the real-time stream, regardless of the
offered load of the former. A poor QoS implementation will result in
the QoS requirements of the real-time stream being violated at
relatively low levels of best-effort traffic. The results of this
test can hence be used to estimate the effectiveness of the QoS
implementation within the DUT or SUT datapaths.
Ahuja, et al. Expires July 5, 2008 [Page 31]
Internet-Draft WLAN Switch Benchmarking Methodology January 2008
The general setup for the test comprises one or more virtual or
physical endstations on the wireless side of the DUT or SUT that
transfer data to/from one or more virtual endstations on the wired
side. Traffic flows of different types are injected in order to
exercise the QoS handling functions of the DUT. This test is only
applicable to DUTs or SUTs that are capable of recognizing and
prioritizing real-time traffic.
5.1.5.2. Test parameters
The following parameters MUST be configured prior to each trial as
specified in Section 3.5.14:
Frame size (for best-effort data), Number of wireless and
Ethernet endstations, Fragmentation, RTS/CTS usage, Security mode
and Number of virtual WLANs.
The baseline DUT or SUT configuration for performing this test
consists of: frame sizes as per 3.4.2, a single wireless endstation,
a single Ethernet endstation, fragmentation off, RTS/CTS disabled,
security not used, and a single virtual WLAN.
5.1.5.3. Procedure
The DUT or SUT is initially set up according to the baseline
configuration, and the wireless endstations are associated with it.
A constant bidirectional stream of real-time traffic is injected into
the DUT or SUT (i.e., one stream from the wireless side to the
Ethernet side and an identical stream from the Ethernet side to the
wireless side).
The real-time traffic frames MUST be formatted to ensure that the DUT
or SUT assigns them a higher priority than normal best-effort data
frames. The frame size and frame rate of the real-time traffic MUST
resemble a single G.711 voice stream (240 byte RTP payloads, 30
frames/second) as closely as possible. The maximum latency, smoothed
interarrival jitter and packet loss MUST be measured for each
direction of each real-time traffic stream over the entire trial
duration.
The tester MUST then inject a stream of best-effort data traffic into
the Ethernet side of the DUT or SUT, directed at the wireless
endstations, using an initial frame size and starting offered load
(computed per Appendix A), and running for the entire trial duration.
The tester MUST monitor the QoS parameters of the real-time traffic
over the trial duration. A search algorithm SHOULD be used to find
the highest value of best-effort offered load (up to and including
the theoretical maximum capacity of the Ethernet medium, minus the
Ahuja, et al. Expires July 5, 2008 [Page 32]
Internet-Draft WLAN Switch Benchmarking Methodology January 2008
bandwidth occupied by the real-time traffic) for which the QoS
threshold is not violated.
The measurements are repeated for each value of frame size.
Identifying tags or signatures MUST be placed in each frame sent to
the DUT or SUT during the measurement interval, so that the real-time
and best-effort data frames can be distinguished from each other and
correlated with the frames received from the DUT.
The tester MUST send learning frames (after endstation connection
setup as applicable) to allow the DUT or SUT to update its address
tables properly. If multiple endstations are used, the traffic
topology MUST be of the partially meshed one-to-many/many-to-one
type.
After the baseline configuration has been tested, the tester MAY
repeat the process with a new configuration, until the desired number
of different configurations have been exercised.
When testing with multiple endstations on the wireless side and/or
Ethernet side, consecutive best-effort data frames transmitted by the
tester to the DUT or SUT MUST have different combinations of source
and destination addresses, and all possible such combinations of
addresses MUST be represented equally within each trial. This
distributes the delay impact and traffic load uniformly among the
endstations. Failure to ensure this can lead to inconsistent
results.
5.1.5.4. Analysis and reporting
The QoS differentiation capability of the DUT or SUT is reported as
the maximum aggregate offered load, in frames/second, of best-effort
data that can be presented to the DUT or SUT without causing the QoS
threshold to be violated. The QoS threshold used MUST be reported as
well.
The test results SHOULD be reported as graphs of maximum aggregate
offered load versus frame size. Separate results MUST be reported
per configuration.
5.1.6. Power-save throughput
5.1.6.1. Objective
To measure the ability of the DUT or SUT to support mobile
endstations in power management mode. Endstations in power
management mode go into a sleep state (including turning off their
radios) frequently, in order to save battery power.
Ahuja, et al. Expires July 5, 2008 [Page 33]
Internet-Draft WLAN Switch Benchmarking Methodology January 2008
WLAN infrastructure devices that support wireless endstations in
power management mode (i.e., sleeping endstations) are required to
accept and buffer frames on behalf of these endstations and signal
the endstations that frames are being buffered for them. The
sleeping endstations will wake periodically at a pre-configured
listen interval and check for buffered frames, going back to sleep
only after transferring the buffered frames (if any). Buffer
management and notification functions must therefore be efficiently
implemented in the DUT or SUT in order to sustain a large population
of endstations in power management mode.
This test measures the power management mode throughput of the DUT or
SUT, and hence its ability to efficiently support many connected but
sleeping endstations.
5.1.6.2. Test parameters
The following parameters are relevant to this test, and MUST be
configured as specified in Section 3.5.14:
Frame size, Listen interval, Number of wireless and Ethernet
endstations, Fragmentation, RTS/CTS usage, Security mode and
Number of virtual WLANs.
The listen interval SHOULD NOT exceed the frame aging time of the DUT
or SUT, and SHOULD be kept the same for all trials within a given
configuration. If desired, the following values (in units of beacon
periods) MAY be used for the listen interval:
2, 4, 6, 8, 10
The baseline DUT or SUT configuration for performing this test
consists of: frame sizes as per 3.4.2, listen interval of 4 beacon
periods, a single wireless endstation, a single Ethernet endstation,
fragmentation off, RTS/CTS disabled, security not used, and a single
virtual WLAN.
5.1.6.3. Procedure
The DUT or SUT is initially set up according to the baseline
configuration. The tester then associates the required number of
wireless endstations with the DUT or SUT and causes these endstations
to immediately enter power-save (sleep) mode. The tester MUST
transmit learning frames to and from the endstations, and MUST then
wait a sufficient length of time to ensure that the endstations have
entered power-save mode successfully.
The tester then transmits test data frames at a starting offered load
Ahuja, et al. Expires July 5, 2008 [Page 34]
Internet-Draft WLAN Switch Benchmarking Methodology January 2008
(computed according to Appendix A) to the Ethernet side of the DUT or
SUT for forwarding to each of the sleeping endstations. The initial
offered load SHOULD be set to the aggregate theoretical maximum
capacity of all of the wireless-side links. A search algorithm is
then used to determine the throughput. The measurement is repeated
for each value of frame size.
After the baseline configuration has been tested, the tester MAY
repeat the process with a new configuration, until the desired number
of different configurations have been exercised.
In tests involving multiple endstations (either on the wireless or
wired side, or both), the tester MUST ensure a uniform distribution
of frames from each source endstation. In addition, all permissible
combinations of source and destination addresses (consistent with the
traffic direction setting) MUST be represented equally within each
trial. This distributes the traffic and power-save buffer load
uniformly among the endstations. Note that this corresponds closely
to the partially-meshed one-to-many/many-to-one topology described in
RFC 2889 [RFC2889].
5.1.6.4. Analysis and reporting
The power-save throughput of the DUT or SUT is computed and reported
(per Section 26.1 of RFC 2544 [RFC2544]) as the maximum offered load,
in frames per second, resulting in zero frame loss rate [RFC1242].
The test results SHOULD be reported as graphs of throughput versus
frame size. Separate results MUST be reported per configuration.
5.2. Control plane tests
Control plane tests comprise the following:
Endstation roaming delay
Endstation roaming rate
Endstation association rate
Endstation capacity
WTP capacity
Reset recovery time
Failover recovery time
Ahuja, et al. Expires July 5, 2008 [Page 35]
Internet-Draft WLAN Switch Benchmarking Methodology January 2008
The endstation roaming delay and rate tests directly measure the
capacity of the DUT or SUT to support mobility on the part of the
endstations, and are generally considered to be important for
enterprise networks. The test results can, for example, indicate
whether VoIP handsets can roam from WTP to WTP without causing a loss
in voice quality (due to dropouts or lost voice samples). It is
generally considered that the roaming delay should be considerably
less than the time between two or three VoIP packets to avoid having
a material impact on voice traffic.
The endstation association rate and capacity tests attempt to
quantify the ability of the DUT or SUT to efficiently support large
numbers of connected endstations. Each endstation is represented by
a significant amount of state maintained within the AC (and sometimes
even the WTPs). The implementation of efficient state management and
update algorithms is necessary in order to avoid issues such as slow
rates of endstation connection or even outright disconnection of
successfully associated endstations when new endstations attempt to
connect.
The WTP capacity test is applicable to ACs only, and measures the
scalability of the DUT. Modern enterprise WLANs may require hundreds
or even thousands of WTPs to be deployed to provide adequate
coverage. This test therefore quantifies the size of the WLAN that
may be practically constructed using the DUT.
Reset recovery and failover recovery measurements are essential
indicators of the robustness, uptime and stability of a WLAN that is
deployed using the DUT or SUT. Rapid recovery from catastrophic
events such as equipment failure or power glitches is a key
requirement for an enterprise network carrying critical data. A long
reset recovery time can cause timeouts and connection drops at the
transport and application layers, and may even cause endstations to
be disconnected.
Data traffic streams in control plane tests are used principally as
indicators of successfully completed events, handshakes or trials.
The frame sizes, flow rates and flow directions of data traffic do
not particularly affect the test results. Hence the test parameter
specifications for control plane tests in this document generally fix
these at convenient values.
5.2.1. Endstation roaming delay
5.2.1.1. Objective
The 802.11 protocol enables a endstation to dynamically disassociate
itself from one WTP and reassociate with another WTP in the same
Ahuja, et al. Expires July 5, 2008 [Page 36]
Internet-Draft WLAN Switch Benchmarking Methodology January 2008
domain. This is done to facilitate the mobility (or roaming) of
endstations within an extended region that constitutes a single
logical LAN covered by multiple WTPs. The time required for
endstations to transition from one WTP to another plays a large role
in the perceived quality and reliability of the mobile system. Long
roaming delays can result in lost data and dropped connections. This
test seeks to determine the delay experienced by endstations when
roaming between WTPs belonging to the DUT or SUT.
In 802.11 networks, endstations are the primary drivers of roaming
behavior, and actually initiate the decision to roam. WTPs and ACs,
are significant contributing factors to the total roaming delay, in
terms of the time required for them to accept and complete connection
and security handshakes and resume traffic flow to and from the
endstation; however, endstations also contribute to some of the
delays. Endstation and WTP roaming time contributions are thus
measured and reported separately.
This test MUST be carried out on a DUT or SUT that involves two or
more physical interfaces on the wireless side; if the SUT includes
WTPs, then two or more WTPs MUST be present.
5.2.1.2. Test parameters
The following parameters MUST be configured prior to each trial as
specified in Section 3.5.14:
Number of wireless endstations, Roaming rate, RTS/CTS usage,
Security mode, DHCP mode and Number of virtual WLANs.
The roaming rate SHOULD be varied between 0.1 and 10 roams per
second. The following discrete values of roaming rate MAY be used:
0.1, 0.2, 1, 5, 10
The baseline DUT or SUT configuration for performing this test
consists of: a single wireless endstation, roaming rate of 0.2 roams/
second, RTS/CTS disabled, security not used, DHCP not used, and a
single virtual WLAN.
Several security enhancements such as preauthentication and PMKID
caching are implemented by vendors in order to speed up roaming. If
these enhancements are available, additional trials MAY be run after
the baseline configuration in order to quantify the effect of these
enhancements.
Ahuja, et al. Expires July 5, 2008 [Page 37]
Internet-Draft WLAN Switch Benchmarking Methodology January 2008
5.2.1.3. Procedure
The tester connects the wireless endstations with the DUT or SUT in a
uniformly distributed manner (i.e., each physical interface or WTP is
presented with the same number of endstations). It then injects a
continuous stream of traffic into the Ethernet side of the DUT or SUT
that is destined for the wireless endstations. The traffic
distribution MUST be uniform, i.e., the same offered load is set up
for all of the wireless endstations.
The frame size of the injected traffic MUST be set as per section
3.4.3. The aggregate injected traffic load, as observed at any
wireless interface, MUST NOT exceed 50% of the theoretical maximum
capacity of that interface, to avoid seriously hampering the ability
of the DUT or SUT to transfer the management frames that are
exchanged during roaming handshakes. Note that the resolution of the
roaming delay measurement is inversely proportional to the frame rate
of the injected traffic, and hence MUST be calculated and reported
with the test results.
The tester then causes the wireless endstations to roam from
interface to interface (from WTP to WTP, if WTPs are present in the
SUT), and measures the roaming delay. After the trial duration has
expired, the traffic is stopped and the minimum, maximum and average
roaming delay contribution of the DUT or SUT, the average number of
lost packets per roam, and the number of failed roams are measured
and reported. The tester SHOULD report these results on a per-
virtual-WLAN basis, and MAY report the results on a per-endstation
basis as well.
As described in Section 3.4.12, the test MUST be performed with a
baseline DUT or SUT setup of intra-subnet, intra-AC roaming.
The DUT or SUT configuration parameters are initially set up and
tested according to the baseline. After the baseline configuration
has been tested, the tester MAY repeat the process with a new set of
configuration parameters, until the desired number of different
configurations have been exercised. In particular, inter-subnet
and/or inter-AC roaming situations SHOULD be tested. If these
situations are tested, each situation SHOULD be tested with the same
combination of test parameters, to enable the results to be compared.
The trial duration MUST be set to allow every endstation to roam at
least once, and SHOULD be set to allow every endstation to perform a
complete circuit of all of the interfaces and return to the starting
point. Note that with a large number of endstations, this may
require a fairly long trial duration.
Ahuja, et al. Expires July 5, 2008 [Page 38]
Internet-Draft WLAN Switch Benchmarking Methodology January 2008
5.2.1.4. Analysis and reporting
The roaming delay contribution of the DUT or SUT is measured by
subtracting the roaming delay contribution of the endstation from the
overall roaming delay, and is expressed in seconds. The roaming
delay contribution of the endstation is calculated by adding up all
the delays (excluding packet transmission delays) incurred by the
endstation due to internal processing, during the process of moving
from one interface/WTP to another interface/WTP.
The average number of lost packets per roam is calculated by
subtracting the total number of data traffic packets received by the
endstations from the total number of data traffic packets sent to the
endstations, and dividing by the total number of roams performed.
The number of failed roams is measured as the number of times that
the DUT or SUT failed to begin transferring data packets to a
endstation after the endstation completed the roaming process.
The results SHOULD be reported in tabular format. Separate results
MUST be reported per DUT or SUT configuration and test scenario
(i.e., intra/inter-subnet and intra/inter-AC). The results MAY be
also reported as profiles (a graph over time) to enable better
understanding of the roaming behavior of the DUT or SUT.
5.2.2. Endstation roaming rate
5.2.2.1. Objective
In addition to the time required for an endstation to transition
between WTPs (endstation roaming delay), the rate at which
endstations can transition from one WTP to another also plays a part
in the perceived responsiveness and reliability of a deployed WLAN.
If the DUT or SUT is incapable of sustaining high roaming rates, then
momentary periods of high mobility (e.g., a number of users
congregating in a conference room) can cause issues such as dropped
connections or handset calls. This test thus seeks to quantify the
maximum rate at which the DUT or SUT can support roaming functions.
This test MUST be carried out on a DUT or SUT that involves two or
more physical interfaces on the wireless side; if the SUT includes
WTPs, then two or more WTPs MUST be present.
5.2.2.2. Test parameters
The following parameters MUST be configured prior to each trial as
specified in Section 3.5.14:
Ahuja, et al. Expires July 5, 2008 [Page 39]
Internet-Draft WLAN Switch Benchmarking Methodology January 2008
Number of wireless endstations, RTS/CTS usage, Security mode,
DHCP mode and Number of virtual WLANs.
The baseline DUT or SUT configuration for performing this test
consists of: a single wireless endstation, RTS/CTS disabled, security
not used, DHCP not used, and a single virtual WLAN.
Several security enhancements such as preauthentication and PMKID
caching are implemented by vendors in order to speed up roaming, and
these enhancements frequently improve the roaming rate measurements
as well. If such enhancements are available, additional trials MAY
be run after the baseline configuration in order to quantify the
effect of these enhancements.
5.2.2.3. Procedure
The tester connects the wireless endstations with the DUT or SUT in a
uniformly distributed manner (i.e., each physical interface or WTP is
presented with the same number of endstations). It then injects a
continuous stream of traffic into the Ethernet side of the DUT or SUT
that is destined for the wireless endstations. The traffic
distribution MUST be uniform, i.e., the same offered load is set up
for all of the wireless endstations.
The frame size of the injected traffic MUST be set as per section
3.4.3. The aggregate traffic load at any wireless interface MUST NOT
exceed 50% of the theoretical maximum capacity of that interface, to
avoid seriously hampering the ability of the DUT or SUT to transfer
the management and control frames that are exchanged during roaming
handshakes.
The tester then causes the wireless endstations to roam from
interface to interface (from WTP to WTP, if WTPs are present in the
SUT) at a constant rate for the duration of the trial. Roam events
MUST be triggered in a serial fashion, i.e., only one roam is
initiated at a time, but the roam processes for different endstations
MAY overlap in order to establish the desired roaming rate. After
the trial duration has expired, the traffic is stopped and the number
of failed roams are measured. A search algorithm is used to
determine the maximum rate at which roams can be initiated without
causing any failed roams. A failed roam is counted either when the
reconnection handshake terminates unsuccessfully, or when the DUT or
SUT fails to begin transferring data packets to an endstation after
the endstation completes the roaming process.
As described in Section 3.4.12, the test MUST be performed with a
baseline DUT or SUT setup of intra-subnet, intra-AC roaming.
Ahuja, et al. Expires July 5, 2008 [Page 40]
Internet-Draft WLAN Switch Benchmarking Methodology January 2008
The DUT or SUT configuration parameters are initially set up and
tested according to the baseline. After the baseline configuration
has been tested, the tester MAY repeat the process with a new set of
configuration parameters, until the desired number of different
configurations have been exercised. In particular, inter-subnet
and/or inter-AC roaming situations SHOULD be tested. If these
situations are tested, each situation SHOULD be tested with the same
combination of test parameters, to enable the results to be compared.
The trial duration MUST be set to allow every endstation to roam at
least once, and SHOULD be set to allow every endstation to perform a
complete circuit of all of the interfaces and return to the starting
point. Note that with a large number of endstations, this may
require a fairly long trial duration.
5.2.2.4. Analysis and reporting
The roaming rate supported by the DUT or SUT is measured and
expressed as the number of roaming events that can be presented to
the DUT or SUT per second without failures.
The results SHOULD be reported in tabular format. Separate results
MUST be reported per DUT or SUT configuration and test scenario
(i.e., intra/inter-subnet and intra/inter-AC).
5.2.3. Endstation association rate
5.2.3.1. Objective
The 802.11 protocol requires that an infrastructure endstation
wishing to communicate must first authenticate and associate itself
with a WTP, performing all the necessary security, ARP and DHCP
functions in the process. The rate at which these functions can be
carried out impacts the time taken for a wireless LAN to recover from
faults and transient conditions, such as a WTP being reset or a group
of endstations being turned on concurrently.
The objective of this test is hence to determine the rate at which
the DUT or SUT can associate endstations.
5.2.3.2. Test parameters
The following parameters MUST be configured prior to each trial as
specified in Section 3.5.14:
Number of wireless endstations, RTS/CTS usage, Security mode,
DHCP mode and Number of virtual WLANs.
Ahuja, et al. Expires July 5, 2008 [Page 41]
Internet-Draft WLAN Switch Benchmarking Methodology January 2008
In addition, the following test parameter MUST be configured to be
the same for all trials:
Association Timeout - The tester MUST wait a predetermined amount
of time for the DUT or SUT to complete all of the handshakes
required for connection setup. If the DUT or SUT fails to
complete the connection process within this time, the association
attempt MUST be considered to have failed, and the connection
attempt MUST be restarted.
Association Retry Limit - The tester MUST limit the number of
times that the connection attempts for each endstation are
repeated (on failure) before giving up and reporting the
endstation as having failed to associate.
The association timeout and association retry limit used by the
tester MUST be reported with the test results.
The baseline DUT or SUT configuration for performing this test
consists of: a single wireless endstation, association timeout of 1
second, association retry limit of 1, RTS/CTS disabled, security not
used, DHCP not used, and a single virtual WLAN.
5.2.3.3. Procedure
The DUT or SUT is first set up according to the baseline
configuration. The tester then presents the required number of
virtual or physical test endstations to the DUT or SUT for
connection, and measures the rate at which the DUT or SUT
successfully completes associations. The tester MUST pipeline the
associations (i.e., begin the association of the next endstation
before the previous endstation has been fully associated) in order to
present the DUT or SUT with as high a load as possible. The tester
MUST record the actual average rate at which new endstations were
presented over the course of the trial period.
If all of the endstations presented to the DUT or SUT fail to
associate, then the tester MUST deauthenticate the associated
endstations, reduce the association rate, and repeat the trial. If
all of the endstations succeed in associating, the tester MUST
increase the association rate and repeat the trial. The process
continues until the maximum association rate is found. A search
algorithm SHOULD be used to speed up the process.
It is recommended that the authentication and association database
capacity test in Section 5.2.3 be performed first to determine the
maximum number of endstations that can successfully associate with
the DUT. The number of virtual endstations presented to the DUT
Ahuja, et al. Expires July 5, 2008 [Page 42]
Internet-Draft WLAN Switch Benchmarking Methodology January 2008
SHOULD be kept below this number.
After the test endstations successfully authenticate and associate
with the DUT or SUT, the tester MUST verify that these endstations
have indeed been associated by transmitting test data frames to the
endstations, and ensure that these data frames are correctly
forwarded by the DUT or SUT. The rate at which verification data
frames are transmitted to the DUT or SUT MUST be well below the
theoretical maximum capacity of the links. The tester MUST ensure
that at least one data frame directed to each endstation is
forwarded.
If the DUT or SUT deauthenticates or disassociates one or more
endstations during the data transfer phase, these MUST be counted as
association failures. If none of the test data frames transmitted to
a endstation are forwarded successfully, this MUST be treated as a
verification failure. If failures do occur, the tester SHOULD
attempt to find a lower rate of association for which no verification
failures are found for all of the test endstations.
After the baseline configuration has been tested, the tester MAY
repeat the process with a new configuration, until the desired number
of different configurations have been exercised. If the DUT or SUT
supports a built-in DHCP server, at least one of the configurations
tested SHOULD include endstations receiving their addresses via DHCP.
(Provisioning IP addresses via DHCP is a very common situation in
WLANs.)
After each trial has been completed, the tester MUST remove the test
endstation associations from the DUT or SUT database by performing
the 802.11 deauthentication procedure for each endstation.
5.2.3.4. Analysis and reporting
The endstation association rate of the DUT or SUT is computed and
reported as the maximum number of associations that can be
successfully performed per second. Association and verification
failures MUST be reported along with the test results.
If the test is repeated for different numbers of endstations, the
results MAY be presented as graphs of endstation association rate
versus number of test endstations.
5.2.4. Endstation capacity
Ahuja, et al. Expires July 5, 2008 [Page 43]
Internet-Draft WLAN Switch Benchmarking Methodology January 2008
5.2.4.1. Objective
The 802.11 protocol requires that an infrastructure endstation
wishing to communicate must authenticate and associate with (i.e.,
connect to) a WTP. In order to track and maintain the connection
state of each endstation, the WTP and/or AC must maintain a
substantial database of endstation states and attributes. Further,
this database must be consulted during all endstation state changes
(e.g., during roaming); thus the resources consumed by this database
places no small overhead on the WLAN infrastructure, and forms an
upper limit on the number of concurrently active endstations that can
be supported by the WLAN.
The objective of this test is hence to determine the number of
endstations that can be supported at one time by the DUT or SUT.
5.2.4.2. Test parameters
The following parameters MUST be configured prior to each trial as
specified in Section 3.5.14:
Endstation association rate, RTS/CTS usage, Security mode, DHCP
mode and Number of virtual WLANs.
In addition, the following test parameter MUST be configured to be
the same for all trials:
Association Timeout - The tester MUST wait a predetermined amount
of time for the DUT or SUT to complete all of the handshakes
required for connection setup. If the DUT or SUT fails to
complete the connection process within this time, the association
attempt MUST be considered to have failed, and the connection
attempt MUST be restarted.
Association Retry Limit - The tester MUST limit the number of
times that the connection attempts for each endstation are
repeated (on failure) before giving up and reporting the
endstation as having failed to associate.
The association timeout and association retry limit used by the
tester MUST be reported with the test results.
The baseline DUT or SUT configuration for performing this test
consists of: association rate of 1 per second, association retry
limit of 1, RTS/CTS disabled, security not used, DHCP not used, and a
single virtual WLAN.
Ahuja, et al. Expires July 5, 2008 [Page 44]
Internet-Draft WLAN Switch Benchmarking Methodology January 2008
5.2.4.3. Procedure
The DUT or SUT is first set up according to the baseline
configuration. The tester then presents virtual or physical test
endstations to the DUT at the required rate for connection, until the
DUT or SUT fails to associate at least 1 endstation even after the
required number of retries have been performed.
After the test endstations successfully authenticate and associate
with the DUT or SUT, the tester MUST verify that these endstations
have indeed been associated by transmitting test data frames to the
endstations, and ensure that these data frames are correctly
forwarded by the DUT or SUT. The rate at which verification data
frames are transmitted to the DUT or SUT MUST be well below the
theoretical maximum capacity of the links. The tester MUST ensure
that at least one data frame directed to each endstation is
forwarded. If multiple virtual WLANs are used, the endstations MUST
be distributed uniformly across them.
If the DUT or SUT deauthenticates or disassociates one or more
endstations during the data transfer phase, these MUST be counted as
association failures. If none of the test data frames transmitted to
a endstation are forwarded successfully, this MUST be treated as a
verification failure. If failures do occur, the tester MUST
decrement the reported endstation capacity by the number of
endstations for which verification failures occurred.
After the baseline configuration has been tested, the tester MAY
repeat the process with a new configuration, until the desired number
of different configurations have been exercised.
After each trial has been completed, the tester MUST remove the test
endstation associations from the DUT or SUT database by performing
the 802.11 deauthentication procedure for each endstation.
5.2.4.4. Analysis and reporting
The endstation capacity of the DUT or SUT is computed and reported as
the maximum number of endstations that can be successfully associated
with the DUT or SUT at the specified association rate. Association
and verification failures SHOULD be reported along with the test
results.
If the test is repeated for different association rates, the results
MAY be presented as graphs of endstation association rate versus
endstation capacity.
Ahuja, et al. Expires July 5, 2008 [Page 45]
Internet-Draft WLAN Switch Benchmarking Methodology January 2008
5.2.5. WTP capacity
5.2.5.1. Objective
To determine the number of WTPs that an AC can successfully support
at one time. This test is only applicable to ACs.
The CAPWAP protocol enables an AC to discover, initialize, configure,
manage and transfer data to/from a number of WTPs. The total number
of WTPs that a single AC can support can range into the hundreds,
implying that the management load on the AC can be quite substantial,
and the ability of the AC to support and maintain the WTPs will play
a substantial part in the performance and uptime of the LAN.
This test therefore measures the ability of the DUT or SUT to support
and maintain a large number of WTPs, each of which is concurrently
supporting a large number of endstations, each of which is sinking
and/or sourcing data traffic.
This test may be carried out with actual (physical) WTPs, or by using
test equipment capable of emulating WTPs and the wireless endstations
behind them. If emulated WTPs are used, they must implement the
management protocol (e.g., CAPWAP) employed between the WTPs and the
ACs. If actual WTPs are used, emulation is not necessary, but the
WTP capacity of the AC must then be determined by manually connecting
and disconnecting physical WTPs.
5.2.5.2. Test parameters
The following parameters MUST be configured prior to each trial as
specified in Section 3.5.14:
Number of wireless endstations per WTP, Offered load per
endstation, Traffic direction, Security mode, DHCP mode and
Number of virtual WLANs.
In addition, the following test parameter MUST be configured to be
the same for all trials:
Initialization Timeout - The tester MUST wait a predetermined
amount of time for the DUT or SUT to discover the WTPs and
complete all of the handshakes required for initialization and
setup, including firmware download. The initialization timeout
MUST not exceed 900 seconds.
The initialization timeout used by the tester MUST be reported with
the test results.
Ahuja, et al. Expires July 5, 2008 [Page 46]
Internet-Draft WLAN Switch Benchmarking Methodology January 2008
The baseline DUT or SUT configuration for performing this test
consists of: a single wireless endstation per WTP, offered load per
endstation of 25% of theoretical maximum medium capacity, downstream
(Ethernet to wireless) traffic flow, initialization timeout of 600
seconds, security not used, DHCP not used, and a single virtual WLAN.
5.2.5.3. Procedure
The DUT or SUT is first set up according to the baseline
configuration. The tester then presents an initial number of virtual
or physical test WTPs to the DUT. The test WTPs SHOULD be presented
as nearly simultaneously as possible. If the DUT or SUT fails to
initialize the number of test WTPs successfully within the
initialization timeout, the tester MUST consider the trial to have
failed, and another trial MUST be performed with a smaller number of
WTPs. If the DUT or SUT succeeds in initializing all of the test
WTPs, the test WTPs should be removed, and the next trial performed
with a larger number, until the no more test WTPs can be connected.
Once all of the test WTPs have been successfully connected and
initialized, the tester MUST associate the configured number of
endstations with each WTP, and then MUST cause traffic to flow to
and/or from each endstation between the wireless and Ethernet sides
of the DUT. The tester MUST first send learning frames (after
endstation connection setup as per the security mode) to allow the
DUT or SUT to update its address tables properly.
In tests involving multiple endstations per WTP, the tester MUST
ensure a uniform distribution of frames to/from each endstation. In
addition, all permissible combinations of source and destination
addresses (consistent with the traffic direction setting) MUST be
represented equally within each trial, to distribute the load of
transmission and reception uniformly.
If the DUT or SUT deauthenticates one or more test endstations, the
tester MUST attempt to reauthenticate these endstations. If the
tester fails to reauthenticate the endstations, then the trial MUST
be considered to have failed, and MUST be repeated with a smaller
number of WTPs. If the DUT or SUT disconnects from one or more WTPs,
or reboots, the trial MUST be considered to have failed, and MUST be
repeated with a smaller number of WTPs. A successful trial is one in
which the specified number of test WTPs remain connected to the DUT
or SUT and the specified number of test endstations are able to
remain associated and successfully pass traffic for the trial
duration.
After the baseline configuration has been tested, the tester MAY
repeat the process with a new configuration, until the desired number
Ahuja, et al. Expires July 5, 2008 [Page 47]
Internet-Draft WLAN Switch Benchmarking Methodology January 2008
of different configurations have been exercised.
The tester MUST remove the test endstation context and the WTP
context from the DUT or SUT database after the completion of each
trial, by performing the 802.11 deauthentication procedure for each
associated endstation and disconnecting or powering down the WTP
after all the associated endstations have been deauthenticated.
5.2.5.4. Analysis and reporting
The WTP capacity of the DUT or SUT is computed and reported as the
maximum number of WTPs that can be simultaneously connected to the
DUT, with associated endstations that can successfully exchange data,
over the entire trial duration.
5.2.6. Reset recovery time
5.2.6.1. Objective
As pointed out in RFC 2544 [RFC2544] and RFC 1242 [RFC1242], the
rapidity with which a WLAN infrastructure device transitions from a
reset state to a fully operational state affects the perceived
availability and stability of a wireless network. For example, an
excessive time required to recover from a reset can force endstations
to begin scanning for other WTPs, cause higher-layer connections to
be dropped, and so on.
This test therefore measures the speed with which a DUT or SUT
recovers from a device or software reset and resumes forwarding
endstation traffic. It is performed on either SUTs comprising WTPs
and ACs, or on DUTs comprising ACs only.
5.2.6.2. Test parameters
The following parameters MUST be configured prior to each trial as
specified in Section 3.5.14:
Number of WTPs, Number of wireless endstations per WTP, Offered
load per endstation, Traffic direction, Security mode, DHCP mode
and Number of virtual WLANs.
In addition, the following test parameter MUST be configured to be
the same for all trials:
Reset duration - The reset duration MUST NOT be less than 5
seconds, and SHOULD be greater than 10 seconds. The tester MUST
subtract the reset duration from the measured traffic
interruption interval. The trial duration MUST be sufficient to
Ahuja, et al. Expires July 5, 2008 [Page 48]
Internet-Draft WLAN Switch Benchmarking Methodology January 2008
cover both the reset duration and the anticipated recovery time
of the DUT or SUT.
The reset duration used by the tester MUST be reported with the test
results.
The baseline DUT or SUT configuration for performing this test
consists of: a single wireless endstation per WTP, offered load per
endstation of 25% of theoretical maximum medium capacity, downstream
(Ethernet to wireless) traffic flow, reset duration of 10 seconds,
security not used, DHCP not used, and a single virtual WLAN.
5.2.6.3. Procedure
The DUT or SUT is set up according to the baseline configuration.
The tester should then associate the virtual or physical
endstation(s) with the WTPs, and transmit test data between the
wireless and Ethernet sides of the DUT or SUT according to the
configured traffic direction for the trial duration.
During the middle of the trial period, the DUT or SUT is reset, and
then allowed to recover normally. The reset process of the DUT or
SUT MUST NOT be artificially short-circuited in any way (e.g., by
providing predefined configurations that are not part of normal
operational practice). The tester MUST monitor the data traffic
being received by the wireless endstations before, during and after
the reset period, and MUST record the timestamps of the last valid
data frame received just after the reset is applied and the first
valid data frame received just after the reset duration completes and
the reset is removed. The reset recovery time is then calculated and
reported as below.
A power-interruption reset test MUST be performed. If the DUT or SUT
is capable of a software reset and/or a hardware reset, then the test
SHOULD be repeated with the software and hardware resets. The
results MUST be reported separately.
Devices that are not considered to be part of the DUT or SUT MUST NOT
be reset. For example, if the setup comprises the tester, one or
more WTPs, and an AC, and only the AC is considered to be part of the
DUT or SUT, then the reset (hardware or software) MUST be applied
only to the AC.
The tester MUST distinguish between valid test traffic frames and
other frames (e.g., corrupted frames, management frames, random data
frames) and MUST use only valid test traffic frames in the
measurement process.
Ahuja, et al. Expires July 5, 2008 [Page 49]
Internet-Draft WLAN Switch Benchmarking Methodology January 2008
If the test endstations are disconnected or disassociated from the
DUT or SUT during the reset duration or the recovery period, the
tester MUST attempt to reconnect the test endstations as soon as
beacons are received from the WTPs during the recovery period.
Disconnection of one or more test endstations during the reset
duration or recovery period MUST be reported with the test results.
5.2.6.4. Analysis and reporting
The reset recovery time MUST be measured and reported as the time, in
seconds, between the last received test data frame just prior to the
application of the reset and the first received test data frame just
following the removal of the reset.
5.2.7. Failover recovery time
5.2.7.1. Objective
To determine the speed with which a DUT or SUT containing redundant
modules or devices can restore service after a failure of a module.
The rapidity with which the DUT or SUT is able to restore service
following failure of one of its components directly affects the
availability and uptime of a wireless network. In critical
situations, it is common to provide backup ACs that are capable of
taking over when a primary AC fails. Rapid restoration of service is
essential to avoid issues such as dropped VoIP calls and lost TCP
connections.
This test quantifies the time taken for a WLAN infrastructure DUT or
SUT to restore service following a failure of a primary AC. Note
that this test is only applicable to DUTs/SUTs that implement
redundant ACs or AC modules. Further, this test is only feasible in
situations where the primary AC or AC module can be disabled or
removed without powering down the DUT or SUT, and while traffic is
flowing.
5.2.7.2. Test parameters
The following parameters MUST be configured prior to each trial as
specified in Section 3.5.14:
Number of WTPs, Number of wireless endstations per WTP, Offered
load per endstation, Traffic direction, Security mode, DHCP mode
and Number of virtual WLANs.
The baseline DUT or SUT configuration for performing this test
consists of: one WTP, a single wireless endstation per WTP, offered
Ahuja, et al. Expires July 5, 2008 [Page 50]
Internet-Draft WLAN Switch Benchmarking Methodology January 2008
load per endstation of 25% of theoretical maximum medium capacity,
downstream (Ethernet to wireless) traffic flow, security not used,
DHCP not used, and a single virtual WLAN.
5.2.7.3. Procedure
The DUT or SUT is set up according to the baseline configuration.
The tester should then associate the virtual or physical
endstation(s) with the WTPs, and transmit test data between the
wireless and Ethernet sides of the DUT or SUT according to the
configured traffic direction for the trial duration.
During the middle of the trial period, the primary AC is disabled in
one of the following ways:
Removal from the chassis (if implemented as a hot-swappable
module)
Powering it down (if implemented as a separately powered device)
Disconnection from the rest of the DUT or SUT (if disconnection
is possible by removing a single link)
Disabling it under software control
The software disable method SHOULD be used only as a last resort, if
no other means of disabling the primary AC is found. The method used
to disable the primary AC MUST be described along with the test
results.
The tester MUST monitor the data traffic being received by the
wireless endstations before and after the primary AC is disabled. If
data traffic is interrupted during the trial for any reason, the
tester MUST record and report the duration of the interruption as the
failover recovery time. Note that it is possible for the failover
recovery to be fully transparent to the data traffic, in which case
the failover recovery time is effectively zero.
The tester MUST distinguish between valid test traffic frames and
other frames (e.g., corrupted frames, management frames, random data
frames) and MUST use only valid test traffic frames in the
measurement process.
If the test endstations are disconnected or disassociated from the
DUT or SUT during the failover recovery period, the tester MUST
attempt to reconnect the test endstations immediately, and continue
to attempt to reconnect the test endstations until successful (or
until the trial ends, whichever is sooner). Disconnection of one or
Ahuja, et al. Expires July 5, 2008 [Page 51]
Internet-Draft WLAN Switch Benchmarking Methodology January 2008
more test endstations during the failover recovery period MUST be
reported with the test results. Failure to restore traffic to one or
more test endstations after the primary AC has been disconnected MUST
be reported with the test results.
5.2.7.4. Analysis and reporting
The failover recovery time MUST be measured and reported as the time,
in seconds, during which data traffic is interrupted after the
primary AC has been disabled.
6. Security Considerations
Documents of this type do not directly affect the security of the
Internet or of corporate networks as long as benchmarking is not
performed on devices or systems connected to operating networks.
Note that performance tests SHOULD be done on with adequate isolation
between the DUT or SUT and the remainder of the network, or with
security systems enabled, to avoid the possibility of compromising
the performance of operating networks in some manner.
7. IANA Considerations
There are no IANA actions requested in this memo. (Note to RFC
Editor: This section may be removed upon publication as a RFC.)
8. References
8.1. Normative References
[802.11] IEEE, "ANSI/IEEE Std 802.11 "Part 11: Wireless LAN Medium
Access Control (MAC) and Physical Layer (PHY)
Specifications," ISO/IEC 8802-11:1999(E), ISBN 0-7381-
1658-0", 1999.
[802.3] IEEE, "ANSI/IEEE Std 802.3, "Part 3: Carrier sense
multiple access with collision detection (CSMA/CD) access
method and physical layer specifications," ISBN 0-7381-
4740-0", 2005.
[RFC1042] Postel, J. and J. Reynolds, "Standard for the transmission
of IP datagrams over IEEE 802 networks", STD 43, RFC 1042,
February 1988.
Ahuja, et al. Expires July 5, 2008 [Page 52]
Internet-Draft WLAN Switch Benchmarking Methodology January 2008
[RFC1112] Deering, S., "Host extensions for IP multicasting", STD 5,
RFC 1112, August 1989.
[RFC1242] Bradner, S., "Benchmarking terminology for network
interconnection devices", RFC 1242, July 1991.
[RFC2119] Bradner, S., "Key words for use in RFCs to Indicate
Requirement Levels", BCP 14, RFC 2119, March 1997.
[RFC2285] Mandeville, R., "Benchmarking Terminology for LAN
Switching Devices", RFC 2285, February 1998.
[RFC2544] Bradner, S. and J. McQuaid, "Benchmarking Methodology for
Network Interconnect Devices", RFC 2544, March 1999.
[RFC2889] Mandeville, R. and J. Perser, "Benchmarking Methodology
for LAN Switching Devices", RFC 2889, August 2000.
[RFC3550] Schulzrinne, H., Casner, S., Frederick, R., and V.
Jacobson, "RTP: A Transport Protocol for Real-Time
Applications", STD 64, RFC 3550, July 2003.
[RFC3918] Stopp, D. and B. Hickman, "Methodology for IP Multicast
Benchmarking", RFC 3918, October 2004.
[RFC4814] Newman, D. and T. Player, "Hash and Stuffing: Overlooked
Factors in Network Device Benchmarking", RFC 4814,
March 2007.
8.2. Informative References
[802.11.2]
IEEE, "IEEE P802.11.2, "Draft Recommended Practice for the
Evaluation of 802.11 Wireless Performance"", 2007.
[G.107] ITU, "ITU-T Recommendation G.107, "The E-model, a
computational model for use in transmission planning"",
2003.
Appendix A. Intended load computations
Calculating intended load for 802.11 media access is complicated by
the number of different parameters that need to be accounted for as
well as the random effect of backoff and management overhead. This
appendix provides formulas for the theoretical maximum capacity of
the media, actual intended load, and inter-burst gap.
Ahuja, et al. Expires July 5, 2008 [Page 53]
Internet-Draft WLAN Switch Benchmarking Methodology January 2008
Note that the instantaneous capacity of the 802.11 medium changes
from transmission to transmission due to the effects of random
backoff after each transmission. The formulas presented here are
therefore expected to be applied over a large volume of traffic,
rather than individual frames or bursts of frames. In addition, the
parameters used in the formulas change for different 802.11 physical
layers and also different data rates used within a particular
physical layer.
A.1. Calculating theoretical maximum media capacity
The theoretical maximum media capacity is calculated assuming
constant-size data frames, transmitted with the minimum frame spacing
according to the 802.11 protocol, with no collisions or retries
occurring.
The following input parameters are defined:
LENGTH - MAC Data frame size in bytes, including FCS. For
fragmented transfers, this is the size of each fragment.
SPEED - PHY data rate for the MAC portion of a DATA frame, in
bits/second.
PLCPTIME - Time required to transmit the PLCP header for the
given 802.11 PHY type and data rate, in seconds.
SLOTTIME - The slot time for the given 802.11 PHY type and data
rate, in seconds.
DIFS - The Distributed Interframe Space (see subclause 9.2.10 of
IEEE 802.11 [802.11]), in seconds.
SIFS - The Short Interframe Space (see subclause 9.2.10 of IEEE
802.11 [802.11]), in seconds.
CWmin - The minimum contention window duration (see subclause
9.2.4 of IEEE 802.11 [802.11]), in slot times.
The following intermediate values are calculated first:
TXTIME - Time required to transmit a single Data frame or
fragment. For transfers that do not involve an RTS/CTS exchange,
this is the time taken to transmit the Data frame plus an
immediately following ACK frame (see 9.2.8 of IEEE 802.11
[802.11]). For transfers involving an RTS/CTS exchange, this is
the time taken to transmit an RTS, CTS, Data and ACK frame.
Ahuja, et al. Expires July 5, 2008 [Page 54]
Internet-Draft WLAN Switch Benchmarking Methodology January 2008
For RTS/CTS based transfers:
TXTIME = (PLCPTIME * 4) + (SIFS * 3) + (((LENGTH + 48) * 8) /
SPEED)
For transfers not involving RTS/CTS:
TXTIME = (PLCPTIME * 2) + SIFS + (((LENGTH + 14) * 8) / SPEED)
AMFI - Average Minimum Frame Interval. This is the minimum legal
interval between the start of a Data frame and the start of the
immediately following Data frame, averaged over a large number of
Data frames.
AMFI = TXTIME + DIFS + ((CWmin * SLOTTIME) / 2)
The theoretical maximum capacity of the medium (abbreviated as CAP),
in bits/second, is then given by:
CAP = (LENGTH * 8) / AMFI
The above formula does not take into account overhead due to
management frames such as beacons and probe requests/responses. The
tester SHOULD separately account for management frame overhead during
a trial and subtract this overhead from the calculated theoretical
capacity in order compensate for the capacity loss due to these
frames.
A.2. Calculating constant intended load
The calculations in this section deal with a constant (steady) load
generated by the tester (i.e., a constant frame pattern). Burst
loads are covered in the next section.
If the DUT or SUT is not to be overloaded, the intended
unidirectional traffic load can range from 0 to 100% of the
theoretical maximum media capacity previously calculated (0 to 50% in
the case of bidirectional traffic streams). See Section 3.5.1 of RFC
2285 [RFC2285] for a full definition of Iload. For the purposes of
this document, the intended load is expressed as a percentage of the
theoretical maximum media capacity, and calculated as Iload using the
following formula:
Iload = (LOAD / CAP) * 100
where LOAD is the load in bits/second, and CAP is calculated as in
Section A.1.
Ahuja, et al. Expires July 5, 2008 [Page 55]
Internet-Draft WLAN Switch Benchmarking Methodology January 2008
In order to actually generate traffic at Iload values less than 100%,
the tester must insert extra spacing between frames to reduce the
traffic load. This extra spacing is referred to here as EFG (Excess
Frame Gap), and is calculated as follows:
EFG = AMFI * ((100 / Iload) - 1)
The actual frame interval therefore becomes (AMFI + EFG). The
traffic pattern generated by the tester hence consists of a Data
frame, the corresponding ACK frame (from the DUT), a gap equal to the
DIFS plus the average minimum backoff time, and a further gap equal
to EFG.
Generating Iload values greater than 100% requires that the tester
violate the backoff rules of the 802.11 protocol. The tests in this
document do not require Iload values greater than 100%.
A.3. Calculating burst intended load
This section deals with the computation of intended load when the
traffic pattern is bursty. A bursty pattern comprises a series of
back-to-back Data/ACK exchanges separated by a DIFS, followed by a
gap, followed by another series of back-to-back exchanges, and so on.
The gap between bursts (referred to as the IBG) is selected based on
the intended load. In addition, the IBG is calculated such that the
Iload for bursty and constant traffic are directly comparable. (See
Section 3.4.3 of RFC 2285 [RFC2285] for a discussion of IBG.)
The following input parameters are defined, in addition to those
defined above:
BURST - Length of burst in frames.
For a given Iload, the IBG is calculated as:
IBG = DIFS + (AMFI * BURST * ((100 / Iload) - 1))
Note that the IBG is measured from the last bit of the ACK frame of
the last data frame in a burst to the first bit of the preamble of
the first data frame in the next burst.
Ahuja, et al. Expires July 5, 2008 [Page 56]
Internet-Draft WLAN Switch Benchmarking Methodology January 2008
Authors' Addresses
Tarunesh Ahuja
Cisco Systems, Inc.
170 West Tasman Dr.
San Jose, California 95134
USA
Phone: +1 408 853 9252
Email: tahuja@cisco.com
Tom Alexander
VeriWave, Inc.
8770 SW Nimbus Ave,
Beaverton, Oregon 97008
USA
Phone: +1 971 327 7490
Email: tom@veriwave.com
Scott Bradner
Harvard University
29 Oxford St.
Cambridge, Massachusetts 02138
USA
Phone: +1 617 495 3864
Email: sob@harvard.edu
Sanjay Hooda
Cisco Systems, Inc.
170 West Tasman Dr.
San Jose, California 95134
USA
Phone: +1 408 527 6403
Email: shooda@cisco.com
Ahuja, et al. Expires July 5, 2008 [Page 57]
Internet-Draft WLAN Switch Benchmarking Methodology January 2008
Jerry Perser
VeriWave, Inc.
5743 Corsa Avenue, Suite 224
Westlake Village, California 91362
USA
Phone: +1 818 889 2071
Email: jperser@veriwave.com
Muninder Sambi
Cisco Systems, Inc.
170 West Tasman Dr.
San Jose, California 95134
USA
Phone: +1 408 525 7298
Email: msambi@cisco.com
Ahuja, et al. Expires July 5, 2008 [Page 58]
Internet-Draft WLAN Switch Benchmarking Methodology January 2008
Full Copyright Statement
Copyright (C) The IETF Trust (2008).
This document is subject to the rights, licenses and restrictions
contained in BCP 78, and except as set forth therein, the authors
retain all their rights.
This document and the information contained herein are provided on an
"AS IS" basis and THE CONTRIBUTOR, THE ORGANIZATION HE/SHE REPRESENTS
OR IS SPONSORED BY (IF ANY), THE INTERNET SOCIETY, THE IETF TRUST AND
THE INTERNET ENGINEERING TASK FORCE DISCLAIM ALL WARRANTIES, EXPRESS
OR IMPLIED, INCLUDING BUT NOT LIMITED TO ANY WARRANTY THAT THE USE OF
THE INFORMATION HEREIN WILL NOT INFRINGE ANY RIGHTS OR ANY IMPLIED
WARRANTIES OF MERCHANTABILITY OR FITNESS FOR A PARTICULAR PURPOSE.
Intellectual Property
The IETF takes no position regarding the validity or scope of any
Intellectual Property Rights or other rights that might be claimed to
pertain to the implementation or use of the technology described in
this document or the extent to which any license under such rights
might or might not be available; nor does it represent that it has
made any independent effort to identify any such rights. Information
on the procedures with respect to rights in RFC documents can be
found in BCP 78 and BCP 79.
Copies of IPR disclosures made to the IETF Secretariat and any
assurances of licenses to be made available, or the result of an
attempt made to obtain a general license or permission for the use of
such proprietary rights by implementers or users of this
specification can be obtained from the IETF on-line IPR repository at
http://www.ietf.org/ipr.
The IETF invites any interested party to bring to its attention any
copyrights, patents or patent applications, or other proprietary
rights that may cover technology that may be required to implement
this standard. Please address the information to the IETF at
ietf-ipr@ietf.org.
Acknowledgment
Funding for the RFC Editor function is provided by the IETF
Administrative Support Activity (IASA).
Ahuja, et al. Expires July 5, 2008 [Page 59]