Internet DRAFT - draft-cheng-sdn-switchtest

draft-cheng-sdn-switchtest







Network Working Group                                           M. Cheng
Internet-Draft                                                    Y. Bao
Intended status: Standards Track                 BII Group Holdings Ltd.
Expires: June 24, 2017                                 December 21, 2016


                SDN Switch Benchmark Test Cases Summary
                     draft-cheng-sdn-switchtest-00

Abstract

   This document summarized the test cases for SDN switches, and
   introduced the test methodologies for each case of them.

Status of This Memo

   This Internet-Draft is submitted in full conformance with the
   provisions of BCP 78 and BCP 79.

   Internet-Drafts are working documents of the Internet Engineering
   Task Force (IETF).  Note that other groups may also distribute
   working documents as Internet-Drafts.  The list of current Internet-
   Drafts is at http://datatracker.ietf.org/drafts/current/.

   Internet-Drafts are draft documents valid for a maximum of six months
   and may be updated, replaced, or obsoleted by other documents at any
   time.  It is inappropriate to use Internet-Drafts as reference
   material or to cite them other than as "work in progress."

   This Internet-Draft will expire on June 24, 2017.

Copyright Notice

   Copyright (c) 2016 IETF Trust and the persons identified as the
   document authors.  All rights reserved.

   This document is subject to BCP 78 and the IETF Trust's Legal
   Provisions Relating to IETF Documents
   (http://trustee.ietf.org/license-info) in effect on the date of
   publication of this document.  Please review these documents
   carefully, as they describe your rights and restrictions with respect
   to this document.  Code Components extracted from this document must
   include Simplified BSD License text as described in Section 4.e of
   the Trust Legal Provisions and are provided without warranty as
   described in the Simplified BSD License.






Cheng & Bao               Expires June 24, 2017                 [Page 1]

Internet-Draft       SDN Switch Benchmark Test Cases       December 2016


Table of Contents

   1.  Introduction  . . . . . . . . . . . . . . . . . . . . . . . .   2
   2.  Test Cases  . . . . . . . . . . . . . . . . . . . . . . . . .   2
     2.1.  Case1 . . . . . . . . . . . . . . . . . . . . . . . . . .   2
     2.2.  Case2 . . . . . . . . . . . . . . . . . . . . . . . . . .   3
     2.3.  Case3 . . . . . . . . . . . . . . . . . . . . . . . . . .   3
     2.4.  Case4 . . . . . . . . . . . . . . . . . . . . . . . . . .   4
     2.5.  Case5 . . . . . . . . . . . . . . . . . . . . . . . . . .   4
     2.6.  Case6 . . . . . . . . . . . . . . . . . . . . . . . . . .   5
     2.7.  Case7 . . . . . . . . . . . . . . . . . . . . . . . . . .   6
     2.8.  Case8 . . . . . . . . . . . . . . . . . . . . . . . . . .   6
     2.9.  Case9 . . . . . . . . . . . . . . . . . . . . . . . . . .   7
     2.10. Case10  . . . . . . . . . . . . . . . . . . . . . . . . .   7
     2.11. Case11  . . . . . . . . . . . . . . . . . . . . . . . . .   8
   3.  Acknowledgements  . . . . . . . . . . . . . . . . . . . . . .   9
   Authors' Addresses  . . . . . . . . . . . . . . . . . . . . . . .   9

1.  Introduction

   This document listed 11 test cases for SDN switch.

2.  Test Cases

2.1.  Case1

   Test Case Name:Flow entry installing time

   Test Purpose:Measure how much time does the switch under test test to
   valid a flow entry.

   Test Methodology:

   1.  Setup connection between switch under test and test tool

   2.  Remove all flow entries of switch

   3.  Add a flow entry to table 0 with a match field of TCP five-tuple
   and an action OFPAT_OUTPUT to an output port

   4.  Wait enough time(e.g. 5 seconds) for switch to valid the flow
   entry

   5.  Send a matched TCP packet to the data plane, record this time as
   Tt

   6.  Receive the packet at the output port, record this time as Tr




Cheng & Bao               Expires June 24, 2017                 [Page 2]

Internet-Draft       SDN Switch Benchmark Test Cases       December 2016


   7.  The result time Tresult = Tr - Tt

   8.  Iterate this test by a pre-configured number(less than the known
   maximum flow entry number) of flow entries installed already or the
   maximum number of flow entries

2.2.  Case2

   Test Case Name:Flow table look up time

   Test Purpose:Measure how much time does the switch under test take to
   look up flow table for a matched packet

   Test Methodology:

   1.  Setup connection between switch under test and test tool

   2.  Remove all flow entries of switch

   3.  Add a flow entry to table 0 with a match field of TCP five-tuple
   and an action OFPAT_OUTPUT to an output port

   4.  Wait enough time(e.g. 5 seconds) for switch to valid the flow
   entry

   5.  Send a matched TCP packet to the data plane, record this time as
   Tt

   6.  Receive the packet at the output port, record this time as Tr

   7.  The result time Tresult = Tr - Tt

   8.  Iterate this test by a pre-configured number(less than the known
   maximum flow entry number) of flow entries installed already or the
   maximum number of flow entries

2.3.  Case3

   Test Case Name:Packet in message performance of table-miss

   Test Purpose:Measures the time sending packet_in message after a
   table-miss flow entry is triggered

   Test Methodology:

   1.  Setup connection between switch under test and test tool

   2.  Remove all flow entries of the switch under test



Cheng & Bao               Expires June 24, 2017                 [Page 3]

Internet-Draft       SDN Switch Benchmark Test Cases       December 2016


   3.  Add a table-miss flow entry matching all wildcards, and a default
   action OFPAT_OUTPUT to port OFPP_CONTROLLER

   4.  Wait enough time(e.g. 5 seconds) for switch to valid the flow
   entry

   5.  Send a packet to data plane, record this time as Tt

   6.  Receive Packet_in messages and record this time as Tr

   7.  The result time Tresult = Tr - Tt

   8.  Iterate this test with different datapath packet throughput

2.4.  Case4

   Test Case Name:Packet in message performance of none table-miss

   Test Purpose:Measures the time sending packet_in message after a
   normal flow entry is triggered

   Test Methodology:

   1.  Setup connection between switch under test and test tool

   2. remove all flow entries of switch

   3. add a flow entry to table 0 with matching in_port and an action
   OFPAT_OUTPUT to port OFPP_CONTROLLER

   4.  Wait enough time(e.g. 5 seconds) for switch to valid the flow
   entry

   5. send a packet to the matched data plane port, record this time as
   Tt

   6. receive Packet_in messages and record this time as Tr

   7.  The result time Tresult = Tr - Tt

   8.  Iterate this test with different datapath packet throughput

2.5.  Case5

   Test Case Name:Flow entry deleting time

   Test Purpose:Measure how much time does the switch under test take to
   delete a flow entry.



Cheng & Bao               Expires June 24, 2017                 [Page 4]

Internet-Draft       SDN Switch Benchmark Test Cases       December 2016


   Test Methodology:

   1.  Setup connection between switch under test and test tool

   2.  Remove all flow entries of switch

   3.  Add a flow entry to table 0 with a match field of TCP five-tuple
   and an action OFPAT_OUTPUT to an output port

   4.  Keep sending TCP packets to dataplane ingress port

   5.  Wait enough time(e.g. 3 seconds) for switch until packets are
   forwarded to the output port

   6.  Send a flow delete message, delete all flow entries, record this
   time as Tt

   7. wait until no packets received at the output port, record the time
   of the last one received packet as Tr

   8.  The result time Tresult = Tr - Tt

   9.  Iterate this test for 100 flow entries and 1000 flow entries
   installed already or the maximum number(if it is less than 1000) of
   flow entries

2.6.  Case6

   Test Case Name:Flow entry modifing time

   Test Purpose:Measure how much time does the switch under test take to
   modify a flow entry.

   Test Methodology:

   1.  Setup connection between switch under test and test tool

   2.  Remove all flow entries of switch

   3.  Add a flow entry to table 0 with a match field of TCP five-tuple
   and an action OFPAT_OUTPUT to an output port

   4.  Keep sending TCP packets to dataplane ingress port

   5.  Wait enough time(e.g. 3 seconds) for switch until packets are
   forwarded to the output port





Cheng & Bao               Expires June 24, 2017                 [Page 5]

Internet-Draft       SDN Switch Benchmark Test Cases       December 2016


   6.  Send a flow modify message, output the packets to another port,
   record this time as Tt

   7. wait until packets received at the ohter output port, record the
   time of the first received packet as Tr

   8.  The result time Tresult = Tr - Tt

   9.  Iterate this test for 100 flow entries and 1000 flow entries
   installed already or the maximum number(if it is less than 1000) of
   flow entries

2.7.  Case7

   Test Case Name:Table capacity

   Test Purpose:Measure the maximum number of flow entries of a certain
   table of a switch

   Test Methodology:

   1.  Setup connection between switch under test and test tool

   2.  Remove all flow entries of switch

   3.  Send an empty body ofp_multipart_request with type
   OFPMP_TABLE_FEATURES, to get the maximum flow entries of a specified
   table

   4.  Keep sending flow_add messages with different priorities, actions
   can be an output action to an output port

   5.  Wait until ofp_error_msg is received, with type
   OFPET_FLOW_MOD_FAILED, and error code OFPFMFC_TABLE_FULL

   6.  Record the number of flow entries sent before the ofp_error_msg
   is triggered as the test result

2.8.  Case8

   Test Case Name:Packet out message throughput/latency time

   Test Purpose:Measure the Packet_out message processing throughput and
   latency of the switch under test

   Test Methodology:

   1.  Setup connection between switch under test and test tool



Cheng & Bao               Expires June 24, 2017                 [Page 6]

Internet-Draft       SDN Switch Benchmark Test Cases       December 2016


   2.  Send packet_out messages with a certain length of packet in its
   data field and action OFPAT_OUTPUT to an output port of the switch
   under test, record the time as Tt

   3.  Receive the packet contains in the data field, record the time as
   Tr

   4.  The result Tresult = Tr - Tt

   5.  Iterate this test for different Packet_out throughput and data
   length

2.9.  Case9

   Test Case Name:Flow entry multipart_reply message test

   Test Purpose:Measure flow statistics update time of a switch

   Test Methodology:

   1.  Setup connection between switch under test and test tool

   2.  Remove all flow entries of switch

   3.  Add a flow entry to table 0 with a match field of TCP five-tuple
   and an action OFPAT_OUTPUT to an output port

   4.  Keep sending an OFPT_MULTIPART_REQUEST message with type of
   OFPMP_FLOW to request statistics of the flow added in step 3

   5.  Send a packet matches the flow added in step 3 to switch from
   dataplane, record the time as Tt

   6.  Wait enough time(e.g. 3 seconds) for switch until flow statistics
   are updated, record the time as Tr

   7.  The result Tresult = Tr - Tt

   8.  Iterate this test for different flows

2.10.  Case10

   Test Case Name:Port multipart_reply message test

   Test Purpose:Measure port statistics update time of a switch

   Test Methodology:




Cheng & Bao               Expires June 24, 2017                 [Page 7]

Internet-Draft       SDN Switch Benchmark Test Cases       December 2016


   1.  Setup connection between switch under test and test tool

   2.  Remove all flow entries of switch

   3.  Send an OFPT_MULTIPART_REQUEST message with type of
   OFPMP_PORT_STATS to request statistics of one port, record the
   rx_packets and rx_bytes

   4.  Keep sending the OFPT_MULTIPART_REQUEST message of step 3.

   5.  Send a packet to switch from dataplane, record the time as Tt

   6.  Wait enough time(e.g. 3 seconds) for switch until port statistics
   are updated, record the time as Tr

   7.  The result Tresult = Tr - Tt

   8.  Iterate this test for sending different amount and kind of
   packets to switch

2.11.  Case11

   Test Case Name:Idle time out timer performance test

   Test Purpose:Test the accuracy of idle timeout of flow entries

   Test Methodology:

   1.  Setup connection between switch under test and test tool

   2.  Remove all flow entries of switch

   3.  Install a flow entry with idle timeout 5s and set the
   OFPFF_SEND_FLOW_REM flag

   4.  Send a matching packet to ingress port within idle timeout and
   record the time as Ts

   5.  Wait until the switch send an OFP_FLOW_REMOVED message to the
   test tool, record the time as Te

   6.  The result time, Tresult = Te - Ts

   7.  Compare Tresult with the hard timeout(5s), the error percentage
   equals (Tresult - 5s)/5s * 100%

   8.  Iterate this test with different number of installed flow entries




Cheng & Bao               Expires June 24, 2017                 [Page 8]

Internet-Draft       SDN Switch Benchmark Test Cases       December 2016


3.  Acknowledgements

   Funding for the RFC Editor function is currently provided by BII
   Group.

Authors' Addresses

   Mike Cheng
   BII Group Holdings Ltd.
   Beijing
   P. R. China

   Email: mikecheng@biigroup.com


   Yaming Bao
   BII Group Holdings Ltd.
   Beijing
   P. R. China

   Email: ymbao@biigroup.cn






























Cheng & Bao               Expires June 24, 2017                 [Page 9]