benchmarking

This document describes an alternative to throughput in "Benchmarking Methodology for Network Interconnect Devices" (RFC 2544) by defining a new methodology called Multiple Loss Ratio Search (MLRsearch). MLRsearch aims to minimize Search Duration, support multiple loss ratio goals, and improve result repeatability and comparability.

MLRsearch is motivated by the pressing need to address the challenges of evaluating and testing the various data plane solutions, especially in software-based networking systems based on Commercial Off-the-Shelf (COTS) CPU hardware vs. purpose-built Application-Specific Integrated Circuit (ASIC) / Network Processing Unit (NPU) / Field-Programmable Gate Array (FPGA) hardware.

RFC 2544 defines a benchmarking methodology for network interconnect devices. RFC 5180 addresses IPv6 specificities, and it also provides a technology update but excludes IPv6 transition technologies. RFC 8219 addresses IPv6 transition technologies, including stateful NAT64. However, none of them discuss how to apply pseudorandom port numbers from RFC 4814 to any stateful NATxy (such as NAT44, NAT64, and NAT66) technologies. This document discusses why using pseudorandom port numbers with stateful NATxy gateways is a difficult problem. It recommends a solution that limits the port number ranges and uses two test phases (phase 1 and phase 2). This document shows how the classic performance measurement procedures (e.g., throughput, frame loss rate, latency, etc.) can be carried out. New performance metrics and measurement procedures are also defined for measuring the maximum connection establishment rate, connection tear-down rate, and connection tracking table capacity.

This document provides benchmarking terminology and methodology for next-generation network security devices, including next-generation firewalls (NGFWs) and next-generation intrusion prevention systems (NGIPSs). The main areas covered in this document are test terminology, test configuration parameters, and benchmarking methodology for NGFWs and NGIPSs. (It is assumed that readers have a working knowledge of these devices and the security functionality they contain.) This document aims to improve the applicability, reproducibility, and transparency of benchmarks and to align the test methodology with today's increasingly complex layer 7 security-centric network application use cases. As a result, this document makes RFC 3511 obsolete.

Fundamental benchmarking methodologies for network interconnect devices of interest to the IETF are defined in RFC 2544. This memo updates the procedures of the test to measure the Back-to-Back Frames benchmark of RFC 2544, based on further experience.

This memo updates Section 26.4 of RFC 2544.

This document defines terminology for benchmarking a Software-Defined Networking (SDN) controller's control-plane performance. It extends the terminology already defined in RFC 7426 for the purpose of benchmarking SDN Controllers. The terms provided in this document help to benchmark an SDN Controller's performance independently of the controller's supported protocols and/or network services.

This document defines methodologies for benchmarking the control-plane performance of Software-Defined Networking (SDN) Controllers. The SDN Controller is a core component in the SDN architecture that controls the behavior of the network. SDN Controllers have been implemented with many varying designs in order to achieve their intended network functionality. Hence, the authors of this document have taken the approach of considering an SDN Controller to be a black box, defining the methodology in a manner that is agnostic to protocols and network services supported by controllers. This document provides a method for measuring the performance of all controller implementations.

This memo describes the contributions of the Open Platform for NFV (OPNFV) project on Virtual Switch Performance (VSPERF), particularly in the areas of test setups and configuration parameters for the system under test. This project has extended the current and completed work of the Benchmarking Methodology Working Group in the IETF and references existing literature. The Benchmarking Methodology Working Group has traditionally conducted laboratory characterization of dedicated physical implementations of internetworking functions. Therefore, this memo describes the additional considerations when virtual switches are implemented on general-purpose hardware. The expanded tests and benchmarks are also influenced by the OPNFV mission to support virtualization of the "telco" infrastructure.

The purposes of this informational document are to establish definitions and describe measurement techniques for data center benchmarking, as well as to introduce new terminology applicable to performance evaluations of data center network equipment. This document establishes the important concepts for benchmarking network switches and routers in the data center and is a prerequisite for the test methodology document (RFC 8239). Many of these terms and methods may be applicable to network equipment beyond the scope of this document as the technologies originally applied in the data center are deployed elsewhere.

The purpose of this informational document is to establish test and evaluation methodology and measurement techniques for physical network equipment in the data center. RFC 8238 is a prerequisite for this document, as it contains terminology that is considered normative. Many of these terms and methods may be applicable beyond the scope of this document as the technologies originally applied in the data center are deployed elsewhere.

Benchmarking methodologies that address the performance of network interconnect devices that are IPv4- or IPv6-capable exist, but the IPv6 transition technologies are outside of their scope. This document provides complementary guidelines for evaluating the performance of IPv6 transition technologies. More specifically, this document targets IPv6 transition technologies that employ encapsulation or translation mechanisms, as dual-stack nodes can be tested using the recommendations of RFCs 2544 and 5180. The methodology also includes a metric for benchmarking load scalability.

The Benchmarking Methodology Working Group has traditionally conducted laboratory characterization of dedicated physical implementations of internetworking functions. This memo investigates additional considerations when network functions are virtualized and performed in general-purpose hardware.

This document provides benchmarking procedures for the Neighbor Discovery Protocol (NDP). It also proposes metrics by which an NDP implementation's scaling capabilities can be measured.

BGP is widely deployed and used by several service providers as the default inter-AS (Autonomous System) routing protocol. It is of utmost importance to ensure that when a BGP peer or a downstream link of a BGP peer fails, the alternate paths are rapidly used and routes via these alternate paths are installed. This document provides the basic BGP benchmarking methodology using existing BGP convergence terminology as defined in RFC 4098.

Modern forwarding devices attempt to minimize any control- and data-plane disruptions while performing planned software changes by implementing a technique commonly known as In-Service Software Upgrade (ISSU). This document specifies a set of common methodologies and procedures designed to characterize the overall behavior of a Device Under Test (DUT), subject to an ISSU event.

This framework describes a practical methodology for benchmarking the traffic management capabilities of networking devices (i.e., policing, shaping, etc.). The goals are to provide a repeatable test method that objectively compares performance of the device's traffic management capabilities and to specify the means to benchmark traffic management with representative application traffic.

This document provides a terminology for benchmarking the Session Initiation Protocol (SIP) performance of devices. Methodology related to benchmarking SIP devices is described in the companion methodology document (RFC 7502). Using these two documents, benchmarks can be obtained and compared for different types of devices such as SIP Proxy Servers, Registrars, and Session Border Controllers. The term "performance" in this context means the capacity of the Device Under Test (DUT) to process SIP messages. Media streams are used only to study how they impact the signaling behavior. The intent of the two documents is to provide a normalized set of tests that will enable an objective comparison of the capacity of SIP devices. Test setup parameters and a methodology are necessary because SIP allows a wide range of configurations and operational conditions that can influence performance benchmark measurements. A standard terminology and methodology will ensure that benchmarks have consistent definitions and were obtained following the same procedures.

This document provides a methodology for benchmarking the Session Initiation Protocol (SIP) performance of devices. Terminology related to benchmarking SIP devices is described in the companion terminology document (RFC 7501). Using these two documents, benchmarks can be obtained and compared for different types of devices such as SIP Proxy Servers, Registrars, and Session Border Controllers. The term "performance" in this context means the capacity of the Device Under Test (DUT) to process SIP messages. Media streams are used only to study how they impact the signaling behavior. The intent of the two documents is to provide a normalized set of tests that will enable an objective comparison of the capacity of SIP devices. Test setup parameters and a methodology are necessary because SIP allows a wide range of configurations and operational conditions that can influence performance benchmark measurements.

Benchmarking methodologies have always relied on test conditions with constant packet sizes, with the goal of understanding what network device capability has been tested. Tests with a constant packet size reveal device capabilities but differ significantly from the conditions encountered in operational deployment, so additional tests are sometimes conducted with a mixture of packet sizes, or "IMIX" ("Internet Mix"). The mixture of sizes a networking device will encounter is highly variable and depends on many factors. An IMIX suited for one networking device and deployment will not be appropriate for another. However, the mix of sizes may be known, and the tester may be asked to augment the fixed-size tests. To address this need and the perpetual goal of specifying repeatable test conditions, this document defines a way to specify the exact repeating sequence of packet sizes from the usual set of fixed sizes and from other forms of mixed-size specification.

This document describes the methodology for benchmarking MPLS Fast Reroute (FRR) protection mechanisms for link and node protection. This document provides test methodologies and testbed setup for measuring failover times of Fast Reroute techniques while considering factors (such as underlying links) that might impact recovery times for real-time applications bound to MPLS Traffic Engineered (MPLS-TE) tunnels.

The Benchmarking Methodology Working Group (BMWG) has been developing key performance metrics and laboratory test methods since 1990, and continues this work at present. The methods described in RFC 2544 are intended to generate traffic that overloads network device resources in order to assess their capacity. Overload of shared resources would likely be harmful to user traffic performance on a production network, and there are further negative consequences identified with production application of the methods. This memo clarifies the scope of RFC 2544 and other IETF BMWG benchmarking work for isolated test environments only, and it encourages new standards activity for measurement methods applicable outside that scope. This document is not an Internet Standards Track specification; it is published for informational purposes.

This document provides a methodology and framework for quantifying the performance impact of the monitoring of IP flows on a network device and the export of this information to a Collector. It identifies the rate at which the IP flows are created, expired, and successfully exported as a new performance metric in combination with traditional throughput. The metric is only applicable to the devices compliant with RFC 5470, "Architecture for IP Flow Information Export". The methodology quantifies the impact of the IP flow monitoring process on the network equipment. This document is not an Internet Standards Track specification; it is published for informational purposes.

This document describes the terminology for benchmarking link-state Interior Gateway Protocol (IGP) route convergence. The terminology is to be used for benchmarking IGP convergence time through externally observable (black-box) data-plane measurements. The terminology can be applied to any link-state IGP, such as IS-IS and OSPF. This document is not an Internet Standards Track specification; it is published for informational purposes.

This document describes the methodology for benchmarking Link-State Interior Gateway Protocol (IGP) Route Convergence. The methodology is to be used for benchmarking IGP convergence time through externally observable (black-box) data-plane measurements. The methodology can be applied to any link-state IGP, such as IS-IS and OSPF. This document is not an Internet Standards Track specification; it is published for informational purposes.

This document provides common terminology and metrics for benchmarking the performance of sub-IP layer protection mechanisms. The performance benchmarks are measured at the IP layer; protection may be provided at the sub-IP layer. The benchmarks and terminology can be applied in methodology documents for different sub-IP layer protection mechanisms such as Automatic Protection Switching (APS), Virtual Router Redundancy Protocol (VRRP), Stateful High Availability (HA), and Multiprotocol Label Switching Fast Reroute (MPLS-FRR). This document is not an Internet Standards Track specification; it is published for informational purposes.

An operational forwarding device may need to be restarted (automatically or manually) for a variety of reasons, an event called a "reset" in this document. Since there may be an interruption in the forwarding operation during a reset, it is useful to know how long a device takes to resume the forwarding operation.

This document specifies a methodology for characterizing reset (and reset time) during benchmarking of forwarding devices and provides clarity and consistency in reset test procedures beyond what is specified in RFC 2544. Therefore, it updates RFC 2544. This document also defines the benchmarking term "reset time" and, only in this, updates RFC 1242. This document is not an Internet Standards Track specification; it is published for informational purposes.

This document defines a set of metrics and their usage to evaluate the performance of end-to-end Session Initiation Protocol (SIP) for telephony services in both production and testing environments. The purpose of this document is to combine a standard set of common metrics, allowing interoperable performance measurements, easing the comparison of industry implementations. [STANDARDS-TRACK]

This document describes a methodology specific to the benchmarking of Multiprotocol Label Switching (MPLS) forwarding devices, limited to the most common MPLS packet forwarding scenarios and delay measurements for each, considering IP flows. It builds upon the tenets set forth in RFC 2544, RFC 1242, and other IETF Benchmarking Methodology Working Group (BMWG) efforts. This document seeks to extend these efforts to the MPLS paradigm. This memo provides information for the Internet community.

The benchmarking methodologies defined in RFC 2544 are IP version independent. However, RFC 2544 does not address some of the specificities of IPv6. This document provides additional benchmarking guidelines, which in conjunction with RFC 2544, lead to a more complete and realistic evaluation of the IPv6 performance of network interconnect devices. IPv6 transition mechanisms are outside the scope of this document. This memo provides information for the Internet community.

The primary purpose of this document is to define terminology specific to the benchmarking of resource reservation signaling of Integrated Services (IntServ) IP routers. These terms can be used in additional documents that define benchmarking methodologies for routers that support resource reservation or reporting formats for the benchmarking measurements. This memo provides information for the Internet community.

Test engineers take pains to declare all factors that affect a given measurement, including intended load, packet length, test duration, and traffic orientation. However, current benchmarking practice overlooks two factors that have a profound impact on test results. First, existing methodologies do not require the reporting of addresses or other test traffic contents, even though these fields can affect test results. Second, "stuff" bits and bytes inserted in test traffic by some link-layer technologies add significant and variable overhead, which in turn affects test results. This document describes the effects of these factors; recommends guidelines for test traffic contents; and offers formulas for determining the probability of bit- and byte-stuffing in test traffic. This memo provides information for the Internet community.

This document describes terminology for the benchmarking of devices that implement traffic control using packet classification based on defined criteria. The terminology is to be applied to measurements made on the data plane to evaluate IP traffic control mechanisms. Rules for packet classification can be based on any field in the IP header, such as the Differentiated Services Code Point (DSCP), or any field in the packet payload, such as port number. This memo provides information for the Internet community.

This document establishes terminology to standardize the description of benchmarking methodology for measuring eBGP convergence in the control plane of a single BGP device. Future documents will address iBGP convergence, the initiation of forwarding based on converged control plane information and multiple interacting BGP devices.This terminology is applicable to both IPv4 and IPv6. Illustrative examples of each version are included where relevant. This memo provides information for the Internet community.

This document provides suggestions for measuring OSPF single router control plane convergence. Its initial emphasis is on the control plane of a single OSPF router. We do not address forwarding plane performance.

NOTE: In this document, the word "convergence" relates to single router control plane convergence only. This memo provides information for the Internet community.

This document explains the terminology and concepts used in OSPF benchmarking. Although some of these terms may be defined elsewhere (and we will refer the reader to those definitions in some cases) we include discussions concerning these terms, as they relate specifically to the tasks involved in benchmarking the OSPF protocol. This memo provides information for the Internet community.

This document discusses the applicability of various tests for measuring single router control plane convergence, specifically in regard to the Open Shortest First (OSPF) protocol. There are two general sections in this document, the first discusses advantages and limitations of specific OSPF convergence tests, and the second discusses more general pitfalls to be considered when routing protocol convergence is tested. This memo provides information for the Internet community.

The purpose of this document is to describe methodology specific to the benchmarking of multicast IP forwarding devices. It builds upon the tenets set forth in RFC 2544, RFC 2432 and other IETF Benchmarking Methodology Working Group (BMWG) efforts. This document seeks to extend these efforts to the multicast paradigm.

The BMWG produces two major classes of documents: Benchmarking Terminology documents and Benchmarking Methodology documents. The Terminology documents present the benchmarks and other related terms. The Methodology documents define the procedures required to collect the benchmarks cited in the corresponding Terminology documents. This memo provides information for the Internet community.

This document describes the terms to be used in a methodology that determines the IP packet forwarding performance of IP routers as a function of the forwarding information base installed within a router. The forwarding performance of an IP router may be dependent upon or may be linked to the composition and size of the forwarding information base installed within a router. This memo provides information for the Internet community.

This document discusses and defines a number of tests that may be used to describe the performance characteristics of ATM (Asynchronous Transfer Mode) based switching devices. In addition to defining the tests this document also describes specific formats for reporting the results of the tests. This memo provides information for the Internet community.

This memo discusses and defines terms associated with performance benchmarking tests and the results of these tests in the context of frame relay switching devices. This memo provides information for the Internet community.

This memo discusses and defines terms associated with performance benchmarking tests and the results of these tests in the context of Asynchronous Transfer Mode (ATM) based switching devices supporting ABR (Available Bit Rate). This memo provides information for the Internet community.

This document is intended to provide methodology for the benchmarking of local area network (LAN) switching devices. This memo provides information for the Internet community.

This memo discusses and defines terms associated with performance benchmarking tests and the results of these tests in the context of Asynchronous Transfer Mode (ATM) based switching devices. This memo provides information for the Internet community.

This document defines terms used in measuring the performance of firewalls. It extends the terminology already used for benchmarking routers and switches with definitions specific to firewalls. [STANDARDS-TRACK]

This document is a republication of RFC 1944 correcting the values for the IP addresses which were assigned to be used as the default addresses for networking test equipment. This memo provides information for the Internet community.

The purpose of this document is to define terminology specific to the benchmarking of multicast IP forwarding devices. This memo provides information for the Internet community.

This document is intended to provide terminology for the benchmarking of local area network (LAN) switching devices. It extends the terminology already defined for benchmarking network interconnect devices in RFCs 1242 and 1944 to switching devices. This memo provides information for the Internet community. It does not specify an Internet standard of any kind.

This memo discusses and defines a number of terms that are used in describing performance benchmarking tests and the results of such tests. This memo provides information for the Internet community. It does not specify an Internet standard.