Differences

This shows you the differences between two versions of the page.

--- wiki:ltd_test_spec_overview [2015/09/13 14:36]
Al Morton [List of Tests and Descriptions]
+++ wiki:ltd_test_spec_overview [2015/12/15 23:04] (current)
Al Morton [LTD Test Spec Overview]
@@ Line 1: / Line 1: @@
 ====== LTD Test Spec Overview ======
-(This page is currently Under Construction)
+(This page is currently Under Construction, but now describes all tests as of Sept 17, 2015)
-The Level Test Design (LTD) Specification is one of the products of the OPNFV vsperf project.  This page provides an overview of the LTD spec in several forms.
+The Level Test Design (LTD) Specification is one of the products of the OPNFV vsperf project. The LTD spec is formatted in ReStructured Text (RST), and presented in HTML form with the project [[http://artifacts.opnfv.org/vswitchperf/docs/test_spec/vswitchperf_ltd.html |Artifacts]].  This wiki page provides an overview of the LTD spec in several forms:
-  - The LTD Spec and its goals are Summarized in a 20 page [[https://tools.ietf.org/html/draft-vsperf-bmwg-vswitch-opnfv-00|Internet Draft ]]
+  - The LTD Spec and its goals are Summarized in a 21 page [[https://tools.ietf.org/html/draft-vsperf-bmwg-vswitch-opnfv-01|Internet Draft ]]
-  - Short descriptions of each of the tests in the LTD Spec
+  - Short descriptions of each of the tests in the LTD Spec (below)
-  - Example Device Under Test (DUT) Set-ups, called Deployment Scenarios
+  - Example Device Under Test (DUT) Set-ups, called Deployment Scenarios at [[https://wiki.opnfv.org/vsperf/test_methodology|Test Methodology ]] page.
+The approach VSPERF has adopted is to take existing tests that are relevant to performance benchmarking of physical switches, and applying them to benchmarking virtual switches (this is to allow for a fair comparison with their physical counterparts). Once this is done, tests that are relevant to benchmarking virtual switches only, will be developed.
 ===== List of Tests and Descriptions =====
@@ Line 12: / Line 14: @@
 === Speed of Activation ===
-   o  Throughput.RFC2889.AddressLearningRate
+   o  Activation.RFC2889.AddressLearningRate
 This test measures the address learning rate of the DUT, in accordance with [[https://tools.ietf.org/html/rfc2889#section-5.8 |section 5.8 of RFC 2889]]. A minimum of 3 DUT ports are required for this test, a learning port where frames first arrive at the DUT, a test port where frames are forwarded and returned to the source, and a monitoring port where the test device listens for flooded or miss-forwarded frames.
@@ Line 21: / Line 23: @@
 In some virtual switch architectures, the first packets of a flow will take the system longer to process than subsequent packets in the flow. This test determines the latency for these packets. The test will measure the latency of the packets as they are processed by the flow-setup-path of the DUT.
+=== Accuracy of Activation ===
+   o  CPDP.Coupling.Flow.Addition
+This Control path Data path Coupling (CPDP) test is a variation on a [[https://tools.ietf.org/html/rfc2544#section-26.1 |RFC 2544]] Throughput test, where the vswitch controller adds and removes flow entries while measuring the data-path performance. The test will determine if activity on the vswitch control (north-bound) interface affects performance in the data-path. The test also assesses average latency during control path activity.
+It is possible to compare the CPDP results with a [[https://tools.ietf.org/html/rfc2544#section-26.1 |RFC 2544]] Throughput test conducted without the control activity, to more clearly quantify the effects, if any.  **Throughput.RFC2544.PacketLossRatio** is not a prerequisite test.
 === Reliability of Activation ===
    o  Throughput.RFC2544.SystemRecoveryTime
+This test measures the DUT's time to recover from exposure to a traffic overload. The recommended overload level is 150% of the offered load corresponding to the Throughput level as determined using [[https://tools.ietf.org/html/rfc2544#section-26.1 |section 26.1 of RFC 2544]]. The test also assess the packet latency at the Throughput level, and checks to assure that the latency has returned to the pre-overload average (within a 10% tolerance).
+Therefore, the **Throughput.RFC2544.PacketLossRatio** is a prerequisite test.
    o  Throughput.RFC2544.ResetTime
+Occasionally, network functions need to be reset to restore normal operation. This test described in [[https://tools.ietf.org/html/rfc6201 | RFC 6201]] updates the procedures in [[https://tools.ietf.org/html/rfc2544#section-26.6 |section 26.2 of RFC 2544]], defining a new benchmark, Reset Time.
 === Scale of Activation ===
-   o  Throughput.RFC2889.AddressCachingCapacity
+   o  Activation.RFC2889.AddressCachingCapacity
 This test measures the address cache capacity of the DUT, in accordance with [[https://tools.ietf.org/html/rfc2889#section-5.7 |section 5.7 of RFC 2889]]. A minimum of 3 DUT ports are required for this test, a learning port where frames first arrive at the DUT, a test port where frames are forwarded and returned to the source, and a monitoring port where the test device listens for flooded or miss-forwarded frames.
@@ Line 43: / Line 60: @@
 This test also assesses the average one-way latency for successfully received packets (note that the average may be misleading if the latency distribution is bi-modal). All Latency tests called for deviate from [[https://tools.ietf.org/html/rfc2544#section-26.2 |section 26.2 of RFC 2544]], and assess the metric more thoroughly.
-   o  Throughput.RFC2544.PacketLossRateFrameModification
+   o  CPU.RFC2544.0PacketLoss
+This CPU intensive test is a variation on a [[https://tools.ietf.org/html/rfc2544#section-26.1 |RFC 2544]] Throughput test (0% loss), where a CPU-intensive application is running on the same DUT as the vswitch. The test will determine if stressful CPU activity on all non-vswitch Cores affects performance in the data-path.
+Although **Throughput.RFC2544.PacketLossRatio** is not a prerequisite test, the results may provide a baseline for unmeasurable affect of a CPU intensive application.
+   o  Throughput.RFC2544.PacketLossRatioFrameModification
 This test assesses the [[https://tools.ietf.org/html/rfc2544#section-26.1 | RFC 2544]] Throughput level with additional processes of packet inspection and modification. The Recommended modification is to change or add a VLAN tag, but many more possibilities exist.
@@ Line 69: / Line 92: @@
 The forwarding rates of Broadcast Frames are assessed in this test, where at least 4 ports are used and 3 are monitored for broadcast transmissions. [[https://tools.ietf.org/html/rfc2889#section-5.10 |Section 5.10 of RFC 2889]] describes the test. It is not intended to overload ingress or egress ports during this test. An error condition occurs if any broadcast frames appear on the paired egress side of their ingress port. The Frame Loss Ratio of [[https://tools.ietf.org/html/rfc2544#section-26.3 | RFC 2544]] should also be reported.
-   o  RFC2889 Broadcast Frame Latency test
+This test also assesses the one-way latency for successfully transmitted broadcast frames, averaged over the latency for each packet broadcasted to all egress ports. [[https://tools.ietf.org/html/rfc2889#section-5.10 |Section 5.10 of RFC 2889]] describes this aspect of the test, as well.
-This test assesses the one-way latency for successfully transmitted broadcast frames, averaged over the latency for each packet broadcasted to all egress ports. [[https://tools.ietf.org/html/rfc2889#section-5.10 |Section 5.10 of RFC 2889]] describes the test.
 === Accuracy of Operation ===
@@ Line 80: / Line 101: @@
 Although [[https://tools.ietf.org/html/rfc2889#section-5.9 |Section 5.9 of RFC 2889]] indicates this test is a pass or fail outcome, it is appropriate to report the offered load of correct frames and the resulting transmitted frame rate.
+   o Throughput.RFC2544.Profile
+This is a test to characterize the loss ratio and latency in the overload region, beyond the [[https://tools.ietf.org/html/rfc2544#section-26.1 | RFC 2544]] Throughput level (with 0% loss ratio). The offered load to achieve the Maximum Forwarding Rate in [[https://tools.ietf.org/html/rfc2889#section-5.1 |Section 5.1 of RFC 2889]] testing prescribes the maximum load for this test.
+Increased traffic expressed as a percentage of the difference between the Throughput level and the Maximum Offered Load (a physical line rate limitation) is applied to the DUT, in values of -50%, -10%, 0%, +10% and +50%. The Forwarding Rate (frames and Mbps), Latency, CPU & Memory Utilization, and any failures observed should be recorded for all frame sizes tested.
 === Reliability of Operation ===
@@ Line 85: / Line 113: @@
    o  Throughput.RFC2889.Soak
-This is the long duration, or "Soak" test to assess the stability of conditions measured in **Throughput.RFC2544.PacketLossRatio** test. Ideally, the Soak test duration is 24 hours.
+This is the long duration, or "Soak" test to assess the stability of conditions measured in the **Throughput.RFC2544.PacketLossRatio** test. Ideally, the Soak test duration is 24 hours.
    o  Throughput.RFC2889.SoakFrameModification
-This is the long duration, or "Soak" test to assess the stability of conditions measured in **Throughput.RFC2544.PacketLossRateFrameModification** test. Ideally, the Soak test duration is 24 hours.
+This is the long duration, or "Soak" test to assess the stability of conditions measured in the **Throughput.RFC2544.PacketLossRateFrameModification** test. Ideally, the Soak test duration is 24 hours.
+   o  PacketDelayVariation.RFC3393.Soak
+This is the long duration, or "Soak" test to assess the stability of delay distributions measured in the **Throughput.RFC2544.PacketLossRatio** test. Ideally, the Soak test duration is 24 hours. The [[https://tools.ietf.org/html/rfc5481 | RFC 5481]] PDV form of delay variation is measured on the traffic, using the 99th percentile, for each 60 second interval during the test. This extended period test will determine of there are any long delays and their frequency of occurrence.
+The prerequisite test is the **Throughput.RFC2544.PacketLossRatio**, which sets the Throughput level for each frame size.
+=== Scalability of Operation ===
+   o  Scalability.RFC2544.0PacketLoss
+This test assesses the [[https://tools.ietf.org/html/rfc2544#section-26.1 | RFC 2544]] Throughput level with a variable number of flows present in the offered load, as recommended in [[https://tools.ietf.org/html/rfc2544#section-12 | section 12 of RFC 2544]], where changing the destination address constitutes a new flow. The number of flows range from 1000 to the Maximum number of supported flows.
+   o  MemoryBandwidth.RFC2544.0PacketLoss.Scalability
+This test assesses the [[https://tools.ietf.org/html/rfc2544#section-26.1 | RFC 2544]] Throughput level while a memory intensive application is running on the same DUT as the vswitch. The test will determine if stressful memory and cache activity, with corresponding memory bus activity, affects performance in the data-path. The test description provides specific requirements for the memory intensive application, including read:write ratio, NUMA architecture considerations, local/remote memory considerations, etc.
+Although **Throughput.RFC2544.PacketLossRatio** is not a prerequisite test, the results may provide a baseline for unmeasurable affect of a memory/cache intensive application.

Wiki

User Tools

Site Tools

Differences

Page Tools