Project: Virtual Provider Edge (vPE) Performance testing

• Proposed name for the project: vPE Performance testing
• Proposed name for the repository: opnfv_vPE_perf_test_repo
• Project Categories: Integration & Testing

Project “vPE Performance testing” will provide comprehensive testing methodology, test suites and test cases to test and the performance of the OPNFV platform under the vPE L3VPN over MPLS use case.

This project is focused on measuring the performance of the different OPNFV platform components like the compute server, hyper-visor, Open vSwitch, Linux Bridge etc., that are involved in the control and data path of the vPE L3 VPN use case. The focus of the performance measurement will be on the following aspects when the vPE is subjected to realistic traffic patterns like that of a real world PE router:

• The performance of the components involved in the vPE data path under traffic created by the CE to CE data, CE-to-Internet flows.
• The performance of the components involved in the vPE control path under traffic created by bfd, route programming, QOS programming)

This project will develop test suites that cover detailed vPE L3VPN performance measurement test cases, test methodologies and platform configurations which will be documented and maintained in a repository for use by other OPNFV testing projects and the community in general.

Approach

The project targets testing of the OPNFV platform in a hosted test-bed environment (i.e. using the OPNFV test labs world wide). The project will leverage the "OPNFV: Base system Functionality testing" project and will leverage the infrastructure, test tools, automation framework delivered bu this project.

The picture below outlines the main test topology used in the vPE performance testing.

vPE Performance test project will deliver a performance testing system along with a set of test traffic patterns, performance measuring metrics, test suites and test cases to test the performance of OPNFV platform when used in the vPE use case under the specific L3VPN over MPLS. The performance measurement is targeted at the following entities in the OPNFV Platform. NOTE: These tests are performed on the VNF, however, the metrics measured under these traffic patterns are on the OPNFV platform that are specified further down below.

• Control Plane setup as a black box with focus on OPNFV platform characteristics.
  • vPE VNF bring-p and set-up time.
  • vPE VNF Destroy and tear down time.
  • VNF Global Routing Convergence time
  • VNF FIB download and ready for forwarding time.
  • L3 VPN VRF bring-up and routing convergence time.
  • Filters (QOS and ACL) programming time.
  • BFD Setup time
• Data Plane performance
  • Data Path non stop forwarding for 24 hours
  • Mix of Traffic Patterns (small, medium and large packets only, iMix traffic etc.)
  • Traffic with mix of IPV4 packet prefixes (/8, /16, /24, /30) at the prescribed ratios.
  • Traffic subjected to packet filters on a per VRF with a maximum of 40 Filters per VRF
  • Traffic subjected QOS treatment with four classes of traffic unclassified, RT, AF, BE.

The following metrics are measured under the above testcases and traffic patterns. Please refer to the follwoing figure to get more clarity:

• Black Box Metrics (VNF as a Black Box)
  • Data NIC bandwidth (bits per second), PPS (Packets Per Second), BPS (Bytes per second), Latency(measured between the ingress and egress of the Physical NICs), Jitter
  • Open vSwitch External Bridge metrics (same as the above).
  • Open vSwitch Integration Bridge metrics (same as the above).
  • VNF Packet Ingress Tap Interface metrics (same as the above)
  • VNF Packet Egress Tap Interface metrics (same as the above)
  • VNF Threads CPU Utilization, Average, Mix, Max during the test interval (per test case)
  • VNF Memory Utilization, Average, Min, Max during the test interval.
  • Cache Performance at the HOST OS(hits, misses)
  • Context Switches Host OS (global)
  • Hypervisor VM Entry/Exits
• VNF White Box metrics
  • Applications inside the VNF (Routing Protocols, BFD, FIBMgr etc.,) CPU utilization (Min, Max, Average)
  • Fib forwarding performance for different traffic mixes, (Min, Average, Max)
  • Per Application Memory Utllization
  • Cache ?

TBD: Add a figure detailing the vswitch bridges etc.

The project requires the following components:

• OPNFV Lab setup with complete access to a set of Bare metal servers for Controller and Compute nodes (minimum configuration: 1x installer server, 2x controller node servers, 3x compute node servers); associated switches and routers.
• OPNFV platform software bundle from the repository that includes several upstream software components.
• OPNFV "Get Started!" software bundles that includes the installer
• OPNFV: Basic Functionality Testing project software bundles, Test suites and test cases.
• Tempest framework and Robot Framework
• Third Party vPE VNFs

The following link provides the detailed test suites and drills down into the test suites/test cases to cover the functional testing.

Function Test Suites

The above test suites/cases are automated via Robot/Jenkins):

• The project leverages "Boot Strap/Get Started" and the "OPNFV Basic Functionality Testing Project"
• The project relies on the following upstream projects:
  • OpenStack Juno release: Components: Nova, Glance, Keystone, Horizon, Neutron, Ceilometer, Heat
  • OpenDaylight Helium release: Components: MDSAL, OVSDB, RESTCONF, ML2 plugin/ODL Neutron drivers
  • Installer: TBD
  • Puppet (for instance configuration)
  • QEMU/KVM
  • OpenWRT (as example VNF - for routing, firewall, NAT)
  • Snort (as example VNF - for IDS)
  • Linux/Centos 7 distribution

• Comitters
  • Palani Chinnakannan (pals@cisco.com)
  • Trevor Cooper (trevor.cooper@intel.com)
  • Prabu.K (Prabu.Kuppuswamy@spirent.com)
  • Ian Wells (iawelss@cisco.com)
  • Chitti Nimmagadda (chitti.nimmagadda@ericsson.com)
• Contributors
  • Palani Chinnakannan (pals@cisco.com)
  • Trevor Cooper (trevor.cooper@intel.com)
  • Frank Brockners (fbrockne@cisco.com)
  • Prabu.K (Prabu.Kuppuswamy@spirent.com)
  • Ian Wells (iawelss@cisco.com)
  • Sajeev Manikkoth (sajeevmanikkoth@gmail.com)
  • Chitti Nimmagadda (chitti.nimmagadda@ericsson.com)

Project deliverable: The project delivers the following components:

• Documentation of Test Suites and test cases that cover the vPE peformance testing including all the metrics measured and various graphs.
• Test software and scripts for testing vPE Performance
• Automated Test Framework based on Tempest/Robot/Jenkins
• Identify Key components in the OPNFV platform that includes further development and improvement to make OPNFV a carrier class platform for hosting VNFs

OPNFV release #1.