Testing is still a key challenge for OPNFV.
All the projects must manage their test strategy (unit, functional, security, performance)
Several specific test projects have been validated by TSC and already deal with:
The dependencies between the feature and test projects for Brahmaputra may be described as follow:
The Testing subgroup will receive requests until milestone C, September 25th.
The test specifications shall be ready no later than milestone D, December 1st. For test specifications received after this date, the Testing subgroup will not commit to delivering the test cases in time for the release. For tests requiring development effort from the test frameworks larger than 2 weeks, test specifications received later than November 15th will be submitted to agreement, depending on test project workload.
Please add your request for B-release test here: https://etherpad.opnfv.org/p/testing_requests_for_b_release
Based on the information collected on the pad, we create a matrix to identify the dependencies between the feature and the test projects.
There are several projects dealing with integration and testing (see https://wiki.opnfv.org/).
see dedicated page
We consider the projects referenced on the wiki main page:
|Project name||Scope||R2 main goals||R2 deliverables|
|Functest||Functest will provide comprehensive testing methodology, test suites and test cases to test and verify OPNFV Platform functionality that covers the VIM and NFVI components.|| - Automation of vIMS |
reference testcases in a reference OPNFV global testcase doc
- Work on testcase auto declaration
|User guide, Installation guide + common testcase reference guide|
|Pharos*||The Pharos project deals with the creation of a federated NFV test capability that is geographically and technically diverse and hosted by different companies in the OPNFV community. This requires developing a baseline specification for an OPNFV "compliant" test environment along with tools, processes and documentation to support integration, testing and collaborative development projects with needed infrastructure and the tooling.|
|PolicyTest||PolicyTest adds network policy testing to the existing suite of OPNFV functional tests.||PolicyTest tickets in Jira|
|Qtip||Qtip project is a "Bottom-Up" approach in characterizing and benchmarking OPNFV Platform. Proper characterization of OPNFV platforms is a critical component to understand how well OPNFV (and by implication upstream components) perform in realistic deployment scenarios, to provide feedback to design teams, to provide platform level performance data to higher layer developers, users, and community at large. Qtip also aims to build necessary testing and benchmarking tools that will be needed to fulfill its goals, to automate when possible, and to provide these tools and automation software to upper layer and other testing projects in OPNFV.||Benchmark suite to provide quantitatve performane metrics for OPNFV platform's compute, networking and storage components|
|VSPERF||develop a generic and architecture agnostic vSwitch testing framework and associated tests, that will serve as a basis for validating the suitability of different vSwitch implementations in a Telco NFV deployment environment|| Our goals for the VSPERF test framework this release are: |
Support our 3 base deployment scenarios: P2P, PVP and PVVP
CI Support and integration for VSPERF
Support Vanilla OVS and OVS with DPDK.
Support all RFC2544 defined tests.
Support IMIX traffic testing with RFC 2544.
Support IXIA, Spirent, Moongen and Xena.
Participate in opnfv_test_dashboard
Support IP routing on the switch, as well as simple port forwarding
RFC2544 Scalability tests (Implementation)
|Yardstick||The goal of the Yardstick Project is to verify the infrastructure compliance when running VNF applications.|| - Methodology for Infrastructure verification from VNF perspective, aligned with ETSI-NFV TST001 |
- Generic test cases for Compute, Storage and Networking areas
- OPNFV Infrastructure verification from VTC and SFC perspective
- Mobility Traffic Profile as Test Stimuli
*: Pharos is not a test project as it does neither define test cases not test tools. Pharos is in charge of the testbeds that can be used by test projects.
As OPNFV is dealing with lots of testing projects, it could make sense to share some definitions..
TBD severity …
TBD who is doing what / …
TBD KPI of a Telco Cloud…
The test coverage is not an easy task. Test projects must focus on NFVI, however, in order to test efficiently NFVI, it may be usefull to test VNFs (which are out of OPNFV scope). We may suggest several views:
See dedicated testing coverage page
Telco representative traffic profiles are necessary for the tests. See Traffic profiles page
A test dashboard shall give a good overview of the different tests on the different PODs. See Test Dashboard wiki