This is an old revision of the document!
Table of Contents
Testing page
Introduction
Testing is still a key challenge for OPNFV.
All the projects must manage their test strategy (unit, functional, security, performance)
Several specific test projects have been validated by TSC and already deal with:
- Define testcases
- Perform tests not covered by a single project
- Create tooling
- Study Performance end to end
OPNFV Brahmaputra Release Test
The Testing subgroup will receive requests until milestone C, September 25th.
The test specifications shall be ready no later than milestone D, December 1st. For test specifications received after this date, the Testing subgroup will not commit to delivering the test cases in time for the release. For tests requiring development effort from the test frameworks larger than 2 weeks, test specifications received later than November 15th will be submitted to agreement, depending on test project workload.
Please add your request for B-release test here: https://etherpad.opnfv.org/p/testing_requests_for_b_release
OPNFV Test ecosystem
There are several projects dealing with integration and testing (see https://wiki.opnfv.org/).
Overview
A global overview can be described as follow:
Guidelines for new Test projects
see dedicated page
Project details
We consider the projects referenced on the wiki main page:
| Project name | Scope |
|---|---|
| Functest | Functest will provide comprehensive testing methodology, test suites and test cases to test and verify OPNFV Platform functionality that covers the VIM and NFVI components. |
| Pharos* | The Pharos project deals with the creation of a federated NFV test capability that is geographically and technically diverse and hosted by different companies in the OPNFV community. This requires developing a baseline specification for an OPNFV "compliant" test environment along with tools, processes and documentation to support integration, testing and collaborative development projects with needed infrastructure and the tooling. |
| Qtip | Qtip project is a "Bottom-Up" approach in characterizing and benchmarking OPNFV Platform. Proper characterization of OPNFV platforms is a critical component to understand how well OPNFV (and by implication upstream components) perform in realistic deployment scenarios, to provide feedback to design teams, to provide platform level performance data to higher layer developers, users, and community at large. Qtip also aims to build necessary testing and benchmarking tools that will be needed to fulfill its goals, to automate when possible, and to provide these tools and automation software to upper layer and other testing projects in OPNFV. |
| VSPERF | develop a generic and architecture agnostic vSwitch testing framework and associated tests, that will serve as a basis for validating the suitability of different vSwitch implementations in a Telco NFV deployment environment |
| Yardstick | The goal of the Yardstick Project is to verify the infrastructure compliance when running VNF applications. |
*: Pharos is not a test project as it does neither define test cases not test tools. Pharos is in charge of the testbeds that can be used by test projects.
Definitions
As OPNFV is dealing with lots of testing projects, it could make sense to share some definitions..
Testcase definition
TBD
Fault management
TBD severity …
Troubleshooting
TBD who is doing what / …
KPI
TBD KPI of a Telco Cloud…
Test coverage
The test coverage is not an easy task. Test projects must focus on NFVI, however, in order to test efficiently NFVI, it may be usefull to test VNFs (which are out of OPNFV scope). We may suggest several views:
- per domain
- per component
- per ETSI domain
See dedicated testing coverage page
Test projects versus Releases
| Project | R1 Arno | R2 Bramaputhra |
|---|---|---|
| Functest | 4 suites automated in CI | - Automation of vIMS reference testcases in a reference OPNFV global testcase doc - Work on testcase auto declaration |
| VPERF | first demo done during hackfest in Prague | More tests reference testcases in a reference OPNFV global testcase doc |
| Qtip | Benchmark suite to provide quantitatve performane metrics for OPNFV platform's compute, networking and storage components | |
| Yardstick | - Methodology for Infrastructure verification from VNF perspective, aligned with ETSI-NFV TST001 - Generic test cases for Compute, Storage and Networking areas - OPNFV Infrastructure verification from VTC and SFC perspective - Mobility Traffic Profile as Test Stimuli |
|
| … |
Traffic profiles for testing
Telco representative traffic profiles are necessary for the tests. See Traffic profiles page
Test Dashboard
A test dashboard shall give a good overview of the different tests on the different PODs. See dedicated page
References
