User Tools

Site Tools


testing

This is an old revision of the document!


Testing page

Introduction

Testing is still a key challenge for OPNFV.

All the projects must manage their test strategy (unit, functional, security, performance)

Several specific test projects have been validated by TSC and already deal with:

  • Define testcases
  • Perform tests not covered by a single project
  • Create tooling
  • Study Performance end to end

OPNFV Brahmaputra Release Test

The dependencies between the feature and test projects for Brahmaputra may be described as follow:

The Testing subgroup will receive requests until milestone C, September 25th.

The test specifications shall be ready no later than milestone D, December 1st. For test specifications received after this date, the Testing subgroup will not commit to delivering the test cases in time for the release. For tests requiring development effort from the test frameworks larger than 2 weeks, test specifications received later than November 15th will be submitted to agreement, depending on test project workload.

Please add your request for B-release test here: https://etherpad.opnfv.org/p/testing_requests_for_b_release

Based on the information collected on the pad, we create a matrix to identify the dependencies between the feature and the test projects.

OPNFV Test ecosystem

There are several projects dealing with integration and testing (see https://wiki.opnfv.org/).

Overview

A global overview can be described as follow:

Guidelines for new Test projects

Project details

We consider the projects referenced on the wiki main page:

Project name Scope R2 main goals R2 deliverables
Functest Functest will provide comprehensive testing methodology, test suites and test cases to test and verify OPNFV Platform functionality that covers the VIM and NFVI components. - Automation of vIMS
reference testcases in a reference OPNFV global testcase doc
- Work on testcase auto declaration
User guide, Installation guide + common testcase reference guide
Pharos* The Pharos project deals with the creation of a federated NFV test capability that is geographically and technically diverse and hosted by different companies in the OPNFV community. This requires developing a baseline specification for an OPNFV "compliant" test environment along with tools, processes and documentation to support integration, testing and collaborative development projects with needed infrastructure and the tooling.
PolicyTest PolicyTest adds network policy testing to the existing suite of OPNFV functional tests. PolicyTest tickets in Jira
STOREPERF
Qtip Qtip project is a "Bottom-Up" approach in characterizing and benchmarking OPNFV Platform. Proper characterization of OPNFV platforms is a critical component to understand how well OPNFV (and by implication upstream components) perform in realistic deployment scenarios, to provide feedback to design teams, to provide platform level performance data to higher layer developers, users, and community at large. Qtip also aims to build necessary testing and benchmarking tools that will be needed to fulfill its goals, to automate when possible, and to provide these tools and automation software to upper layer and other testing projects in OPNFV. Benchmark suite to provide quantitatve performane metrics for OPNFV platform's compute, networking and storage components
Transformer
VSPERF develop a generic and architecture agnostic vSwitch testing framework and associated tests, that will serve as a basis for validating the suitability of different vSwitch implementations in a Telco NFV deployment environment Support our 3 base deployment scenarios: P2P, PVP and PVVP, CI Support and integration for VSPERF, Support Vanilla OVS and OVS with DPDK. Support all RFC2544 defined tests. Support IMIX traffic testing with RFC 2544. Support IXIA, Spirent, Moongen and Xena. Participate in opnfv_test_dashboard https://jira.opnfv.org/issues/?jql=project%20%3D%20VSPERF%20AND%20labels%20%3D%20R2
Yardstick The goal of the Yardstick Project is to verify the infrastructure compliance when running VNF applications. - Methodology for Infrastructure verification from VNF perspective, aligned with ETSI-NFV TST001
- Generic test cases for Compute, Storage and Networking areas
- OPNFV Infrastructure verification from VTC and SFC perspective
- Mobility Traffic Profile as Test Stimuli

*: Pharos is not a test project as it does neither define test cases not test tools. Pharos is in charge of the testbeds that can be used by test projects.

Definitions

As OPNFV is dealing with lots of testing projects, it could make sense to share some definitions..

Testcase definition

TBD

Fault management

TBD severity …

Troubleshooting

TBD who is doing what / …

KPI

TBD KPI of a Telco Cloud…

Test coverage

The test coverage is not an easy task. Test projects must focus on NFVI, however, in order to test efficiently NFVI, it may be usefull to test VNFs (which are out of OPNFV scope). We may suggest several views:

  • per domain
  • per component
  • per ETSI domain

See dedicated testing coverage page

Traffic profiles for testing

Telco representative traffic profiles are necessary for the tests. See Traffic profiles page

Test Dashboard

A test dashboard shall give a good overview of the different tests on the different PODs. See dedicated page

References

testing.1443701314.txt.gz · Last modified: 2015/10/01 12:08 by Maryam Tahhan