Brahmaputra testing includes:

1. Brahmaputra scenarios testing for release verification:
   • executed on PODs dedicated for CI production;
   • master and stable branches;
   • testing projects: Functest and Yardstick.
2. Brahmaputra additional testing:
   • executed on CI-connected PODs, which are not allocated for CI production;
   • stable branch;
   • testing projects: all with deliverables to Brahmaputra and feature projects with specific test needs (e.g. extended measurements, tests which cause disturbances on the environment).

Note: This wiki is a snapshot in time. Definition of these common files is still work in progress!

Minutes of breakout sessions during Design summit

• Nov 9th: http://ircbot.wl.linuxfoundation.org/meetings/opnfv-yardstick/2015/opnfv-yardstick.2015-11-09-23.19.html
• Nov 10th: http://ircbot.wl.linuxfoundation.org/meetings/opnfv-yardstick/2015/opnfv-yardstick.2015-11-11-00.38.html

see https://wiki.opnfv.org/pharos_rls_b_labs#status_of_production_ci_resources_for_release_b

Test cases advertized in release B must be declared in the database through the Test collection API.

src: Test collection API (23/11/2015)

After D milestone, we have

• 16 projects are declared for testing
• 88 testcases declared

A high level document will be created based on the testcases declared in the Database and will be part of the B-release deliverables of the testing group

We must collect the OPNFV config need for these tests and capture them in a config file.

The testcases declared in the DB are described at a high level however an url field is available to point to a better description either on the wiki on on project specific deliverable.

Additionnaly as the results must be stored in DB, it is recommended to add a virtual "status" testcase by project. This virtual testcase could be used to provide a global summary of each test project.

criteria for B-release:

• matrix (installer/test cases)
• test case documented
• test case in DB
• run functest and yardstick - 4 successful consecutive iterations in CI

Note discussions on the config files to reflect the tables: https://wiki.opnfv.org/genesis/configuration-files-discussion

Scenario indication shall be undestand as follow: [X].[Y], with

• X = installer (apex|compass|fuel|joid)
• Y = scenario id as defined in the scenario section (e.g. 1 = bm + odl i.e. Arno)

Which feature do they need to achieve their testing?

In Arno we just had two different deployment scenarios (one based on Fuel, one based on Foreman/Quickstack as installer). With Brahmaputra we’ll target a larger set. Given that we can think of more combinations than we have resources and time, we’ll need to nail those scenarios that we care about for Brahmaputra.

What is a scenario? A scenario is a particular deployment, i.e. the deployed composition of a set of components and their associated configuration. A scenario would ultimately be described through a set of configuration files (which is why we’re defining the common config files right now - https://wiki.opnfv.org/genesis/configuration-files-discussion#common_config_files). Apart from the description of the configuration, a scenario would come with its own documentation and eventually with its own test cases (e.g. to prove the specific functionality enabled by the scenario).

Which scenarios do we take on for Brahmaputra? The “starting point” scenarios are naturally those that we inherited from Arno SR1 – which is a bare-metal or virtual deployment of (Openstack, OpenDaylight, OVS, KVM) on either Centos or Ubuntu. But we obviously desire a lot of additional ones – especially to also reflect the various feature developments for Brahmaputra.

Call to action:

• Definition: If you, your project or group-of-projects desire a scenario for Brahmaputra other than the ones we inherited from Arno SR1, please add your scenario to the table below. Arno SR1 scenarios are reflected as (A.1, C.1, F.1, J.1) and (A.2, C.2, F.2, J.2).
• Ownership: When adding a scenario, please specify the owner of the scenario. It is the owner of a scenario who is responsible for creating the scenario (config files, debugging/maintenance, documentation, documentation of test results, specific tests etc.). Owners of a scenario are likely feature projects. Note that installers are typically not an owner (Arno was an exception), because installers only install features and components, but take no responsibility for the correct functioning of the features/components and the associated documentation. Also note that scenarios without owners are unlikely to happen…

Available scenarios mean that the scenarios listed on below page are currently running on CI. Click Jenkins link to access console log of latest build. Jenkins view per PODs

• bm: bare metal
• virt: virtual

Note: all the Apex scenario are based on Centos 7, all the scenario on compass, fuel and joid are based on Ubuntu 14.04

Scenario indication shall be understand as follow: [X].[Y], with

• X = installer (apex|compass|fuel|joid)
• Y = scenario id as defined in the scenario section (e.g. 1 = bm + odl i.e. Arno)

NB: each scenario needs a leader..

possible scenarios listed but no leader yet:

• X.6: bm, ocl-ha
• X.9: virt, Centos/Ubuntu 14,
• X.10: bm, Centos 7/Ubuntu 14, VSPERF

The PODs dedicated for CI production activities for Brahmaputra release are to be used for stable and latest release verification, including testing using Functest and Yardstick. Those PODs are fully allocated in daily basis for these activites.

In order to cover verification for all the testing frameworks delivering in Brahmaputra, it is necessary to use other OPNFV PODs, connected to CI, but not assigned to CI production.

The PODs listed in Pharos are eligible for the additional testing, see: https://wiki.opnfv.org/pharos_rls_b_labs#community_lab_resources_-_operational

The table below shows the additional testing and the lab used for it.