User Tools

Site Tools


brahmaputra_testing_page

This is an old revision of the document!


Brahmaputra Testing

Scope

Brahmaputra testing includes:

  1. Brahmaputra scenarios testing for release verification:
    • executed on PODs dedicated for CI production;
    • master and stable branches;
    • testing projects: Functest and Yardstick.
  2. Brahmaputra additional testing:
    • executed on CI-connected PODs, which are not allocated for CI production;
    • stable branch;
    • testing projects: all with deliverables to Brahmaputra and feature projects with specific test needs (e.g. extended measurements, tests which cause disturbances on the environment).

Note: This wiki is a snapshot in time. Definition of these common files is still work in progress!

Minutes of breakout sessions during Design summit

Labs

Test Cases

Test cases advertized in release B must be declared in the database through the Test collection API.

src: Test collection API (23/11/2015)

After D milestone, we have

  • 16 projects are declared for testing
  • 88 testcases declared

A high level document will be created based on the testcases declared in the Database and will be part of the B-release deliverables of the testing group

We must collect the OPNFV config need for these tests and capture them in a config file.

The testcases declared in the DB are described at a high level however an url field is available to point to a better description either on the wiki on on project specific deliverable.

Additionnaly as the results must be stored in DB, it is recommended to add a virtual "status" testcase by project. This virtual testcase could be used to provide a global summary of each test project.

Success Criteria

criteria for B-release:

  • matrix (installer/test cases)
  • test case documented
  • test case in DB
  • run functest and yardstick - 4 successful consecutive iterations in CI

Matrixes

Note discussions on the config files to reflect the tables: https://wiki.opnfv.org/genesis/configuration-files-discussion

Test Projects

Scenario indication shall be undestand as follow: [X].[Y], with

  • X = installer (apex|compass|fuel|joid)
  • Y = scenario id as defined in the scenario section (e.g. 1 = bm + odl i.e. Arno)

Which feature do they need to achieve their testing?

Test project Config Need Scenario Tested with Apex with Compass with Fuel with Joid Comments
armband - Need an ARM-based lab (ANEA) - - - - -
copper - - - - - - associated with functest
doctor - A.* - - - - associated with functest
functest need to detect the controller to run either odl/onos/ovno suite *.* - - Y Y -
ha - - - - - - -
ipvsix Need to disable odl-l3 and enable neutron-l3-agent as part of the end-state of installation due to lack of support of IPv6 L3 routing in ODL Lithium *.3, *.4 - - - - Use the OS+ODL environment to set up a service VM acting as an IPv6 vRouter. Because ODL Lithium doesn't support IPv6 L3 routing, the installed OS+ODL environment needs to use neutron-l3-agent instead of odl-l3
kvmfornfv - - - - - - -
qtip - *.* - - - - -
ovsnfv custom OVS F.11 - - - - -
ovno opencontrail *.6 - - - - associated with functest
promise - F.*, J.* - - - - associated with functest
sdnvpn ODL needs VPN service karaf feature enabled *.* - - Y - associated with functest and yardstick
sfc Needs ODL Beryllium with SFC, GBP, and OVSDB SB configured. Also needs Tack installed via post-install scripts. F.9 - - Y - associated with functest and yardstick
vnfgraph - - - - - -
vsperf Need Ixia Loader (e.g. Intel POD3) F.10 - - - - -
yardstick - *.* - - - - -
bottlenecks - - - Y -

Test Scenarios for Brahmaputra

In Arno we just had two different deployment scenarios (one based on Fuel, one based on Foreman/Quickstack as installer). With Brahmaputra we’ll target a larger set. Given that we can think of more combinations than we have resources and time, we’ll need to nail those scenarios that we care about for Brahmaputra.

What is a scenario? A scenario is a particular deployment, i.e. the deployed composition of a set of components and their associated configuration. A scenario would ultimately be described through a set of configuration files (which is why we’re defining the common config files right now - https://wiki.opnfv.org/genesis/configuration-files-discussion#common_config_files). Apart from the description of the configuration, a scenario would come with its own documentation and eventually with its own test cases (e.g. to prove the specific functionality enabled by the scenario).

Which scenarios do we take on for Brahmaputra? The “starting point” scenarios are naturally those that we inherited from Arno SR1 – which is a bare-metal or virtual deployment of (Openstack, OpenDaylight, OVS, KVM) on either Centos or Ubuntu. But we obviously desire a lot of additional ones – especially to also reflect the various feature developments for Brahmaputra.

Call to action:

  • Definition: If you, your project or group-of-projects desire a scenario for Brahmaputra other than the ones we inherited from Arno SR1, please add your scenario to the table below. Arno SR1 scenarios are reflected as (A.1, C.1, F.1, J.1) and (A.2, C.2, F.2, J.2).
  • Ownership: When adding a scenario, please specify the owner of the scenario. It is the owner of a scenario who is responsible for creating the scenario (config files, debugging/maintenance, documentation, documentation of test results, specific tests etc.). Owners of a scenario are likely feature projects. Note that installers are typically not an owner (Arno was an exception), because installers only install features and components, but take no responsibility for the correct functioning of the features/components and the associated documentation. Also note that scenarios without owners are unlikely to happen…

Available Scenarios

Available scenarios mean that the scenarios listed on below page are currently running on CI.

Click Jenkins link to access console log of latest build. Jenkins view per PODs

Candidate Scenarios

NB: each scenario needs a leader..

ID Description Apex Compass Fuel Joid
Owner Status Owner Status Owner Status Owner Status
1 bm odl_l3-ha Apex Compass Fuel
2 virt odl_l3-ha Fuel Joid
3 bm odl_l2-ha IPv6 GENESIS-72 to bypass the gap in ODL Lithium that doesn't support IPv6 L3 Routing IPv6 see Apex comment IPv6 see Apex comment
4 virt odl_l2-ha IPv6 see Apex comment on scenario 3 IPv6 see Apex comment on scenario 3 IPv6 see Apex comment on scenario 3 IPv6 see A.3 note
5 virt onos-ha ONOSFW ONOSFW ONOSFW
9 virt ODL Beryllium with SFC, GBP, and OVSDB SouthBound. OVS with NSH. Fuel/SFC
10 bm VSPERF Fuel
11 bm, nosdn-dpdk_ovs * Ubuntu 14.0.4 * No ODL Backend * networking-ovs-dpdk mech driver * libvirt > 1.2.10 * Qemu 2.1+ * One NIC must be provided solely for OVS-DPDK for the private network (and I guess in case DVR is enabled two interfaces needs to be reserved – Public and private - not sure yet). OVSNFV FuncTest tests only (initially)
12 bm, Ubuntu 14, ODL with VPN Service enabled, BGPVPN with ODL backend Fuel/SDNVPN
13 virt, Ubuntu 14, ODL with VPN Service enabled, BGPVPN with ODL backend Fuel/SDNVPN
15 bm nosdn-ovs-ha
16 bm nosdn-kvm-ha
17 bm nosdn-ovs_kvm-ha
18 virt os-opencontrail-nofeature-ha Apex Compass N/A JOID
19 bm os-opencontrail-nofeature-ha Apex Compass N/A JOID
20 virt, Ubuntu 14, ONOS with OpenStack neutron-l3-agent being disabled ONOSFW
21 bm, Ubuntu 14, ONOS with OpenStack neutron-l3-agent being disabled ONOSFW
22 virt, CentOS 7, with ONOS neutron-l3-agent being disabled ONOSFW
23 virt, Ubuntu 14, ONOS with OpenStack neutron-l3-agent being disabled ONOSFW
24 virt, Ubuntu 14, ONOS with OpenStack neutron-l3-agent being disabled ONOSFW

possible scenarios listed but no leader yet:

  • X.6: bm, ocl-ha
  • X.9: virt, Centos/Ubuntu 14,
  • X.10: bm, Centos 7/Ubuntu 14, VSPERF

Scenario and Jenkins Job Naming Scheme

It has been proposed to construct and align short scenario names in order to ease the Releng and Test Projects work to proceed with CI work and release verification.

An important aspect to point out is that even though short scenario names can give an idea regarding what a specific scenario is about, it is always important/necessary to look at and work with actual configuration files.

This naming convention is only valid for short scenario names and the details (config files, etc.) are handled within Genesis.

Both naming scenarios and Jenkins jobs have been touched upon during Daily Release Meeting on 2016-01-13 and there were no objections. See this link for minutes.

Scenario Naming

Short scenario names should follow the scheme below.

os-[controller]-[feature]-[mode]-[option]

Details of the fields are

  • os: mandatory
    • possible value: os
    • please note that this field is needed in order to select parent jobs to list and do blocking relations between them.
  • [controller]: mandatory
    • example values: nosdn, ocl, odl, onos
  • [feature]: mandatory
    • example values: nofeature, kvm, ovs
  • [mode]: mandatory
    • possible values: ha, noha
  • [option]: optional
    • used for the scenarios those do not fit into naming scheme.
    • optional field in the short scenario name should not be included if there is no optional scenario.

Examples are

  • os-nosdn-kvm-noha
  • os-odl_l2-nofeature-ha
  • os-onos-kvm_ovs-noha

Please note that the dashes (-) are the separator between fields. Underscores (_) can be used for separating the values/words if a field contains more than one value (ie kvm_ovs).

This scenario naming scheme is currently in use/followed by compass, fuel, and joid.

Jenkins Job Structure

Jenkins jobs switched using Jenkins MultiJob plugin in order to have an easy way to see what happened with a specific execution, trigger and control related jobs together.

Please take a look at the examples from this or this link to see job structuring.

In this new job structure, there is 1 parent job per scenario-pod-branch, controlling the executions of deploy, functest, and yardstick jobs. Parent jobs are used for setting triggers, triggering jobs manually or automatically, or grouping related jobs together.

There is 1 job for each of the deploy, functest, and yardstick for pod-branch and these jobs take scenario short names from parent job and do what they are supposed to do with given scenario on the POD.

This job structure is currently in use/followed by compass, fuel and joid.

Jenkins Job Naming

Names of Jenkins Jobs require alignment as well in order to ease the triggering, configuration, management, and checking logs.

The jobs on Jenkins should be named according to below naming scheme.

  • Parent/Multi Jobs: [installer]-[scenario]-[pod]-[loop]-[branch]
    • [installer]: apex, compass, fuel, joid
    • [scenario]: only the scenario names that fit into scenario naming scheme above.
    • [pod]: all the PODs that are assigned to specific installer and currently connected to Jenkins.
    • [loop]: daily
    • [branch]: master or brahmaputra
  • Deploy Jobs: [installer]-deploy-[pod]-[loop]-[branch]
    • [installer]: apex, compass, fuel, joid
    • [pod]: all the PODs that are assigned to specific installer and currently connected to Jenkins.
    • [loop]: daily
    • [branch]: master or brahmaputra
  • Test Jobs: [testproject]-[installer]-[pod]-[loop]-[branch]
    • [testproject]: functest, yardstick, bottlenecks, qtip
    • [installer]: apex, compass, fuel, joid
    • [pod]: all the PODs that are assigned to specific installer and currently connected to Jenkins.
    • [loop]: daily or suite
    • [branch]: master or brahmaputra

Example Parent/Multi Job names are

Example Deploy Job names are

Example Test Job names are

See all the OPNFV Platform CI jobs on this link.

This Jenkins Job naming scheme is currently in use/followed by compass, fuel and joid.

Brahmaputra Testing using non-CI production PODs

The PODs dedicated for CI production activities for Brahmaputra release are to be used for stable and latest release verification, including testing using Functest and Yardstick. Those PODs are fully allocated in daily basis for these activites.

In order to cover verification for all the testing frameworks delivering in Brahmaputra, it is necessary to use other OPNFV PODs, connected to CI, but not assigned to CI production.

The PODs listed in Pharos are eligible for the additional testing, see: https://wiki.opnfv.org/pharos_rls_b_labs#community_lab_resources_-_operational

The table below shows the additional testing and the lab used for it.

Test Project Scope of testing Lab used Installer Comments
Yardstick Test cases for HA Project China Mobile Fuel On demand testing, disturbs the POD; Responsible for testing: Yardstick + HA; responsible for documentation: Yardstick, with review from HA
Yardstick Yardstick generic test cases + Test cases for virtual Traffic Classifier, part of Yardstick Ericsson POD1 Fuel Daily Jenkins Job; Responsible for testing: Yardstick; responsible for documentation: Yardstick
Yardstick Yardstick generic test cases, additional measurements ZTE Fuel Daily Jenkins Job; Responsible for testing: Yardstick; Responsible for documentation: Yardstick
Yardstick Test cases for Parser Project Huawei Compass On demand, verify Parser Tosca2Heat; Responsible for testing: Yardstick + Parser; responsible for documentation: Yardstick, with review from Parser
QTIP QTIP test suite Dell POD1 Fuel Daily Jenkins Job for QTIP test suite. Responsible for documentation
QTIP QTIP test suite Dell POD3 Fuel Daily Jenkins Job for QTIP test suite.
VSPERF VSPERF test suite Intel POD3 None Daily Jenkins Job for VSPERF test suite.
bottlenecks rubbos and VSTF test suite Huawei Compass Daily Jenkins Job; Responsible for testing: bottlenecks; Responsible for documentation: bottlenecks
brahmaputra_testing_page.1453108035.txt.gz · Last modified: 2016/01/18 09:07 by hongbo tian