Brahmaputra testing includes:
Note: This wiki is a snapshot in time. Definition of these common files is still work in progress!
Minutes of breakout sessions during Design summit
Test cases advertized in release B must be declared in the database through the Test collection API.
After D milestone, we have
A high level document will be created based on the testcases declared in the Database and will be part of the B-release deliverables of the testing group
We must collect the OPNFV config need for these tests and capture them in a config file.
The testcases declared in the DB are described at a high level however an url field is available to point to a better description either on the wiki on on project specific deliverable.
Additionnaly as the results must be stored in DB, it is recommended to add a virtual "status" testcase by project. This virtual testcase could be used to provide a global summary of each test project.
criteria for B-release:
Note discussions on the config files to reflect the tables: https://wiki.opnfv.org/genesis/configuration-files-discussion
Scenario indication shall be undestand as follow: [X].[Y], with
Which feature do they need to achieve their testing?
|Test project||Config Need||Scenario||Tested with Apex||with Compass||with Fuel||with Joid||Comments|
|armband||-||Need an ARM-based lab (ANEA)||-||-||-||-||-|
|copper||-||-||-||-||-||-||associated with functest|
|doctor||-||A.*||-||-||-||-||associated with functest|
|functest||need to detect the controller to run either odl/onos/ovno suite||*.*||-||-||Y||Y||-|
|ipvsix||Need to disable odl-l3 and enable neutron-l3-agent as part of the end-state of installation due to lack of support of IPv6 L3 routing in ODL Lithium||*.3, *.4||-||-||-||-||Use the OS+ODL environment to set up a service VM acting as an IPv6 vRouter. Because ODL Lithium doesn't support IPv6 L3 routing, the installed OS+ODL environment needs to use neutron-l3-agent instead of odl-l3|
|ovno||opencontrail||*.6||-||-||-||-||associated with functest|
|promise||-||F.*, J.*||-||-||-||-||associated with functest|
|sdnvpn||ODL needs VPN service karaf feature enabled||*.*||-||-||Y||-||associated with functest and yardstick|
|sfc||Needs ODL Beryllium with SFC, GBP, and OVSDB SB configured. Also needs Tack installed via post-install scripts.||F.9||-||-||Y||-||associated with functest and yardstick|
|vsperf||Need Ixia Loader (e.g. Intel POD3)||F.10||-||-||-||-||-|
In Arno we just had two different deployment scenarios (one based on Fuel, one based on Foreman/Quickstack as installer). With Brahmaputra we’ll target a larger set. Given that we can think of more combinations than we have resources and time, we’ll need to nail those scenarios that we care about for Brahmaputra.
What is a scenario? A scenario is a particular deployment, i.e. the deployed composition of a set of components and their associated configuration. A scenario would ultimately be described through a set of configuration files (which is why we’re defining the common config files right now - https://wiki.opnfv.org/genesis/configuration-files-discussion#common_config_files). Apart from the description of the configuration, a scenario would come with its own documentation and eventually with its own test cases (e.g. to prove the specific functionality enabled by the scenario).
Which scenarios do we take on for Brahmaputra? The “starting point” scenarios are naturally those that we inherited from Arno SR1 – which is a bare-metal or virtual deployment of (Openstack, OpenDaylight, OVS, KVM) on either Centos or Ubuntu. But we obviously desire a lot of additional ones – especially to also reflect the various feature developments for Brahmaputra.
Call to action:
Available scenarios mean that the scenarios listed on below page are currently running on CI.
Click Jenkins link to access console log of latest build. Jenkins view per PODs
NB: each scenario needs a leader..
|3||bm odl_l2-ha||IPv6||GENESIS-72 to bypass the gap in ODL Lithium that doesn't support IPv6 L3 Routing||IPv6||see Apex comment||IPv6||see A.3 note; dropped for B-release|
|4||virt odl_l2-ha||IPv6||see Apex comment on scenario 3||IPv6||see Apex comment on scenario 3||IPv6||see Apex comment on scenario 3||IPv6||see A.3 note; dropped for B-release|
|9||virt ODL Beryllium with SFC, GBP, and OVSDB SouthBound. OVS with NSH.||Apex||Fuel/SFC|
|11||bm, nosdn-dpdk_ovs * Ubuntu 14.0.4 * No ODL Backend * networking-ovs-dpdk mech driver * libvirt > 1.2.10 * Qemu 2.1+ * One NIC must be provided solely for OVS-DPDK for the private network (and I guess in case DVR is enabled two interfaces needs to be reserved – Public and private - not sure yet).||OVSNFV||FuncTest tests only (initially)|
|12||bm, Ubuntu 14, ODL with VPN Service enabled, BGPVPN with ODL backend||Fuel/SDNVPN|
|13||virt, Ubuntu 14, ODL with VPN Service enabled, BGPVPN with ODL backend||Fuel/SDNVPN|
|15||bm nosdn-ovs-ha||dropped for B-release||dropped for B-release||dropped for B-release||dropped for B-release|
|16||bm nosdn-kvm-ha||dropped for B-release||dropped for B-release||dropped for B-release||dropped for B-release|
|17||bm nosdn-ovs_kvm-ha||dropped for B-release||dropped for B-release||dropped for B-release||dropped for B-release|
|18||virt os-opencontrail-nofeature-ha||Apex||Compass||dropped for B-release||N/A||JOID||dropped for B-release|
|19||bm os-opencontrail-nofeature-ha||Apex||Compass||dropped for B-release||N/A||JOID||dropped for B-release|
|20||virt, Ubuntu 14, ONOS with OpenStack neutron-l3-agent being disabled||ONOSFW|
|21||bm, Ubuntu 14, ONOS with OpenStack neutron-l3-agent being disabled||ONOSFW|
|22||virt, CentOS 7, with ONOS neutron-l3-agent being disabled||ONOSFW|
|23||virt, Ubuntu 14, ONOS with OpenStack neutron-l3-agent being disabled||ONOSFW||ONOSFW|
possible scenarios listed but no leader yet:
It has been proposed to construct and align short scenario names in order to ease the Releng and Test Projects work to proceed with CI work and release verification.
An important aspect to point out is that even though short scenario names can give an idea regarding what a specific scenario is about, it is always important/necessary to look at and work with actual configuration files.
This naming convention is only valid for short scenario names and the details (config files, etc.) are handled within Genesis.
Both naming scenarios and Jenkins jobs have been touched upon during Daily Release Meeting on 2016-01-13 and there were no objections. See this link for minutes.
Short scenario names should follow the scheme below.
Details of the fields are
Please note that the dashes (-) are the separator between fields. Underscores (_) can be used for separating the values/words if a field contains more than one value (ie kvm_ovs).
This scenario naming scheme is currently in use/followed by compass, fuel, and joid.
Jenkins jobs use Freestyle project type.
In this new job structure, there is 1 main job per scenario-pod-branch, controlling the executions of deploy, functest, and yardstick jobs. Main jobs are used for setting triggers, triggering jobs manually or automatically.
There is 1 job for each of the deploy, functest, and yardstick for pod-branch and these jobs take scenario short names from parent job and do what they are supposed to do with given scenario on the POD.
This job structure is currently in use/followed by compass, fuel and joid.
Names of Jenkins Jobs require alignment as well in order to ease the triggering, configuration, management, and checking logs.
The jobs on Jenkins should be named according to below naming scheme.
Example Parent/Main Job names are
Example Deploy Job names are
Example Test Job names are
See all the OPNFV Platform CI jobs on this link.
This Jenkins Job naming scheme is currently in use/followed by compass, fuel and joid.
The PODs dedicated for CI production activities for Brahmaputra release are to be used for stable and latest release verification, including testing using Functest and Yardstick. Those PODs are fully allocated in daily basis for these activites.
In order to cover verification for all the testing frameworks delivering in Brahmaputra, it is necessary to use other OPNFV PODs, connected to CI, but not assigned to CI production.
The PODs listed in Pharos are eligible for the additional testing, see: https://wiki.opnfv.org/pharos_rls_b_labs#community_lab_resources_-_operational
The table below shows the additional testing and the lab used for it.
|Test Project||Scope of testing||Lab used||Installer||Comments|
|Yardstick||Test cases for HA Project||China Mobile||Fuel||On demand testing, disturbs the POD; Responsible for testing: Yardstick + HA; responsible for documentation: Yardstick, with review from HA|
|Yardstick||Yardstick generic test cases + Test cases for virtual Traffic Classifier, part of Yardstick||Ericsson POD1||Fuel||Daily Jenkins Job; Responsible for testing: Yardstick; responsible for documentation: Yardstick|
|Yardstick||Yardstick generic test cases, additional measurements||ZTE||Fuel||Daily Jenkins Job; Responsible for testing: Yardstick; Responsible for documentation: Yardstick|
|Yardstick||Test cases for Parser Project||Huawei||Compass||On demand, verify Parser Tosca2Heat; Responsible for testing: Yardstick + Parser; responsible for documentation: Yardstick, with review from Parser|
|QTIP||QTIP test suite||Dell POD1||Fuel||Daily Jenkins Job for QTIP test suite. Responsible for documentation|
|QTIP||QTIP test suite||Dell POD3||Fuel||Daily Jenkins Job for QTIP test suite.|
|VSPERF||VSPERF test suite||Intel POD3||None||Daily Jenkins Job for VSPERF test suite.|
|bottlenecks||rubbos and VSTF test suite||Huawei||Compass||Daily Jenkins Job; Responsible for testing: bottlenecks; Responsible for documentation: bottlenecks|