This is an old revision of the document!
Note working page ...no decision yet...to be discussed during functest and test weekly meeting
Admin
Introduction
All the test projects generate results in different format. The goal of a testing dashboard is to provide a consistent view of the different tests from the different projects.
Overview
We may describe the overall system dealing with test and testbeds as follow:
We may distinguish
the data collection: Test projects push their data using a REST
API
the production of the test dashboard: the LF WP develop a portal that will call the
API to retrieve the data and build the dashboard
Test collection
The test collection API is under review, it consists in a simple Rest API associated a Mongo DB.
Most of the tests will be run from the CI but test collection shall not affect CI in any way that is one of the reason to externalize data collection and not using a jenkins Plugin.
Dashboard
A module shall extract the raw results pushed by the test projects to create a testing dashboard. This dashboard shall be able to show:
The number of testcases/test projects/ people involved/organization involved
All the tests executed on a a given POD
All the tests from a test project
All the testcases executed on all the POD
All the testcases in critical state for version X installed with installer I on testbed T
….
It shall be possible to filter per:
POD
Test Project
Testcase
OPNFV version
Installer
…
For each test case we shall be able to see:
Error rate versus time
Duration versus time
Packet loss
Delay
….
And also the severity (to be commonly agreed) of the errors…
description | comment |
critical | not acceptable for release |
major | failure rate? , failed on N% of the installers? |
minor | |
The expected load will not be a real constraint.
In fact the tests will be run less that 5 times a day and we do not need real time (upgrade evey hours would be enough).
Role of the test projects
Each test project shall:
declare itself using test collection
API (see
Description)
declare the testcases
push the results into the test DB
Describe their needs to LF web team, explain which values shall be used and which graphs shall be built based on the test_results pushed into the DB. The results stored may be different, we shall try to use the same structure when possible to automate the generation of the graphs. The post-processing of the data will be realized by the LF web team. It will be devlopped in PHP and optimized (caching, ..).
First studies for dashboarding
Example of home made solution on functest/vPing