opnfv_test_dashboard
Differences
This shows you the differences between two versions of the page.
| Both sides previous revision Previous revision Next revision | Previous revision | ||
|
opnfv_test_dashboard [2015/10/09 15:32] Morgan Richomme |
opnfv_test_dashboard [2016/01/19 14:33] (current) Peter Bandzi [Test results server] |
||
|---|---|---|---|
| Line 1: | Line 1: | ||
| - | ====== Note working page ...no decision yet...to be discussed during functest and test weekly meeting ====== | + | ====== Testing Dashboard ====== |
| - | ===Admin === | + | ==== Admin ==== |
| It was decided to adopt the Mongo DB/API/Json approach [[http://ircbot.wl.linuxfoundation.org/meetings/opnfv-testperf/2015/opnfv-testperf.2015-09-03-16.03.html|dashboard meeting minutes]] | It was decided to adopt the Mongo DB/API/Json approach [[http://ircbot.wl.linuxfoundation.org/meetings/opnfv-testperf/2015/opnfv-testperf.2015-09-03-16.03.html|dashboard meeting minutes]] | ||
| Line 12: | Line 12: | ||
| - | === Introduction === | + | ==== Introduction ==== |
| All the test projects generate results in different format. The goal of a testing dashboard is to provide a consistent view of the different tests from the different projects. | All the test projects generate results in different format. The goal of a testing dashboard is to provide a consistent view of the different tests from the different projects. | ||
| - | === Overview === | + | ==== Overview ==== |
| Line 24: | Line 24: | ||
| We may distinguish | We may distinguish | ||
| - | * the data collection | + | * the data collection: Test projects push their data using a REST API |
| - | * the production of the test dashboard | + | * the production of the test dashboard: the LF WP develop a portal that will call the API to retrieve the data and build the dashboard |
| == Test collection == | == Test collection == | ||
| Line 43: | Line 43: | ||
| * All the testcases in critical state for version X installed with installer I on testbed T | * All the testcases in critical state for version X installed with installer I on testbed T | ||
| * .... | * .... | ||
| - | * | ||
| It shall be possible to filter per: | It shall be possible to filter per: | ||
| Line 62: | Line 61: | ||
| And also the severity (to be commonly agreed) of the errors... | And also the severity (to be commonly agreed) of the errors... | ||
| - | ^ ^description ^comment ^ | + | ^description ^comment ^ |
| - | | critical | | testcase failed on 80% of the installers? | | + | | critical | not acceptable for release | |
| - | | major | | | | + | | major | failure rate? , failed on N% of the installers?| |
| - | | minor | | | | + | | minor | | |
| - | The expected load will not be a real constraint. In fact the tests will be run less that 5 times a day and we do not need real time (upgrade evey hours would be enough). | + | The expected load will not be a real constraint. |
| + | |||
| + | In fact the tests will be run less that 5 times a day and we do not need real time (upgrade evey hours would be enough). | ||
| Line 76: | Line 77: | ||
| * declare the testcases | * declare the testcases | ||
| * push the results into the test DB | * push the results into the test DB | ||
| - | * Describe their needs to LF web team, explain which values shall be used and which graphs shall be built based on the test_results pushed into the DB. The post-processing of the data will be realized by the LF web team. It will be devlopped in PHP and be optimized. | + | * Create a <my_project>2Dashboard.py file into [[https://git.opnfv.org/cgit/releng/tree/utils/test/result_collection_api/dashboard|releng]], This file indicate how to produce "ready to dashboard" data set, that are exposed afterwards through the API |
| + | ^ Project ^ testcases ^ Dashboard ready description ^ | ||
| + | | Functest | vPing | graph1: duration = f(time) \\ graph 2 bar graph (tests run / tests OK) | | ||
| + | | ::: | Tempest | graph1: duration = f(time) \\ graph 2 (nb tests run, nb tests failed)=f(time) \\ graph 3: bar graph nb tests run, nb failed | | ||
| + | | ::: | odl | | | ||
| + | | ::: | rally-* | | | ||
| + | | yardstick | Ping | graph1: duration = f(time) \\ graph 2 bar graph (tests run / tests OK) | | ||
| + | | ::: | | | | ||
| + | | VSPERF | | | | ||
| + | | QTip | | | | ||
| - | === Solutions for dashboarding === | ||
| - | * home made solution (see below) | + | |
| + | |||
| + | ==== First studies for dashboarding ==== | ||
| + | |||
| + | * home made solution | ||
| * [[http://bitergia.com/| bitergia]], as used for code in OPNFV [[http://projects.bitergia.com/opnfv/browser/]] | * [[http://bitergia.com/| bitergia]], as used for code in OPNFV [[http://projects.bitergia.com/opnfv/browser/]] | ||
| * [[https://wiki.jenkins-ci.org/display/JENKINS/Dashboard+View | Jenkins Dashboard view plugin]] | * [[https://wiki.jenkins-ci.org/display/JENKINS/Dashboard+View | Jenkins Dashboard view plugin]] | ||
| Line 89: | Line 102: | ||
| * [[http://butleranalytics.com/5-free-open-source-bi/|BI solution]] | * [[http://butleranalytics.com/5-free-open-source-bi/|BI solution]] | ||
| - | === Example of home made solution on functest/vPing === | + | ==== Visualization examples ==== |
| + | |||
| + | * Example of home made solution on functest/vPing: [[vPing4Dashboard example]] | ||
| + | * Example view of using the ELK stack (elasticsearch, logstash, kibana): | ||
| + | * [[opnfv_test_dashboard/opnfv_kibana_dashboards]] | ||
| + | * Visualize Functest (vPing/Tempest) results: [[opnfv_test_dashboard/functest_elk_example|ELK example for FuncTest]] | ||
| + | |||
| + | ===== Test results server ===== | ||
| + | |||
| + | Test results server: | ||
| + | |||
| + | * Single server which hosts and visualizes all OPNFV test results | ||
| + | * Testresults.opnfv.org (also testdashboard.opnfv.org) - 130.211.154.108 | ||
| + | * The server will host | ||
| + | * Test results portal / landing page (nginx) | ||
| + | * Test results data base (MongoDB) | ||
| + | * Yardstick specific data base (InfluxDB) | ||
| + | * ELK stack - with Kibana to serve as Test Dashboard | ||
| + | * Grafana (for Yardstick results visualization) | ||
| + | * (future) - use Kafka as message broker and hook up data-bases (ES, Mongo, ..) to Kafka | ||
| + | |||
| + | Port assignment (for FW): | ||
| + | * 80 - nginx - landingpage | ||
| + | |||
| + | Port assignment (local) | ||
| + | * 5000 - logstash | ||
| + | * 5601 - Kibana | ||
| + | * 8083, 8086 - InfluxDB | ||
| + | * 8082, tornado | ||
| + | * 3000 - Grafana | ||
| + | * 9200-9300 - Elasticsearch APIs | ||
| - | see [[vPing4Dashboard example]] | ||
opnfv_test_dashboard.1444404734.txt.gz · Last modified: 2015/10/09 15:32 by Morgan Richomme
Page Tools
Except where otherwise noted, content on this wiki is licensed under the following license: Creative Commons Attribution 4.0 International License
