Wiki

This page is read only. You can view the source, but not change it. Ask your administrator if you think this is wrong.
====== Testing Dashboard ====== 


==== Admin ====

It was decided to adopt the Mongo DB/API/Json approach [[http://ircbot.wl.linuxfoundation.org/meetings/opnfv-testperf/2015/opnfv-testperf.2015-09-03-16.03.html|dashboard meeting minutes]]

{{:opnfv_dashboard_demo_support.pptx|
Dashboard 1st Demo suppport}}

[[https://etherpad.opnfv.org/p/functiontest | etherpad]]


==== Introduction ====

All the test projects generate results in different format. The goal of a testing dashboard is to provide a consistent view of the different tests from the different projects.


==== Overview ====


We may describe the overall system dealing with test and testbeds as follow:
{{ :dashboards2.jpg?600 |}}

We may distinguish
  * the data collection: Test projects push their data using a REST API
  * the production of the test dashboard: the LF WP develop a portal that will call the API to retrieve the data and build the dashboard

== Test collection ==

The test collection API is [[https://gerrit.opnfv.org/gerrit/#/c/1134/|under review]], it consists in a simple Rest API associated a Mongo DB.

Most of the tests will be run from the CI but test collection shall not affect CI in any way that is one of the reason to externalize data collection and not using a jenkins Plugin.


== Dashboard ==

A module shall extract the raw results pushed by the test projects to create a testing dashboard. This dashboard shall be able to show:
  * The number of testcases/test projects/ people involved/organization involved
  * All the tests executed on a a given POD
  * All the tests from a test project
  * All the testcases executed on all the POD
  * All the testcases in critical state for version X installed with installer I on testbed T
  * ....

It shall be possible to filter per:
  * POD
  * Test Project
  * Testcase
  * OPNFV version
  * Installer
  * ...

For each test case we shall be able to see:
  * Error rate versus time
  * Duration versus time
  * Packet loss
  * Delay
  * ....

And also the severity (to be commonly agreed) of the errors...

^description ^comment ^ 
| critical |  not acceptable for release |
| major | failure rate? , failed on N% of the installers?|
| minor | |

The expected load will not be a real constraint. 

In fact the tests will be run less that 5 times a day and we do not need real time (upgrade evey hours would be enough).  


== Role of the test projects ==

Each test project shall:
  * declare itself using test collection API (see [[collection_of_test_results|Description]])
  * declare the testcases
  * push the results into the test DB
  * Create a <my_project>2Dashboard.py file into [[https://git.opnfv.org/cgit/releng/tree/utils/test/result_collection_api/dashboard|releng]], This file indicate how to produce "ready to dashboard" data set, that are exposed afterwards through the API

^ Project      ^ testcases  ^ Dashboard ready description                                                                                                  ^
| Functest     | vPing      | graph1: duration = f(time) \\ graph 2 bar graph (tests run / tests OK)                                                       |
| :::          | Tempest    | graph1: duration = f(time) \\ graph 2 (nb tests run, nb tests failed)=f(time) \\ graph 3: bar graph nb tests run, nb failed  |
| :::          | odl        |                                                                                                                              |
| :::          | rally-*    |                                                                                                                              |
| yardstick    | Ping       | graph1: duration = f(time) \\ graph 2 bar graph (tests run / tests OK)                                                       |
| :::          |            |                                                                                                                              |
| VSPERF       |            |                                                                                                                              |
| QTip         |            |                                                                                                                              |




==== First studies for dashboarding ====

  * home made solution
  * [[http://bitergia.com/| bitergia]], as used for code in OPNFV [[http://projects.bitergia.com/opnfv/browser/]]
  * [[https://wiki.jenkins-ci.org/display/JENKINS/Dashboard+View | Jenkins Dashboard view plugin]]
  * [[https://wiki.jenkins-ci.org/display/JENKINS/Logstash+Plugin | Logstash plugin for Jenkins]] that could be used with [[https://www.youtube.com/watch?v=Epe63Uu-IO0 | ELK]] solution
  * [[https://wiki.jenkins-ci.org/display/JENKINS/Remote+access+API|remote Jenkins API]] + drupal + js lib ([[https://wiki.jenkins-ci.org/display/JENKINS/Remote+access+API|jqplot]], ..)
  * [[http://www.predictiveanalyticstoday.com/open-source-dashboard-software/| 15 opensource solution for dashboarding]]
  * [[http://butleranalytics.com/5-free-open-source-bi/|BI solution]]

==== Visualization examples ====

  * Example of home made solution on functest/vPing: [[vPing4Dashboard example]]
  * Example view of using the ELK stack (elasticsearch, logstash, kibana):
    * [[opnfv_test_dashboard/opnfv_kibana_dashboards]]
    * Visualize Functest (vPing/Tempest) results: [[opnfv_test_dashboard/functest_elk_example|ELK example for FuncTest]]

===== Test results server =====

Test results server:

  * Single server which hosts and visualizes all OPNFV test results 
    * Testresults.opnfv.org (also testdashboard.opnfv.org) - 130.211.154.108
  * The server will host
    * Test results portal / landing page (nginx)
    * Test results data base (MongoDB)
    * Yardstick specific data base (InfluxDB)
    * ELK stack - with Kibana to serve as Test Dashboard
    * Grafana (for Yardstick results visualization)
    * (future) - use Kafka as message broker and hook up data-bases (ES, Mongo, ..) to Kafka

Port assignment (for FW): 
  * 80 - nginx - landingpage

Port assignment (local)
  * 5000 - logstash
  * 5601 - Kibana
  * 8083, 8086 - InfluxDB
  * 8082, tornado
  * 3000 - Grafana
  * 9200-9300 - Elasticsearch APIs
Wiki

User Tools

Site Tools

Page Tools
