Differences

This shows you the differences between two versions of the page.

--- opnfv_test_dashboard [2015/09/10 16:13]
Morgan Richomme
+++ opnfv_test_dashboard [2016/01/19 14:33] (current)
Peter Bandzi [Test results server]
@@ Line 1: / Line 1: @@
-====== Note working page ...no decision yet...to be discussed during functest and test weekly meeting ======
+====== Testing Dashboard ======
-===Admin ===
+==== Admin ====
 It was decided to adopt the Mongo DB/API/Json approach [[http://ircbot.wl.linuxfoundation.org/meetings/opnfv-testperf/2015/opnfv-testperf.2015-09-03-16.03.html|dashboard meeting minutes]]
@@ Line 12: / Line 12: @@
-=== Introduction ===
+==== Introduction ====
 All the test projects generate results in different format. The goal of a testing dashboard is to provide a consistent view of the different tests from the different projects.
-=== Overview ===
+==== Overview ====
@@ Line 24: / Line 24: @@
 We may distinguish
-  * the data collection
+  * the data collection: Test projects push their data using a REST API
-  * the production of the test dashboard
+  * the production of the test dashboard: the LF WP develop a portal that will call the API to retrieve the data and build the dashboard
 == Test collection ==
-The test collection API is [[https://gerrit.opnfv.org/gerrit/#/c/1134/|under review]], it consists in a simple Rest API associated with simple json format collected in a Mongo DB.
+The test collection API is [[https://gerrit.opnfv.org/gerrit/#/c/1134/|under review]], it consists in a simple Rest API associated a Mongo DB.
 Most of the tests will be run from the CI but test collection shall not affect CI in any way that is one of the reason to externalize data collection and not using a jenkins Plugin.
@@ Line 43: / Line 43: @@
   * All the testcases in critical state for version X installed with installer I on testbed T
   * ....
-  *
 It shall be possible to filter per:
@@ Line 62: / Line 61: @@
 And also the severity (to be commonly agreed) of the errors...
-^ ^description ^comment ^
+^description ^comment ^
-| critical | | testcase failed on 80% of the installers? |
+| critical |  not acceptable for release |
-| major | | |
+| major | failure rate? , failed on N% of the installers?|
-| minor | | |
+| minor | |
+The expected load will not be a real constraint.
+In fact the tests will be run less that 5 times a day and we do not need real time (upgrade evey hours would be enough).
@@ Line 74: / Line 77: @@
   * declare the testcases
   * push the results into the test DB
-  * Process the raw result to provide a set of json pre-formated result file according to what the project wants to show. These json files will be used to generate automatically the dashboard (web page including js graph)
+  * Create a <my_project>2Dashboard.py file into [[https://git.opnfv.org/cgit/releng/tree/utils/test/result_collection_api/dashboard|releng]], This file indicate how to produce "ready to dashboard" data set, that are exposed afterwards through the API
+^ Project      ^ testcases  ^ Dashboard ready description                                                                                                  ^
+| Functest     | vPing      | graph1: duration = f(time) \\ graph 2 bar graph (tests run / tests OK)                                                       |
+| :::          | Tempest    | graph1: duration = f(time) \\ graph 2 (nb tests run, nb tests failed)=f(time) \\ graph 3: bar graph nb tests run, nb failed  |
+| :::          | odl        |                                                                                                                              |
+| :::          | rally-*    |                                                                                                                              |
+| yardstick    | Ping       | graph1: duration = f(time) \\ graph 2 bar graph (tests run / tests OK)                                                       |
+| :::          |            |                                                                                                                              |
+| VSPERF       |            |                                                                                                                              |
+| QTip         |            |                                                                                                                              |
-=== Open Questions ===
-The test projects shall agree on
-  * the data model dealing with the test dashboarding (testbed / Testcase / test project / Test results / version / installers / ..)
-  * the filter criteria (pod, installer, version, date (30 days/1 year/since beginning of the project),..
-  * the label of the graphs (if it shall be customizable/possible to add new label but all the test projects shall use the same) => shall be described in the json for dashboard
-=== Proposal for a simple home made solution ===
+==== First studies for dashboarding ====
-Based on Arno, related to the result collection API could look like:
+  * home made solution
-  * test-projects.json: description of the tests projects
+  * [[http://bitergia.com/| bitergia]], as used for code in OPNFV [[http://projects.bitergia.com/opnfv/browser/]]
-  * test-functest.json: description of functest test project
+  * [[https://wiki.jenkins-ci.org/display/JENKINS/Dashboard+View | Jenkins Dashboard view plugin]]
-  * test-functest-vping.json: raw result of 3 executions of vPing on 2 PODs (LF1 and LF2)
+  * [[https://wiki.jenkins-ci.org/display/JENKINS/Logstash+Plugin | Logstash plugin for Jenkins]] that could be used with [[https://www.youtube.com/watch?v=Epe63Uu-IO0 | ELK]] solution
+  * [[https://wiki.jenkins-ci.org/display/JENKINS/Remote+access+API|remote Jenkins API]] + drupal + js lib ([[https://wiki.jenkins-ci.org/display/JENKINS/Remote+access+API|jqplot]], ..)
+  * [[http://www.predictiveanalyticstoday.com/open-source-dashboard-software/| 15 opensource solution for dashboarding]]
+  * [[http://butleranalytics.com/5-free-open-source-bi/|BI solution]]
-The projects are described as follow in the database:
+==== Visualization examples ====
-http://213.77.62.197:80/test_projects
- {
-     "meta": {
-         "total": 1,
-         "success": true
-     },
-     "test_projects": [
-         {
-             "test_cases": 0,
-             "_id": "55e05b08514bc52914149f2d",
-             "creation_date": "2015-08-28 12:58:48.602898",
-             "name": "functest",
-             "description": "OPNFV Functional testing project"
-         }
-     ]}
-If you want to know the testcases of this project
+  * Example of home made solution on functest/vPing: [[vPing4Dashboard example]]
-http://213.77.62.197:80/test_projects/functest/cases
+  * Example view of using the ELK stack (elasticsearch, logstash, kibana):
+    * [[opnfv_test_dashboard/opnfv_kibana_dashboards]]
+    * Visualize Functest (vPing/Tempest) results: [[opnfv_test_dashboard/functest_elk_example|ELK example for FuncTest]]
- {
+===== Test results server =====
-  "meta": {
-    "total": 1,
-    "success": true
-  },
-  "test_cases": [
-    {
-      "_id": "55e05dba514bc52914149f2e",
-      "creation_date": "2015-08-28 13:10:18.478986",
-      "name": "vPing",
-      "description": "This  test consist in send exchange PING requests between two created VMs"
-    }
-  ]}
+Test results server:
-Functest used to run automatically 4 suites:
+  * Single server which hosts and visualizes all OPNFV test results
-  * vPing
+    * Testresults.opnfv.org (also testdashboard.opnfv.org) - 130.211.154.108
-  * ODL
+  * The server will host
-  * Rally
+    * Test results portal / landing page (nginx)
-  * Rally
+    * Test results data base (MongoDB)
+    * Yardstick specific data base (InfluxDB)
+    * ELK stack - with Kibana to serve as Test Dashboard
+    * Grafana (for Yardstick results visualization)
+    * (future) - use Kafka as message broker and hook up data-bases (ES, Mongo, ..) to Kafka
-The results are available on [[https://build.opnfv.org/ci/view/functest/|Jenkins]]
+Port assignment (for FW):
-Functest calls the API to store the raw results of each tests.
+  * 80 - nginx - landingpage
-  curl -X POST -H "Content-Type: application/json" -d '{"details": {"timestart": 123456, "duration": 66, "status": "OK"}, "project_name": "functest", "pod_id": 1, "case_name": "vPing"}' http://213.77.62.197:80/results
+Port assignment (local)
+  * 5000 - logstash
+  * 5601 - Kibana
+  * 8083, 8086 - InfluxDB
+  * 8082, tornado
+  * 3000 - Grafana
+  * 9200-9300 - Elasticsearch APIs
-From a Python client the API can be invoked as follow:
- def push_results_to_db(payload):
-    url = TEST_DB
-    params = {"project_name": "functest", "case_name": "vPing", "pod_id": 1,
-              "details": payload}
-    headers = {'Content-Type': 'application/json'}
-    r = requests.post(url, data=json.dumps(params), headers=headers)
-    logger.debug(r)
-with the payload var
-  {'timestart': start_time_ts, 'duration': duration,'status': test_status})
-Please note that that can put whatever you want in your payload. For vPing we put the useful information, for Rally we can directly put all the json report already built by Rally ([[http://artifacts.opnfv.org/functest/lf_pod1.localdomain/2015-06-18_06-09-20/results/rally/opnfv-authenticate.json|json result for Rally opnfv-authenticate scenario]])
-In the DB you can see the results
- http://213.77.62.197:80/results?projects=functest&case=vPing
- {
-  "meta": {
-    "total": 17
-  },
-  "test_results": [
-    {
-      "project_name": "functest",
-      "description": null,
-      "creation_date": "2015-08-28 16:43:00.965000",
-      "case_name": "vPing",
-      "details": {
-        "timestart": 123456,
-        "duration": 66,
-        "status": "OK"
-      },
-      "_id": "55e08f94514bc52b1791f949",
-      "pod_id": 1
-    },
-    ..........
-   {
-      "project_name": "functest",
-      "description": null,
-      "creation_date": "2015-09-04 07:17:51.239000",
-      "case_name": "vPing",
-      "details": {
-        "timestart": 1441350977.858555,
-        "duration": 84.5,
-        "status": "OK"
-      },
-      "_id": "55e9459f514bc52b1791f959",
-      "pod_id": 1
-    }
-  ]}
-The raw results will be stored in the Mongo DB (json files)
-A module will perform post processing (cron) to generate json file usable for dashboarding (like in bitergia). In the exemple we build json file for LF2 POD (we could do it for the last 30 days, last 365 days, since the beginning)
-several files shall be produced according to the filters
-For vPing we shall produce:
-  * result-functest-vPing-<date>-<POD>-<installer>-<version>.json
-with
-  * date = 30 (days), 365 (days), since the begining of the project
-  * POD = LF-test1, Dell-test2, Orange-test1,...
-  * installer = foreman, fuel, joid
-  * version = stable Arno, Arno SR1, Stable Brahmaputra, ...
-The result-functest-vPing.json (all installers/POD/version since first day of the project) could look like:
-   {
-    {
-        "description": "vPing results for Dashboard"
-    },
-    {
-        "info": {
-            "xlabel": "time",
-            "type": "graph",
-            "ylabel": "duration"
-        },
-        "data_set": [
-            {
-                "y": 66,
-                "x": "2015-08-28 16:43:00.965000"
-            },
-   .................
-            {
-                "y": 84.5,
-                "x": "2015-09-04 07:17:51.239000"
-            }
-        ],
-        "name": "vPing duration"
-    },
-    {
-        "info": {
-            "type": "bar"
-        },
-        "data_set": [
-            {
-                "Nb Success": 17,
-                "Nb tests": 17
-            }
-        ],
-        "name": "vPing status"
-    }}
-=== Solutions for dashboarding ===
-  * home made solution (see below)
-  * [[http://bitergia.com/| bitergia]], as used for code in OPNFV [[http://projects.bitergia.com/opnfv/browser/]]
-  * [[https://wiki.jenkins-ci.org/display/JENKINS/Dashboard+View | Jenkins Dashboard view plugin]]
-  * [[https://wiki.jenkins-ci.org/display/JENKINS/Logstash+Plugin | Logstash plugin for Jenkins]] that could be used with [[https://www.youtube.com/watch?v=Epe63Uu-IO0 | ELK]] solution
-  * [[https://wiki.jenkins-ci.org/display/JENKINS/Remote+access+API|remote Jenkins API]] + drupal + js lib ([[https://wiki.jenkins-ci.org/display/JENKINS/Remote+access+API|jqplot]], ..)
-  * [[http://www.predictiveanalyticstoday.com/open-source-dashboard-software/| 15 opensource solution for dashboarding]]
-  * [[http://butleranalytics.com/5-free-open-source-bi/|BI solution]]

Wiki

User Tools

Site Tools

Differences

Page Tools