User Tools

Site Tools


pharos_testing

Differences

This shows you the differences between two versions of the page.

Link to this comparison view

Both sides previous revision Previous revision
Next revision
Previous revision
Last revision Both sides next revision
pharos_testing [2015/03/20 11:01]
Morgan Richomme
pharos_testing [2015/06/23 15:31]
Morgan Richomme
Line 1: Line 1:
-====== Base principles of the testing ======+Pharos is dedicated to Hardware recommendation and testbed definition
  
-Key objectives:​ +functional ​tests can be found on functest project page: [[opnfv_functional_testing|Project:​ OPNFV - Base system functionality testing]]
-  * Define tooling for tests +
-  * Define test suites (SLA) +
-  * Installation and configuration of the tools +
-  * Automate test with CI +
  
-Pad dedicated to func test: https://​etherpad.opnfv.org/​p/​functiontestrelease1activities +Test strategy ​(draftfrom Arno to B-Release{{:test_strategy.pptx|}}
- +
-IRC chan: #​opnfv-testperf +
- +
-Procedure shall be independant from the installer.  +
-The procedure shall be performed on any of the installer. +
- +
-===== Test tools  ===== +
- +
-  * [[https://​wiki.openstack.org/​wiki/​Rally|Rally]] ​(benchmark, Tempest) +
-  * [[http://​robotframework.org/​|Robot framework]] +
-  * [[http://​sipp.sourceforge.net/​|Sipp]] (for SIP related test e.g vIMS) +
- +
-TODO: shall we be more prescritive on the toloing environment (creation of the VM, installation of the tools)? +
- +
-  +
-===== Test scenarios ​ ===== +
- +
-=== Overview === +
- +
-for release 1 we target the automation of the following tests +
-  * Rally scenarios including tempest +
-  * vPing (boot 2 VM, VM1 ping VM2, delete VM) +
-  * vIMS (Sipp VM trigering basic calls (REGISTER, INVITE) to clearwater compact vIMS VM) +
-  * ODL scenario? +
- +
-List of testcases can be found [[https://​wiki.opnfv.org/​doku.php?​id=opnfv_functional_testing/​functest_test_cases|here]] +
- +
-=== Rally ===  +
- +
-The default scenario are: +
-  * authenticate +
-  * nova +
-  * cinder ​     +
-  * glance ​  +
-  * keystone ​  +
-  * neutron ​  +
-  * quotas ​  +
-  * requests ​    +
-  * tempest-do-not-run-against-production ​  +
-  * heat      +
-  * mistral ​  +
-  * sahara ​                                 +
-  * vm +
-  * ceilometer +
-  * designate +
-  * dummy    +
-  * zaqar +
- +
-the first ones (authenticate,​ nova, cinder, glance, keystone, neutron, quotas, requests, tempest-do-not-run-against-production) can be re-used. +
-However scenario shall be tuned especially for the bench +
-  * which image (size, OS,..) TODOcheck for reco in Spirent ETSI NFV doc on test +
-  * which SLA (booting time, error rate,..) +
- +
-Noteduring first manual launched on alpha Orange platform installed with opensteak installer, there were lots of errors (196) running Tempest scenario and some in Rally scenario (results to be analyzed)  +
- +
-Studies on the testcase shall be done +
- +
-=== Open question === +
-  * Shall we create new scenario (to remove tests or add new ones)? +
-  * not sure to be able to play the default scenario (ping VM) => which tool? +
- +
- +
- +
-===== Test automation ​ ===== +
- +
-To be discussed with CI team asap  +
- +
- +
-===== status regarding installers ===== +
- +
- +
-^                ^ Jira ref                                  ^ Documentation ​                                                                                        ^ Manual test  ^ Result analysis ​ ^ Automated test  ^ BGS link                                                         ^ Comments ​ ^ +
-| Rally Bench    | https://​jira.opnfv.org/​browse/​FUNCTEST-1 ​ | installation procedure described https://​github.com/​Orange-OpenSource/​opnfv/​blob/​master/​docs/​TEST.md ​ | OK           ​| ​                 |                 | Tested with opensteack #3; Rally natively integrated in Fuell #1  |           | +
-| Rally Tempest ​ | https://​jira.opnfv.org/​browse/​FUNCTEST-2 ​ |                                                                                                       | OK           ​| ​                 |                 | Tested with opensteack #3; Rally natively integrated in Fuell #1  |           | +
-| vPing          | https://​jira.opnfv.org/​browse/​FUNCTEST-3 ​ |                                                                                                       ​| ​             |                  |                 ​| ​                                                                 |           | +
-| vIMS           | https://​jira.opnfv.org/​browse/​FUNCTEST-4 ​ | Based on Clearwater solution ​                                                                         |              |                  |                 ​| ​                                                                 |           | +
-| ODL            | https://​jira.opnfv.org/​browse/​FUNCTEST-5 ​ |                                                                                                       ​| ​             |                  |                 ​| ​                                                                 |           | +
- +
- +
- +
- +
- +
- +
-  +
-  +
- +
pharos_testing.txt · Last modified: 2015/11/05 22:16 by Trevor Cooper