User Tools

Site Tools


pharos_testing

Differences

This shows you the differences between two versions of the page.

Link to this comparison view

Both sides previous revision Previous revision
Next revision
Previous revision
Last revision Both sides next revision
pharos_testing [2015/03/19 08:05]
Morgan Richomme [Table]
pharos_testing [2015/06/23 15:31]
Morgan Richomme
Line 1: Line 1:
-====== Base principles of the testing ======+Pharos is dedicated to Hardware recommendation and testbed definition
  
-Key objectives:​ +functional ​tests can be found on functest project page: [[opnfv_functional_testing|Project:​ OPNFV - Base system functionality testing]]
-  * Define tooling for tests +
-  * Define test suites (SLA) +
-  * Installation and configuration of the tools +
-  * Automate test with CI +
  
-Procedure shall be independant from the installer.  +Test strategy ​(draftfrom Arno to B-Release{{:test_strategy.pptx|}}
-The procedure shall be performed on any of the installer. +
- +
-===== Test tools  ===== +
- +
-  * [[https://​wiki.openstack.org/​wiki/​Rally|Rally]] ​(benchmark, Tempest) +
-  * [[http://​robotframework.org/​|Robot framework]] +
-  * [[http://​sipp.sourceforge.net/​|Sipp]] (for SIP related test e.g vIMS) +
- +
-TODO: shall we be more prescritive on the toloing environment (creation of the VM, installation of the tools)? +
- +
-  +
-===== Test scenarios ​ ===== +
- +
-=== Overview === +
- +
-for release 1 we target the automation of the following tests +
-  * Rally scenarios including tempest +
-  * vPing (boot 2 VM, VM1 ping VM2, delete VM) +
-  * vIMS (Sipp VM trigering basic calls (REGISTER, INVITE) to clearwater compact vIMS VM) +
-  * ODL scenario? +
- +
-=== Rally ===  +
- +
-The default scenario are: +
-  * authenticate +
-  * nova +
-  * cinder ​     +
-  * glance ​  +
-  * keystone ​  +
-  * neutron ​  +
-  * quotas ​  +
-  * requests ​    +
-  * tempest-do-not-run-against-production ​  +
-  * heat      +
-  * mistral ​  +
-  * sahara ​                                 +
-  * vm +
-  * ceilometer +
-  * designate +
-  * dummy    +
-  * zaqar +
- +
-the first ones (authenticate,​ nova, cinder, glance, keystone, neutron, quotas, requests, tempest-do-not-run-against-production) can be re-used. +
-However scenario shall be tuned especially for the bench +
-  * which image (size, OS,..) TODOcheck for reco in Spirent ETSI NFV doc on test +
-  * which SLA (booting time, error rate,..) +
- +
-Noteduring first manual launched on alpha Orange platform installed with opensteak installer, there were lots of errors (196) running Tempest scenario and some in Rally scenario (results to be analyzed)  +
- +
-Studies on the testcase shall be done +
- +
-=== Open question === +
-  * Shall we create new scenario (to remove tests or add new ones)? +
-  * not sure to be able to play the default scenario (ping VM) => which tool? +
- +
- +
- +
-===== Test automation ​ ===== +
- +
-To be discussed with CI team asap  +
- +
- +
-===== status regarding installers ===== +
- +
- +
-^                ^ Jira ref                                  ^ Documentation ​                                                                                        ^ Manual test  ^ Result analysis ​ ^ Automated test  ^ BGS link                                                           ^ Comments ​ ^ +
-| Rally Bench    | https://​jira.opnfv.org/​browse/​FUNCTEST-1 ​ | installation procedure described https://​github.com/​Orange-OpenSource/​opnfv/​blob/​master/​docs/​TEST.md ​ | OK           ​| ​                 |                 | tested on Orange alpha testbed, Rally natively integrated in Fuel  |           | +
-| Rally Tempest ​ | https://​jira.opnfv.org/​browse/​FUNCTEST-2 ​ |                                                                                                       ​| ​             |                  |                 ​| ​                                                                   |           | +
-| vPing          | https://​jira.opnfv.org/​browse/​FUNCTEST-3 ​ |                                                                                                       ​| ​             |                  |                 ​| ​                                                                   |           | +
-| vIMS           | https://​jira.opnfv.org/​browse/​FUNCTEST-4 ​ | based on clearwater solution ​                                                                         |              |                  |                 ​| ​                                                                   |           | +
-| ODL            | https://​jira.opnfv.org/​browse/​FUNCTEST-5 ​ |                                                                                                       ​| ​             |                  |                 ​| ​                                                                   |           | +
- +
- +
- +
- +
- +
-  +
-  +
- +
pharos_testing.txt · Last modified: 2015/11/05 22:16 by Trevor Cooper