User Tools

Site Tools


pharos_testing

Differences

This shows you the differences between two versions of the page.

Link to this comparison view

Both sides previous revision Previous revision
Next revision
Previous revision
pharos_testing [2015/03/19 07:42]
Morgan Richomme [status regarding installers]
pharos_testing [2015/11/05 22:16]
Trevor Cooper
Line 1: Line 1:
-====== Base principles of the testing ======+Pharos is dedicated to test infrastructure (specification,​ process, ...)
  
-Key objectives:​ +Functional ​tests can be found on functest project page: [[opnfv_functional_testing|Project:​ OPNFV - Base system functionality testing]]
-  * Define tooling for tests +
-  * Define test suites (SLA) +
-  * Installation and configuration of the tools +
-  * Automate test with CI +
  
-Procedure shall be independant ​from the installer.  +Test strategy ​from Arno to B-Release{{:test_strategy.pptx|}}
-The procedure shall be performed on any of the installer. +
- +
-===== Test tools  ===== +
- +
-  * [[https://​wiki.openstack.org/​wiki/​Rally|Rally]] (benchmark, Tempest) +
-  * [[http://​robotframework.org/​|Robot framework]] +
-  * [[http://​sipp.sourceforge.net/​|Sipp]] (for SIP related test e.g vIMS) +
- +
-TODO: shall we be more prescritive on the toloing environment (creation of the VM, installation of the tools)? +
- +
-  +
-===== Test scenarios ​ ===== +
- +
-=== Overview === +
- +
-for release 1 we target the automation of the following tests +
-  * Rally scenarios including tempest +
-  * vPing (boot 2 VM, VM1 ping VM2, delete VM) +
-  * vIMS (Sipp VM trigering basic calls (REGISTER, INVITE) ​to clearwater compact vIMS VM) +
-  * ODL scenario? +
- +
-=== Rally ===  +
- +
-The default scenario are: +
-  * authenticate +
-  * nova +
-  * cinder ​     +
-  * glance ​  +
-  * keystone ​  +
-  * neutron ​  +
-  * quotas ​  +
-  * requests ​    +
-  * tempest-do-not-run-against-production ​  +
-  * heat      +
-  * mistral ​  +
-  * sahara ​                                 +
-  * vm +
-  * ceilometer +
-  * designate +
-  * dummy    +
-  * zaqar +
- +
-the first ones (authenticate,​ nova, cinder, glance, keystone, neutron, quotas, requests, tempest-do-not-run-against-production) can be re-used. +
-However scenario shall be tuned especially for the bench +
-  * which image (size, OS,..) TODOcheck for reco in Spirent ETSI NFV doc on test +
-  * which SLA (booting time, error rate,..) +
- +
-Noteduring first manual launched on alpha Orange platform installed with opensteak installer, there were lots of errors (196) running Tempest scenario and some in Rally scenario (results to be analyzed)  +
- +
-Studies on the testcase shall be done +
- +
-=== Open question === +
-  * Shall we create new scenario (to remove tests or add new ones)? +
-  * not sure to be able to play the default scenario (ping VM) => which tool? +
- +
- +
- +
-===== Test automation ​ ===== +
- +
-To be discussed with CI team asap  +
- +
- +
-===== status regarding installers ===== +
- +
- +
-^ "Func test" ^ "​Jira" ​ ^ "​Description/​Documentation"​ ^ "​Manual test" ^ "​Result analysis"​ ^ "​Automated test" ^ "BGS link" ^ "​Comments"​ ^  +
-|Rally Bench| |installation procedure described [[https://​github.com/​Orange-OpenSource/​opnfv/​blob/​master/​docs/​TEST.md]]|OK| | |tested on Orange alpha testbed, Rally natively integrated in Fuel| | +
-|Rally Tempest| | | | | | | | +
-|vPing| | | | | | | | +
-|vIMS| |based on clearwater solution| | | | | | +
-|ODL| | | | | | | | +
- +
- +
- +
- +
-  +
-  +
- +
pharos_testing.txt · Last modified: 2015/11/05 22:16 by Trevor Cooper