This shows you the differences between two versions of the page.
Both sides previous revision Previous revision Next revision | Previous revision Next revision Both sides next revision | ||
pharos_testing [2015/03/19 08:05] Morgan Richomme [Table] |
pharos_testing [2015/04/08 12:29] Morgan Richomme |
||
---|---|---|---|
Line 1: | Line 1: | ||
- | ====== Base principles of the testing ====== | + | Pharos is dedicated to Hardware recommendation and testbed definition |
- | Key objectives: | + | functional tests can be found on functest project page: [[opnfv_functional_testing|Project: OPNFV - Base system functionality testing]] |
- | * Define tooling for tests | + | |
- | * Define test suites (SLA) | + | |
- | * Installation and configuration of the tools | + | |
- | * Automate test with CI | + | |
- | + | ||
- | Procedure shall be independant from the installer. | + | |
- | The procedure shall be performed on any of the installer. | + | |
- | + | ||
- | ===== Test tools ===== | + | |
- | + | ||
- | * [[https://wiki.openstack.org/wiki/Rally|Rally]] (benchmark, Tempest) | + | |
- | * [[http://robotframework.org/|Robot framework]] | + | |
- | * [[http://sipp.sourceforge.net/|Sipp]] (for SIP related test e.g vIMS) | + | |
- | + | ||
- | TODO: shall we be more prescritive on the toloing environment (creation of the VM, installation of the tools)? | + | |
- | + | ||
- | + | ||
- | ===== Test scenarios ===== | + | |
- | + | ||
- | === Overview === | + | |
- | + | ||
- | for release 1 we target the automation of the following tests | + | |
- | * Rally scenarios including tempest | + | |
- | * vPing (boot 2 VM, VM1 ping VM2, delete VM) | + | |
- | * vIMS (Sipp VM trigering basic calls (REGISTER, INVITE) to clearwater compact vIMS VM) | + | |
- | * ODL scenario? | + | |
- | + | ||
- | === Rally === | + | |
- | + | ||
- | The default scenario are: | + | |
- | * authenticate | + | |
- | * nova | + | |
- | * cinder | + | |
- | * glance | + | |
- | * keystone | + | |
- | * neutron | + | |
- | * quotas | + | |
- | * requests | + | |
- | * tempest-do-not-run-against-production | + | |
- | * heat | + | |
- | * mistral | + | |
- | * sahara | + | |
- | * vm | + | |
- | * ceilometer | + | |
- | * designate | + | |
- | * dummy | + | |
- | * zaqar | + | |
- | + | ||
- | the first ones (authenticate, nova, cinder, glance, keystone, neutron, quotas, requests, tempest-do-not-run-against-production) can be re-used. | + | |
- | However scenario shall be tuned especially for the bench | + | |
- | * which image (size, OS,..) TODO: check for reco in Spirent ETSI NFV doc on test | + | |
- | * which SLA (booting time, error rate,..) | + | |
- | + | ||
- | Note: during first manual launched on alpha Orange platform installed with opensteak installer, there were lots of errors (196) running Tempest scenario and some in Rally scenario (results to be analyzed) | + | |
- | + | ||
- | Studies on the testcase shall be done | + | |
- | + | ||
- | === Open question === | + | |
- | * Shall we create new scenario (to remove tests or add new ones)? | + | |
- | * not sure to be able to play the default scenario (ping VM) => which tool? | + | |
- | + | ||
- | + | ||
- | + | ||
- | ===== Test automation ===== | + | |
- | + | ||
- | To be discussed with CI team asap | + | |
- | + | ||
- | + | ||
- | ===== status regarding installers ===== | + | |
- | + | ||
- | + | ||
- | ^ ^ Jira ref ^ Documentation ^ Manual test ^ Result analysis ^ Automated test ^ BGS link ^ Comments ^ | + | |
- | | Rally Bench | https://jira.opnfv.org/browse/FUNCTEST-1 | installation procedure described https://github.com/Orange-OpenSource/opnfv/blob/master/docs/TEST.md | OK | | | tested on Orange alpha testbed, Rally natively integrated in Fuel | | | + | |
- | | Rally Tempest | https://jira.opnfv.org/browse/FUNCTEST-2 | | | | | | | | + | |
- | | vPing | https://jira.opnfv.org/browse/FUNCTEST-3 | | | | | | | | + | |
- | | vIMS | https://jira.opnfv.org/browse/FUNCTEST-4 | based on clearwater solution | | | | | | | + | |
- | | ODL | https://jira.opnfv.org/browse/FUNCTEST-5 | | | | | | | | + | |
- | + | ||
- | + | ||
- | + | ||
- | + | ||
- | + | ||
- | + | ||
- | + | ||
- | + |