User Tools

Site Tools


pharos_testing

This is an old revision of the document!


Base principles of the testing

Key objectives:

  • Define tooling for tests
  • Define test suites (SLA)
  • Installation and configuration of the tools
  • Automate test with CI

Pad dedicated to func test: https://etherpad.opnfv.org/p/functiontestrelease1activities

IRC chan: #opnfv-testperf

Procedure shall be independant from the installer. The procedure shall be performed on any of the installer.

Test tools

Intel POD2 (contact Trevor Cooper) is dedicated to functional testing.

Functional tests shall be

  • independant from the installer (Fuel, Foreman/Puppet, Juju,..)
  • automated and integrated in CI

TODO: shall we be more prescritive on the toloing environment (creation of the VM, installation of the tools)?

Test scenarios

Overview

for release 1 we target the automation of the following tests

  • Rally scenarios including tempest
  • vPing (boot 2 VM, VM1 ping VM2, delete VM)
  • vIMS (Sipp VM trigering basic calls (REGISTER, INVITE) to clearwater compact vIMS VM)
  • ODL scenario?

List of testcases can be found here

Rally

The default scenario are:

  • authenticate
  • nova
  • cinder
  • glance
  • keystone
  • neutron
  • quotas
  • requests
  • tempest-do-not-run-against-production
  • heat
  • mistral
  • sahara
  • vm
  • ceilometer
  • designate
  • dummy
  • zaqar

the first ones (authenticate, nova, cinder, glance, keystone, neutron, quotas, requests, tempest-do-not-run-against-production) can be re-used. However scenario shall be tuned especially for the bench

  • which image (size, OS,..) TODO: check for reco in Spirent ETSI NFV doc on test
  • which SLA (booting time, error rate,..)

Note: during first manual launched on alpha Orange platform installed with opensteak installer, there were lots of errors (196) running Tempest scenario and some in Rally scenario (results to be analyzed)

Studies on the testcase shall be done

Open question

  • Shall we create new scenario (to remove tests or add new ones)?
  • not sure to be able to play the default scenario (ping VM) ⇒ which tool?

Test automation

Octopus etherpad: https://etherpad.opnfv.org/p/octopus

Community platforms connected to CI

  • Ericsson
  • Intel (in progress)
  • Huawei
  • Orange (TBDone)

status regarding installers

Jira ref Documentation Manual test Result analysis Automated test BGS link Comments
Rally Bench https://jira.opnfv.org/browse/FUNCTEST-1 installation procedure described https://github.com/Orange-OpenSource/opnfv/blob/master/docs/TEST.md OK Installed on Jump host server of Intel POD 2 #1; Rally natively integrated in Fuell #2;Tested with opensteack #3; Morgan. Tempest not working on POD - same issue than on #3, patch applied but seems there is a pb ⇒ contact openstack-rally. Rally Test suite can be performed but flavours, images are missing on Openstack SUT deployed on POD 2
Tempest https://jira.opnfv.org/browse/FUNCTEST-2 OK Use of khalisi for foreman/puppet #1; Rally natively integrated in Fuell #2;Tested with opensteack #3;
vPing https://jira.opnfv.org/browse/FUNCTEST-3 Sama
vIMS https://jira.opnfv.org/browse/FUNCTEST-4 Based on Clearwater solution Martin
ODL https://jira.opnfv.org/browse/FUNCTEST-5 Peter ⇒ https://etherpad.opnfv.org/p/robotframework
pharos_testing.1428047059.txt.gz · Last modified: 2015/04/03 07:44 by Morgan Richomme