Key objectives:

• Define tooling for tests
• Define test suites (SLA)
• Installation and configuration of the tools
• Automate test with CI

Procedure shall be independant from the installer. The procedure shall be performed on any of the installer.

• Rally (benchmark, Tempest)
• Robot framework
• Sipp (for SIP related test e.g vIMS)

The default scenario are:

• authenticate
• nova
• cinder
• glance
• keystone
• neutron
• quotas
• requests
• tempest-do-not-run-against-production
• heat
• mistral
• sahara
• vm
• ceilometer
• designate
• dummy
• zaqar

the first ones (authenticate, nova, cinder, heat, glance, keystone, neutron, quotas, requests, tempest-do-not-run-against-production) can be re-used. However scenario shall be tuned especially for the bench

• which image (size, OS,..)
• which SLA (booting time, error rate,..)

Note: during first manual launched on alpha Orange platform installed with opensteak installer, there were lots of errors (196) running Tempest scenario and some in Rally scenario (results to be analyzed)

Studies on the testcase shall be done

• Shall we create new scenario (to remove tests or add new ones)?
• not sure to be able to play the default scenario (ping VM) ⇒ which tool?

To be discussed with CI team asap