User Tools

Site Tools


functest_release_1

This is an old revision of the document!


Functest for Release 1 (Arno)

Arno stable 4/6/2015

R1 follow up

At a glance:

POD1

Date Config Tempest vPing Rally Bench ODL
22/05 O/S HA no ODL 100 Tests / 26 Failures* OK some errors on Neutron and Nova benchs N.A
03/06 target (O/S non HA + ODL) 106 Tests / 14 Failures OK some errors on Neutron and Nova benchs NOK
04/06 target 96 /13 OK some erros in Neutron and Nova benchs NOK

POD2

Date Config Tempest vPing Rally Bench ODL
22/05 target (O/S HA + ODL) 105 Tests / 32 Failures* NOK errors in Cinder, glance, Neutron, Nova benchs N.A
28/05 target 92-104 tests with 41-34 failures * NOK either Boot Error or Timeout (both VMs properly booted but no ping seen in nova console errors in Cinder, Nova and Neutron benchs 18 tests / 12 failures
03/06 target 98 tests with 37 failures* OK errors in Cinder, Nova and Neutron benchs 18 tests / 3 failures **
04/06 target 98/37 OK erros in Cinder, Nova & Neutron benchs 18/3

*: stable - 10 "false" failures identified linked to keystone identity v3 ⇒ admin_domain_name=default must be precised in tempest.conf

* *: Due to a known ODL bug: https://bugs.opendaylight.org/show_bug.cgi?id=3052

Scope

for release 1 we target the automation of the following tests

  • bench Rally scenarios
  • tempest Rally
  • vPing (boot 2 VM, VM1 ping VM2, delete VM)
  • ODL scenario / Robot framework

SUT

At the end of a Fresh install, the status of the OPNFV solutions according to the selected installer can be summarized as follow:

Fuel Foreman
Images TestVM (Cirros) none
Networks public: net04_ext+router, private: net04 public: provider-network + router
Flavors m1.tiny, m1.small, m1.medium, m1.large, m1.xlarge, m1.micro m1.small, m1.medium, m1.large, m1.xlarge
OpenStack creds OS_USERNAME=admin OS_PASSWORD=admin, OS_TENANT_NAME=admin OS_USERNAME=admin OS_PASSWORD=octopus OS_TENANT_NAME=admin

Functest configuration script

See reference doc

Testcases follow-up

Test automation

See Octopus etherpad: https://etherpad.opnfv.org/p/octopus

Community platforms connected to CI

  • Ericsson
  • Intel
  • Huawei
  • Orange

Arno SR1 29/9/2015

For SR1, we use only one POD (LF POD2). This POD is used to test 2 version (Arno stable, Master(SR1) on the 2 installers (Fuel, Foreman)

Version CI vPing ODL Rally Tempest Comments
Arno stable foreman OK OK (~ 130s) failed OK OK (107 test, 25 failed)
Arno stable fuel not run (deploy failed) used to run with similar results than master
Arno master foreman OK OK (~130s) NOK OK OK (113 tests, 30 failed)
Arno master fuel OK OK (~80s) idem Arno master foreman OK OK (113 tests, 13 failed)

Arno SR0 vs SR1

Version CI vPing ODL Rally Tempest Comments
Fuel SR0 OK OK (~ 80s) http://tinyurl.com/oucukrq ODL container no integrated in installation OK OK (96 tests, 13 failed) http://tinyurl.com/oha7cr8
Fuel SR1 #39 OK OK (~ 22s !!!) (average on last successful tests: 43s ) 18 tests total, 15 passed, 3 failed OK OK (106 tests,16 failed)
Foreman SR0 OK OK (~100s) http://tinyurl.com/oucukrq 18 tests/3 errors. Due to a known ODL bug: http://tinyurl.com/pasevk9 OK OK (98 tests, 37 failed) http://tinyurl.com/oha7cr8
Foreman SR1 #17 OK OK (105s) (average on last 5 succesful tests: 110s) 18 tests total, 15 passed, 3 failed OK OK (107 tests, 29 failed)
functest_release_1.1443625769.txt.gz · Last modified: 2015/09/30 15:09 by Jose Lausuch