User Tools

Site Tools


functest_release_2

This is an old revision of the document!


Brahmaputra Summary

Please note that you may have a dynamic view per installer, pod or scenario

New Functest dashboard look.

this table reflects the status but it is not dynamically up to date as CI never stops..

We consider hereafter only baremetal scenario.

Icon description:

: 4+ consecutive runs passing the success criteria
: <4 successful consecutive runs but passing the success criteria
: close to pass the success criteria
: 0% success, not passing

Fuel

POD: ericsson-pod2
source: https://build.opnfv.org/ci/view/functest/job/functest-fuel-ericsson-pod2-daily-brahmaputra/

Scenario ready for Release:

  • nosdn
  • old_l2
  • onos
  • nosdn-ovs

odl_l3 blocked by vPing

bgpvpn not mature

Scenario VIM Controllers Features Comments
vPing SSH vPing userdata Tempest [1] Rally ODL ONOS OVNO PROMISE vIMS SDNVPN
nosdn-nofeature N.R N.R N.R N.R criteria reached on the 18/2
Tempest rate 89% but some errors can be explained by config so can be removed from the equation leading to > 90%
vIMS suite #37, the previous and following ones were OK
odl_l3-ha [3] N.R N.R N.R Tempest success rate 83%
odl_l2-ha N.R N.R N.R Tempest success rate 83% but errors mainly in test_server_action and orchestration.stacks. Some run exclude these tests and lead to > 90%, when included we have reproducible results
odl_l2-bgpvpn-noha N.R N.R
onos-ha [2] N.R N.R N.R N.R N.R Tempest success rate 85% but test cases related to nova metadata shall be excluded leading to 90%
nosdn-ovs-ha N.R N.R N.R N.R Tempest success rate 87%
nosdn-kvm-ha
nosdn-ovs_kvm-ha

[1]: Detected some errors (~7-8) possibly due to insufficient IPs in the public network ip range. Investigation in progress. Possible fix by Jonas: https://gerrit.opnfv.org/gerrit/#/c/10159/

[2]: It failed 2 times in the last 5 runs.. waiting for more runs to see if it is impracticable. Ashley proposed a patch to fix this and yardstick problems with SSH in this scenario

[3]: last 2 runs it has failed



Apex

Scenario ready for Release:

  • old_l2

Doctor feature ready for the release

we may assume that nosdn is working but never run in CI

odl_l3 and onos blocked by vPing and Tempest success rate

POD: opnfv-jump-1
source: https://build.opnfv.org/ci/view/functest/job/functest-apex-opnfv-jump-1-daily-brahmaputra/

Scenario VIM Controllers Features Comments
vPing SSH vPing userdata Tempest Rally ODL ONOS OVNO PROMISE vIMS Doctor SDNVPN
odl_l3-ha N.R N.R N.R. N.R Tempest rate not stable (from 70 to 90%), number of tests run varies a lot from one run to another
3 ODL test cases partly failed
odl_l2-ha N.R N.R N.R. N.R. Tempest rate not stable (from 60 to 90%), number of tests run varies a lot from one run to another
3 ODL test cases partly failed
onos-ha N.R [1] N.R N.R N.R. N.R. N.R. Number of tempest test run varies but success rate always above 90%
ONOS suite run #54 failed due to connection to git (was OK in the 4 previous run)
odl_l2-sfc
ocl-nofeature-ha

[1] Nova metadata service is not working properly with ONOS yet. Maybe in SR1.



Compass

POD: huawei-us-deploy-bare-1
source: https://build.opnfv.org/ci/view/functest/job/functest-compass-huawei-us-deploy-bare-1-daily-brahmaputra/

Scenarios ready for Release:

  • nosdn
  • odl_l2
  • onos

Ocl planned for SR1

Scenario VIM Controllers Features Comments
vPing SSH vPing userdata Tempest Rally ODL ONOS OVNO PROMISE vIMS SDNVPN
nosdn-nofeature-ha N.R N.R N.R N.R N.R success criteria reached 8/2
odl_l2-ha N.R N.R N.R N.R success criteria reached 9/2
onos-ha N.R [1] N.R N.R. N.R N.R N.R success criteria reached 4/2
ocl-nofeature-ha N.R N.R N.R N.R planned for SR1, first run on 19/2

[1]: There are 11 failed tests out of 210 which (we think) are related to nova metadata service not supported by onos yet. Some tests need to pass the ssh-key to the instances using that service. However, it passes the success criteria of >=90%



Joid

POD: orange-pod2
source: https://build.opnfv.org/ci/view/functest/job/functest-joid-orange-pod2-daily-master/

Scenarios ready for Release:

  • odl_l2

nosdn scenario shall be OK but not run 4 times, will be added next time

vIMS use case fails but it is linked to configuration, solutions are discussed on joid etherpad.

onos scenario (run only 2 times on Orange POD 2) fails due to vPing/ssh

Scenario VIM Controllers Features Comments
vPing SSH vPing userdata Tempest Rally ODL ONOS OVNO PROMISE vIMS SDNVPN
nosdn-nofeature [1] N.R N.R N.R N.R criteria reached on the 21 ( vIMS excluded). Tempest rate 88% but some errors can be explained by config so can be removed from the equation leading to 93%
odl_l2-ha N.R N.R N.R Criteria reached on the 21 (vIMS excluded). Tempest rate = 98%
onos-ha N.R. N.R N.R. N.R. N.R
ocl-nofeature-ha


IMPORTANT: intel-pod5 and intel-pod6 won't count for the results report until we fix the possible problems why we get those big timeouts.

POD: intel-pod5
source: https://build.opnfv.org/ci/view/joid/job/functest-joid-intel-pod5-daily-brahmaputra/

Scenario VIM Controllers Features Iteration Comments
vPing SSH vPing userdata Tempest Rally [1] ODL ONOS OVNO PROMISE vIMS SDNVPN
nosdn-nofeature N.R N.R N.R N.R
odl_l2-ha N.R N.R N.R
onos-ha N.R N.R N.R N.R
ocl-nofeature-ha

[1] Rally takes 4 hr to execute. Something is wrong here…


POD: intel-pod6
source: https://build.opnfv.org/ci/view/functest/job/functest-joid-intel-pod6-daily-master/

Scenario VIM Controllers Features Iteration Comments
vPing SSH vPing userdata Tempest Rally ODL ONOS OVNO PROMISE vIMS SDNVPN
nosdn-nofeature [1] N.R N.R N.R N.R
odl_l2-ha [1] N.R N.R N.R 10 errors on promise test case out of 32
onos-ha [2] N.R. N.R N.R. N.R 10 errors on promise test case out of 32
ocl-nofeature-ha

[1] No runs found with this scenario on this pod

[2]: It seems the instances don't get an IP from the neutron dhcp agent…



Brahmaputra Troubleshooting

functest_release_2.1456473980.txt.gz · Last modified: 2016/02/26 08:06 by Morgan Richomme