User Tools

Site Tools


functextnexttaks

This is an old revision of the document!


Functest R2 Brahmaputra page

Release Info

Summary

Criteria Functest Status
1. Project name Functest
2. Project status indicator for milestone C readiness: Green/Yellow/Red; GREEN = on track; YELLOW = working minor issues to get back on track; RED = offtrack with major issues Yellow/Red
3.Name of person providing status Morgan Richomme, Jose Lausuch
4.a link to your Jira tasks (filtered on R2) Functest R2 JIRA
5.a link to your Jira dashboard JIRA Dashboard Kanban

Brahmaputra Summary

Please note that you may have a dynamic view per installer, pod or scenario

New Functest dashboard look.

this table reflects the status but it is not dynamically up to date as CI never stops..

We consider hereafter only baremetal scenario.

Icon description:

: 4+ consecutive runs passing the success criteria
: <4 successful consecutive runs but passing the success criteria
: close to pass the success criteria
: 0% success, not passing

Fuel

POD: ericsson-pod2
source: https://build.opnfv.org/ci/view/functest/job/functest-fuel-ericsson-pod2-daily-brahmaputra/

Scenario VIM Controllers Features Iteration Comments
vPing SSH vPing userdata Tempest [1] Rally ODL ONOS OVNO PROMISE vIMS SDNVPN
nosdn-nofeature N.R N.R N.R N.R Tempest success rate 86%
odl_l3-ha [3] N.R N.R N.R Tempest success rate 87%
odl_l2-ha N.R N.R N.R Tempest success rate 83%
odl_l2-bgpvpn-noha
onos-ha [2] N.R N.R N.R N.R N.R Tempest success rate 87%
nosdn-ovs-ha N.R N.R N.R N.R Tempest success rate 87%
nosdn-kvm-ha
nosdn-ovs_kvm-ha

[1]: Detected some errors (~7-8) possibly due to insufficient IPs in the public network ip range. Investigation in progress. Possible fix by Jonas: https://gerrit.opnfv.org/gerrit/#/c/10159/

[2]: It failed 2 times in the last 5 runs.. waiting for more runs to see if it is impracticable. Ashley proposed a patch to fix this and yardstick problems with SSH in this scenario

[3]: last 2 runs it has failed



Apex

POD: opnfv-jump-1
source: https://build.opnfv.org/ci/view/functest/job/functest-apex-opnfv-jump-1-daily-brahmaputra/

Scenario VIM Controllers Features Iteration Comments
vPing SSH vPing userdata Tempest Rally ODL ONOS OVNO PROMISE vIMS SDNVPN
odl_l3-ha N.R N.R N.R. N.R Tempest success rate 70%
odl_l2-ha N.R N.R N.R. N.R. Tempest success rate 82%
onos-ha N.R [1] N.R N.R N.R. N.R. N.R. Tempest success rate 87 %
odl_l2-sfc
ocl-nofeature-ha

[1] Nova metadata service is not working properly with ONOS yet. Maybe in SR1.



Compass

POD: huawei-us-deploy-bare-1
source: https://build.opnfv.org/ci/view/functest/job/functest-compass-huawei-us-deploy-bare-1-daily-brahmaputra/

Scenarios ready for Release:

  • nosdn
  • odl_l2
  • onos

Ocl planned for SR1

Scenario VIM Controllers Features Comments
vPing SSH vPing userdata Tempest Rally ODL ONOS OVNO PROMISE vIMS SDNVPN
nosdn-nofeature-ha N.R N.R N.R N.R N.R success criteria reached 8/2
odl_l2-ha N.R N.R N.R N.R success criteria reached 9/2
onos-ha N.R [1] N.R N.R. N.R N.R N.R success criteria reached 4/2
ocl-nofeature-ha N.R N.R N.R N.R planned for SR1, first run on 19/2

[1]: There are 11 failed tests out of 210 which (we think) are related to nova metadata service not supported by onos yet. Some tests need to pass the ssh-key to the instances using that service. However, it passes the success criteria of >=90%



Joid

POD: orange-pod2
source: https://build.opnfv.org/ci/view/functest/job/functest-joid-orange-pod2-daily-master/

Scenarios ready for Release:

  • nosdn
  • odl_l2

vIMS use case fails but it is linked to configuration, solutions are discussed on joid etherpad.

onos scenario (run only 2 times on Orange POD 2) fails due to vPing/ssh

Scenario VIM Controllers Features Comments
vPing SSH vPing userdata Tempest Rally ODL ONOS OVNO PROMISE vIMS SDNVPN
nosdn-nofeature [1] N.R N.R N.R N.R criteria reached on the 21 ( vIMS excluded). Tempest rate 88% but some errors can be explained by config so can be removed from the equation leading to 93%
odl_l2-ha N.R N.R N.R Criteria reached on the 21 (vIMS excluded). Tempest rate = 98%
onos-ha N.R. N.R N.R. N.R. N.R
ocl-nofeature-ha


IMPORTANT: intel-pod5 and intel-pod6 won't count for the results report until we fix the possible problems why we get those big timeouts.

POD: intel-pod5
source: https://build.opnfv.org/ci/view/joid/job/functest-joid-intel-pod5-daily-brahmaputra/

Scenario VIM Controllers Features Iteration Comments
vPing SSH vPing userdata Tempest Rally [1] ODL ONOS OVNO PROMISE vIMS SDNVPN
nosdn-nofeature N.R N.R N.R N.R
odl_l2-ha N.R N.R N.R
onos-ha N.R N.R N.R N.R
ocl-nofeature-ha

[1] Rally takes 4 hr to execute. Something is wrong here…


POD: intel-pod6
source: https://build.opnfv.org/ci/view/functest/job/functest-joid-intel-pod6-daily-master/

Scenario VIM Controllers Features Iteration Comments
vPing SSH vPing userdata Tempest Rally ODL ONOS OVNO PROMISE vIMS SDNVPN
nosdn-nofeature [1] N.R N.R N.R N.R
odl_l2-ha [1] N.R N.R N.R 10 errors on promise test case out of 32
onos-ha [2] N.R. N.R N.R. N.R 10 errors on promise test case out of 32
ocl-nofeature-ha

[1] No runs found with this scenario on this pod

[2]: It seems the instances don't get an IP from the neutron dhcp agent…



Brahmaputra Troubleshooting

Associated Functest Wiki pages

Feature projects dependencies

Feature project actions for functest Contacts
Copper if integrated in B-release, goal will consist in adding tempest/Rally tests related to Congress Brian
Doctor test fully managed by Doctor team. Goal for functest consists in integrating the tests into the suite and run it on the different target CI labs Ryota
ONOSFW create a test suite Managed by Qinglong
OVNO see minutes meeting with Stuart Stuart
PolicyTest extend ODL test suites with GBP tests already created upstream in ODL Kieth / Peter & Amaged
Promise wait for feedback peter Lee
SDNVPN RF tests for ODL VPN service using ODL REST API (import from ODL) Tim Irnich / José

SFC managed by yardstick

Studies

VNF

New scenarios

For Arno, we reused existing upstream suites (except vIMS). For Rally bench scenario, we aggregated the test in 1 scenario per module but we did not create original scenarios. It is time to think to our own scenario…

Exploitation of existing tests

Rally bench tests will provide figures (booting time, authentication,..). These tests will be performed on several testbed with the same OPNFV solution and the same tooling. The difference will thus be linked to the hardware and/or network configuration. It will be interesting to collect all the results (available under ./results/opnfv-[module]-[date].json under each platform) and try to interprate them.

New tools

We shall consider the integration of new tools such as [[yardstick]|yardstick]

Synchro with ETSI Test group

Synchronisation to be planned (e.g. pre deployement document discussed in Prague ⇒ how to create testcases to be integrated in continuous integration..)

Roadmap

imagined in July 2015

functextnexttaks.1456136397.txt.gz · Last modified: 2016/02/22 10:19 by Morgan Richomme