Wiki

This page is read only. You can view the source, but not change it. Ask your administrator if you think this is wrong.

====== Brahmaputra Summary ======

Please note that you may have a dynamic view per [[http://testresults.opnfv.org/proto/index2.html|installer]], [[http://testresults.opnfv.org/proto/index-pod.html|pod]] or [[http://testresults.opnfv.org/proto/index-scenario.html|scenario]]

New Functest [[http://testresults.opnfv.org/dashboard/|dashboard]] look.

this table reflects the status but it is not dynamically up to date as CI never stops.. 

We consider hereafter only baremetal scenario.


Icon description:

{{:weather-clear.png?32}} : 4+ consecutive runs passing the success criteria\\
{{:weather-few-clouds.png?32}} : <4 successful consecutive runs but passing the success criteria\\
{{:weather-overcast.png?32}} : close to pass the success criteria\\
{{:weather-storm.png?32}} : 0% success, not passing\\


=== Fuel ===
POD: ericsson-pod2 \\
source: https://build.opnfv.org/ci/view/functest/job/functest-fuel-ericsson-pod2-daily-brahmaputra/

Scenario ready for Release:

  * nosdn
  * old_l2
  * onos
  * nosdn-ovs

odl_l3 blocked by vPing

bgpvpn not mature



^ Scenario            ^  VIM                                                                                                                              |||^  Controllers                                                  ||^  Features                                                                                    ||^  Comments                                                                                                                                                                                                    ^
|                     | vPing SSH                           | vPing userdata                  | Tempest [1]                     | Rally                      | ODL                        | ONOS                       | OVNO  | PROMISE                         | vIMS                            | SDNVPN                     |                                                                                                                                                                                                                           |
| nosdn-nofeature     | {{:weather-clear.png?32}}           | {{:weather-clear.png?32}}       | {{:weather-few-clouds.png?32}}  | {{:weather-clear.png?32}}  | N.R                        | N.R                        | N.R   | {{:weather-clear.png?32}}       | {{:weather-few-clouds.png?32}}  | N.R                       | criteria reached on the 18/2 \\ Tempest rate 89% but some errors can be explained by config so can be removed from the equation leading to > 90% \\ vIMS suite #37, the previous and following ones were OK  |
| odl_l3-ha           | {{:weather-few-clouds.png?32}} [3]  | {{:weather-clear.png?32}}       | {{:weather-overcast.png?32}}    | {{:weather-clear.png?32}}  | {{:weather-clear.png?32}}  | N.R                        | N.R   | {{:weather-clear.png?32}}       | {{:weather-clear.png?32}}       | N.R                                | Tempest success rate 83%                                                                                                                                                                                     |
| odl_l2-ha           | {{:weather-clear.png?32}}           | {{:weather-clear.png?32}}       | {{:weather-few-clouds.png?32}}  | {{:weather-clear.png?32}}  | {{:weather-clear.png?32}}  | N.R                        | N.R   | {{:weather-clear.png?32}}       | {{:weather-clear.png?32}}       | N.R                                   | Tempest success rate 83% but errors mainly in test_server_action and orchestration.stacks. Some run exclude these tests and lead to > 90%, when included we have reproducible results                        |
| odl_l2-bgpvpn-noha  | {{:weather-few-clouds.png?32}}      | {{:weather-few-clouds.png?32}}  | {{:weather-overcast.png?32}}    | {{:weather-storm.png?32}}  | {{:weather-clear.png?32}}  | N.R                        | N.R   | {{:weather-few-clouds.png?32}}  | {{:weather-storm.png?32}}       | {{:weather-storm.png?32}}  |             | Run only once                                                                                                                                                            
| onos-ha             | {{:weather-clear.png?32}} [2]       | N.R                             | {{:weather-few-clouds.png?32}}  | {{:weather-clear.png?32}}  | N.R                        | {{:weather-clear.png?32}}  | N.R   | {{:weather-clear.png?32}}       | N.R                             | N.R                        |            Tempest success rate 85% but test cases related to nova metadata shall be excluded leading to 90%                                                                                                            |
| nosdn-ovs-ha        | {{:weather-clear.png?32}}           | {{:weather-clear.png?32}}       | {{:weather-overcast.png?32}}    | {{:weather-clear.png?32}}  | N.R                        | N.R                        | N.R   | {{:weather-clear.png?32}}       | {{:weather-clear.png?32}}       | N.R                        | Tempest success rate 87%                                                                                                                                                                                     |
| nosdn-kvm-ha        |                                     |                                 |                                 |                            |                            |                            |       |                                 |                                                         |             |                                                                                                                                                                                                              |
| nosdn-ovs_kvm-ha    |                                     |                                 |                                 |                            |                            |                            |       |                                 |                                                            |             |                                                                                                                                                                                                              |







[1]: Detected some errors (~7-8) possibly due to insufficient IPs in the public network ip range. Investigation in progress. Possible fix by Jonas: https://gerrit.opnfv.org/gerrit/#/c/10159/

[2]: It failed 2 times in the last 5 runs.. waiting for more runs to see if it is impracticable. Ashley proposed a patch to fix this and yardstick problems with SSH in this scenario

[3]: last 2 runs it has failed

\\
\\



=== Apex ===


Scenario ready for Release:

  * old_l2

Doctor feature ready for the release

we may assume that nosdn is working but never run in CI

odl_l3 and onos blocked by vPing and Tempest success rate

POD: opnfv-jump-1\\
source: https://build.opnfv.org/ci/view/functest/job/functest-apex-opnfv-jump-1-daily-brahmaputra/ \\

^ Scenario          ^  VIM                                                                                                               |||^  Controllers                                                       ||^  Features                                                                         |||^  Comments                                                                                                                                                 ^
|                   | vPing SSH                  | vPing userdata             | Tempest                       | Rally                       | ODL                             | ONOS                       | OVNO  | PROMISE    | vIMS                       | Doctor                           | SDNVPN  |                                                                                                                                                           |
| odl_l3-ha         | {{:weather-storm.png?32}}  | {{:weather-storm.png?32}}  | {{:weather-overcast.png?32}}  | {{:weather-clear.png?32}}   | {{:weather-few-clouds.png?32}}  | N.R                        | N.R   | N.R.       | {{:weather-storm.png?32}}  | {{:weather-few-clouds.png?32}}   | N.R     | Tempest rate not stable (from 70 to 90%), number of tests run varies a lot from one run to another \\ 3 ODL test cases partly failed                      |
| odl_l2-ha         | {{:weather-clear.png?32}}  | {{:weather-clear.png?32}}  | {{:weather-overcast.png?32}}  |   {{:weather-clear.png?32}} | {{:weather-few-clouds.png?32}}  | N.R                        | N.R   | N.R.       | {{:weather-clear.png?32}}  | {{:weather-clear.png?32}}        | N.R.    | Tempest rate not stable (from 60 to 90%), number of tests run varies a lot from one run to another \\ 3 ODL test cases partly failed                      |
| onos-ha           | {{:weather-storm.png?32}}  | N.R [1]                    | {{:weather-clear.png?32}}     |  {{:weather-clear.png?32}}  | N.R                             | {{:weather-clear.png?32}}  | N.R   | N.R.       | N.R.                       |  {{:weather-few-clouds.png?32}}  | N.R.    | Number of tempest test run varies but success rate always above 90% \\ ONOS suite run #54 failed due to connection to git (was OK in the 4 previous run)  |
| odl_l2-sfc        |                            |                            |                               |                             |                                 |                            |       |            |                            |                                  |         |                                                                                                                                                           |
| ocl-nofeature-ha  |                            |                            |                               |                             |                                 |                            |       |            |                            |                                  |         |                                                                                                                                                           |





[1] Nova metadata service is not working properly with ONOS yet. Maybe in SR1. 



\\
\\



=== Compass === 
POD: huawei-us-deploy-bare-1 \\
source: https://build.opnfv.org/ci/view/functest/job/functest-compass-huawei-us-deploy-bare-1-daily-brahmaputra/

Scenarios ready for Release:
  * nosdn
  * odl_l2
  * onos

Ocl planned for SR1


^ Scenario            ^  VIM                                                                                                                |||^  Controllers                                                                       ||^  Features                                       ||^  Comments                           ^
|                     | vPing SSH                  | vPing userdata             | Tempest                         | Rally                      | ODL                        | ONOS                       | OVNO                       | PROMISE    | vIMS                       | SDNVPN  |                                     |
| nosdn-nofeature-ha  | {{:weather-clear.png?32}}  | {{:weather-clear.png?32}}  | {{:weather-clear.png?32}}       | {{:weather-clear.png?32}}  | N.R                        | N.R                        | N.R                        | N.R        | {{:weather-clear.png?32}}  | N.R     | success criteria reached 8/2        |
| odl_l2-ha           | {{:weather-clear.png?32}}  | {{:weather-clear.png?32}}  | {{:weather-clear.png?32}}       | {{:weather-clear.png?32}}  | {{:weather-clear.png?32}}  | N.R                        | N.R                        | N.R        | {{:weather-clear.png?32}}  | N.R     | success criteria reached 9/2        |
| onos-ha             | {{:weather-clear.png?32}}  | N.R                        | {{:weather-clear.png?32}}  [1]  | {{:weather-clear.png?32}}  | N.R                        | {{:weather-clear.png?32}}  | N.R.                       | N.R        | N.R                        | N.R     | success criteria reached 4/2        |
| ocl-nofeature-ha    | {{:weather-storm.png?32}}  | {{:weather-storm.png?32}}  | {{:weather-storm.png?32}}       | {{:weather-storm.png?32}}  | N.R                        | N.R                        | {{:weather-storm.png?32}}  | N.R        | {{:weather-storm.png?32}}  | N.R     | planned for SR1, first run on 19/2  |


[1]: There are 11 failed tests out of 210 which (we think) are related to nova metadata service not supported by onos yet. Some tests need to pass the ssh-key to the instances using that service. However, it passes the success criteria of >=90%




\\
\\

=== Joid ===
POD: orange-pod2 \\
source: https://build.opnfv.org/ci/view/functest/job/functest-joid-orange-pod2-daily-master/ 

Scenarios ready for Release:
  * odl_l2

nosdn scenario shall be OK but not run 4 times, will be added next time

vIMS use case fails but it is linked to configuration, solutions are discussed on joid etherpad. 

onos scenario (run only 2 times on Orange POD 2) fails due to vPing/ssh


^ Scenario             ^  VIM                                                                                                                   |||^  Controllers                                                       ||^  Features                                                       ||^ Comments                                                                                                                                                               ^
|                      | vPing SSH                  | vPing userdata             | Tempest                       | Rally                           | ODL                        | ONOS                            | OVNO  | PROMISE                    | vIMS                       | SDNVPN  |  
| nosdn-nofeature [1]  | {{:weather-clear.png?32}}  | {{:weather-clear.png?32}}  | {{:weather-clear.png?32}}     | {{:weather-clear.png?32}}       | N.R                        | N.R                             | N.R   | {{:weather-clear.png?32}}  | {{:weather-storm.png?32}}  | N.R     | criteria reached on the 21 ( vIMS excluded). Tempest rate 88% but some errors can be explained by config so can be removed from the equation leading to 93%  |
| odl_l2-ha            | {{:weather-clear.png?32}}  | {{:weather-clear.png?32}}  | {{:weather-clear.png?32}}     | {{:weather-clear.png?32}}       | {{:weather-clear.png?32}}  | N.R                             | N.R   | {{:weather-clear.png?32}}  | {{:weather-storm.png?32}}  | N.R     | Criteria reached on the 21 (vIMS excluded). Tempest rate = 98%                                                                                               |
| onos-ha              | {{:weather-storm.png?32}}  | N.R.                       | {{:weather-overcast.png?32}}  | {{:weather-few-clouds.png?32}}  | N.R                        | {{:weather-few-clouds.png?32}}  | N.R.  | {{:weather-clear.png?32}}  | N.R.                       | N.R     |                                                                                                                                                              |
| ocl-nofeature-ha     |                            |                            |                               |                                 |                            |                                 |       |                            |                  |                                                                                                                                                              |






\\

IMPORTANT: intel-pod5 and intel-pod6 won't count for the results report until we fix the possible problems why we get those big timeouts.

POD: intel-pod5 \\
source: https://build.opnfv.org/ci/view/joid/job/functest-joid-intel-pod5-daily-brahmaputra/
^ Scenario          ^  VIM                                                                                                                    |||^  Controllers                                  ||^  Features                                                            ||^  Iteration  ^  Comments  ^
|                   | vPing SSH                     | vPing userdata                | Tempest                    | Rally [1]                     | ODL                             | ONOS  | OVNO  | PROMISE                         | vIMS                    | SDNVPN  |                         ||
| nosdn-nofeature   |                               |                               |                            |                               | N.R                             | N.R   | N.R   |                                 |                            | N.R     |             |            |
| odl_l2-ha         | {{:weather-overcast.png?32}}  | {{:weather-overcast.png?32}}  | {{:weather-storm.png?32}}  | {{:weather-overcast.png?32}}  | {{:weather-few-clouds.png?32}}  | N.R   | N.R   | {{:weather-few-clouds.png?32}}  | {{:weather-storm.png?32}}  | N.R     |             |            |
| onos-ha           |                               | N.R                           |                            |                               | N.R                             |       | N.R   |                                 |                            | N.R     |             |            |
| ocl-nofeature-ha  |                               |                               |                            |                               |                                 |       |       |                                 |                            |         |             |            |




[1] Rally takes 4 hr to execute. Something is wrong here...


\\

POD: intel-pod6 \\
source: https://build.opnfv.org/ci/view/functest/job/functest-joid-intel-pod6-daily-master/
^ Scenario             ^  VIM                                                                                                                          |||^  Controllers                                                            ||^  Features                                                          ||^  Iteration  ^  Comments                                 ^
|                      | vPing SSH                       | vPing userdata                  | Tempest                    | Rally                           | ODL                             | ONOS                            | OVNO  | PROMISE                       | vIMS                    | SDNVPN  |                                                        ||
| nosdn-nofeature [1]  |                                 |                                 |                            |                                 | N.R                             | N.R                             | N.R   |                               |                            | N.R     |             |                                           |
| odl_l2-ha [1]        | {{:weather-few-clouds.png?32}}  | {{:weather-few-clouds.png?32}}  | {{:weather-storm.png?32}}  | {{:weather-few-clouds.png?32}}  | {{:weather-few-clouds.png?32}}  | N.R                             | N.R   | {{:weather-overcast.png?32}}  | {{:weather-storm.png?32}}  | N.R     |             | 10 errors on promise test case out of 32  |
| onos-ha              | {{:weather-storm.png?32}} [2]   | N.R.                            | {{:weather-storm.png?32}}  | {{:weather-overcast.png?32}}    | N.R                             | {{:weather-few-clouds.png?32}}  | N.R.  | {{:weather-overcast.png?32}}  | {{:weather-storm.png?32}}  | N.R     |             | 10 errors on promise test case out of 32  |
| ocl-nofeature-ha     |                                 |                                 |                            |                                 |                                 |                                 |       |                               |                            |         |             |                                           |


[1] No runs found with this scenario on this pod 

[2]: It seems the instances don't get an IP from the neutron dhcp agent...


\\
\\

==== Brahmaputra Troubleshooting ====

  * [[vPing_Brahmaputra_page|vPing]]
  * [[odl_Brahmaputra_page|odl]]
  * [[onos_brahmaputra_page|onos]]
  * [[tempest_Brahmaputra_page|tempest]]
  * [[vIms_Brahmaputra_page|vIMS]]
  * [[rally_Brahmaputra_page | Rally]]
Wiki

User Tools

Site Tools

Page Tools
