User Tools

Site Tools


pharos_rls_b_labs

This is an old revision of the document!


Status of OPNFV Lab Resources

This Pharos page explains the current status of Labs for support of development, test and CI activities. For future we plan to have dashboards with live information.

Tracking actual capabilities, status and ongoing usage of our diverse community lab resources is non-trivial … however its needed to support projects and releases with accurate planning. To achieve this we need easy visibility of our lab resources, their capabilities as well as current and past usage.

Information on Linux Foundation lab access and usage is found here https://wiki.opnfv.org/pharos/lf_lab

Information on project needs from labs was collected from lab owners by a survey (closed 16 Oct 2015) … https://www.surveymonkey.com/results/SM-CYC35WH2

  • 11 community labs currently are used by projects and releng activities
  • Labs that are not Pharos compliant may still provide servers as dev resources
  • Some labs still need to publish developer access process and guides
  • Some labs are not yet setup under Jenkins control
  • Labs that suffer from low bandwidth or unstable connections can still be used for specific project activities

Status of production CI resources for release B

See OPNFV CI Process Proposal for Brahmaputra https://wiki.opnfv.org/octopus/brahmaputra_ci_process

The references "stable" and "latest" mean the branches. PODs marked as "stable" will run against stable/brahmaputra branch and PODs marked as "latest" will run agains master branch. The testing will be done against both branches but the results from stable/brahmaputra counts when it comes to release candidate evaluation.

POD Installer Stable/Latest POD/Resources available & Tested with installer POD has Jenkins configured for automated deploys and tested Notes
LF POD 1 Apex stable Yes - Fully operational
Jenkins: apex-deploy-baremetal-master
Jenkins: Yes, Deploy: No, Test: No Still running against latest
Huawei SC POD1 Compass stable Yes - Fully operational Jenkins: Yes, Deploy: No, Test: No N/A
LF POD 2 Fuel stable Yes - Fully operational Jenkins: Yes, Deploy: Yes, Test: Yes N/A
Intel POD 5 Joid stable Yes - Fully operational
Jenkins: joid-odl-intel-pod5-daily-brahmaputra
Jenkins: Yes, Deploy: Yes, Test: Yes N/A
Intel POD 2 Apex latest Moved to new environment … waiting for Tim Rozet (Apex) to test and confirm if he still sees any issues Jenkins: No, Deploy: No, Test: No Apex is currently only working on LF POD 1 but will get back to use Intel POD 2
Intel POD 8 Compass latest Yes - Fully operational Jenkins: Yes, Deploy: No, Test: No N/A
Ericsson POD2 Fuel latest Yes - Network troubleshooting WIP Jenkins: Yes, Deploy: Yes, Test: Yes N/A
Intel POD 6 Joid latest Yes - Fully operational
Jenkins: joid-odl-intel-pod6-daily-master
Jenkins: Yes, Deploy: Yes, Test: Yes N/A
Intel POD 4 x5 Standalone Servers N/A Yes - Fully operational (Investigating increasing Internet bandwidth which is an issue for downloading large images frequently) Jenkins: Yes, Deploy: N/A, Test: N/A Build & virtual deploy resources
LF POD 3 x4 Standalone servers N/A No - WIP with 1 server Jenkins: No, Deploy: N/A, Test: N/A Build & virtual deploy resources
1 server is reserved for Joid virtual and WIP to take it into operation

See also: Jenkins OPNFV Platform CI main page

Production CI Resources Reservation

Test projects raised the need to access and work on CI Production PODs for troubleshooting purposes. The table below tries to start the discussion around how this can be done.

In ideal case, the PODs should be left in the state as it was failed but due to large number of scenarios, this will probably not be possible most of the times so that's why it is important to leave time for test projects on PODs to have the troubleshooting done on the POD where the problem occurred.

All the communication regarding PODs should be done in respective IRC Channels: #opnfv-testperf, #opnfv-yardstick, #opnfv-apex, #opnfv-fuel, #opnfv-compass, #opnfv-joid.

POD Installer Branch Reserved for CI Reserved for Troubleshooting Notes
LF POD1 Apex master 01:00UTC - 13:00UTC 16:00UTC - 01:00UTC
Booked for Yardstick: 16:00 UTC - 18:00UTC
Booked for Functest: 18:00 UTC - 20:00 UTC
Huawei SC POD1 Compass stable 10:00UTC - 22:00UTC 22:00UTC - 10:00UTC
Booked for Yardstick: 02:00 UTC - 04:00 UTC
LF POD2 Fuel stable 17:00UTC - 08:00UTC 11:00UTC - 17:00UTC
Booked for Yardstick: 13:00 UTC - 15:00 UTC
Booked for Functest: 15:00 UTC - 17:00 UTC
Intel POD5 Joid stable 17:00UTC - 05:00UTC 05:00UTC - 17:00UTC
Booked for Yardstick: 14:00 UTC - 16:00 UTC
Booked for Functest: 16:00 UTC - 17:00 UTC
Intel POD2 Apex N/A N/A N/A
Intel POD8 Compass master 22:00UTC - 10:00UTC 10:00UTC - 22:00UTC
Booked for Yardstick: 10:00 UTC - 12:00 UTC
Booked for Functest: 13:00 UTC - 15:00 UTC
Ericsson POD2 Fuel master 05:00UTC - 20:00UTC 20:00UTC - 05:00UTC
Booked for Yardstick: 20:00 UTC - 22:00 UTC
Intel POD6 Joid master 05:00UTC - 17:00UTC 17:00UTC - 05:00UTC
Booked for Yardstick: 17:00 UTC - 19:00 UTC
Booked for Functest: 19:00 UTC - 20:00 UTC

Notes:

  • It is assumed that the CI will run 3 scenarios per Installer/POD until the code freeze.
  • The reservations are done based on the locations of active contributors of installer and test projects.
  • The table tries to leave 1 POD per installer for troubleshooting purposes mainly to be used by test projects.
  • Approximate time to run full loop per scenario (deploy + functest + yardstick). These times are taken out from Jenkins jobs and might obviously be longer or shorter.
    • Apex : 4 hours per scenario
    • Compass: 4 hours per scenario
    • Fuel : 5 hours per scenario
    • Joid : 4 hours per scenario

Once the code freeze hits, time to do troubleshooting on the PODs dedicated for stable branch will go down drastically given the number of scenarios. Due to this, the troubleshooting work needs to be done on PODs running against master if the problem in question can be troubleshooted there.

It is also possible that installer teams might prefer test teams to use a certain POD longer than what is stated above and test teams get more time to work on certain POD for troubleshooting. But this should not result in not running CI for the branch on the POD in question or else this will increase the possibility of introducing faults to stable branch. The PODs must not turn into development resources!

Timezones

  • Beijing: UTC+8
  • CET : UTC+1
  • PDT : UTC-7

Community Lab Resources - Operational

Labs listed here are operational and being actively used for either development (features and/or tools), test (feature testing or dedicated to test project), production (dedicated with 24/7 SLA). Note a community lab can provide one or more of these capabilities.

For allocation of PODs for testing release B (there are 4 installers) see https://wiki.opnfv.org/brahmaputra_testing_page

Lab Pharos compliant PODs Stand-alone servers Type / Usage Production CI operational Project CI (Test) operational Developer resources with access guide and process published
China Mobile (Beijing) 1 POD 1 (Fuel) - dev for Functest and Yardstick tools No Yes Yes
Dell (Santa Clara) 3 POD 1 - available for CI allocation; POD 2 Qtip dev and test No (TBD) Yes Yes
Enea 1 non-HA server config with ARM servers No No No
Ericsson (Montreal) 2 4 Blades POD 1 (Fuel) - Yardstick testing (proj CI); POD 2 (Fuel) - production CI; N servers used for Dev based on Demand Yes Yes Yes
Huawei (Santa Clara) 2 2 POD 1 (Compass) - production CI; POD 2 - available for dev allocation; 2x servers - VMs available for dev allocation No (TBD) Yes No
Huawei (Xia-an) 2 4 POD 1 - available for proj allocation; CI POD 2 - available for proj CI allocation; 4x servers - dev for Compass No No No
Intel (Portland) 9 3 (POD 3 - VSPERF) POD 1 - Apex dev/test; POD 2 - Apex production CI; POD 3 - VSPERF test environment with Ixia traffic generator, OVSNFV, kvm4nfv; POD 4 - CI builds and virt deploys; POD 5 (JOID) - production CI; POD 6 - JOID dev/test; POD 7 - Fuel dev/test; POD 8 - Compass production CI; POD 9 - STORPERF / Bottlenecks Yes Yes Yes
Orange (Paris) 1 POD 1 (Fuel) - used for PoCs (vCPE), Functest dev, proj CI No Yes Yes
Orange (Lanion) 1 POD 1 - dev for JOID and Functest No No No
Spirent (San Jose) 12 Individual servers are used for various dev activities No No Yes
ZTE (Nan Jing) 1 POD 1 - dev/test for Functest and Yardstick No No Yes

Community Lab Resources - Upcoming

The following organization are actively working on providing additional OPNFV community labs …

1. Cable labs – currently have 2 PODs that are currently private resources  
2. CENGN – waiting on hardware to complete build out  
pharos_rls_b_labs.1453992392.txt.gz · Last modified: 2016/01/28 14:46 by Fatih Degirmenci