OPNFV Brahmaputra Release will integrate multiple installers, controllers and artifacts that are produced by feature projects. In order to work with this many combinations and test the platform properly, it is important to make sure that the OPNFV Platform is deployed using known/working versions of installers since the platform itself needs to be tested. Any deployment issues that may be caused by installers could result in loss of hours for testing projects.

Current CI process does not support this since the CI is driven by installers. This means that we always build/use the latest versions of installers, try to deploy the platform using this latest version and then run the testing. Time to time the deployment fails, requiring manual intervention to deploy a known/working version of installer.

In following sections, you will see the current CI process and the proposal for OPNFV Brahmaputra Release.

As highlighted in the previous section, current CI process is driven by the installers. All installers have installer specific daily jobs that run jobs in certain order; build, deploy, and test as shown on below diagram. Build and deploy jobs are also installer specific and test jobs execute same tests for all installers.

installer-daily

1. installer-build: Checkout the latest version of the installer repo, build the ISO, and store the ISO on OPNFV Artifact repository as latest.
   1. Installer metadata gets stored in a file named latest.properties in OPNFV Artifact Repository.
   2. If installer is not ISO based, this step does not exist.
2. installer-deploy: Checkout the latest version of the installer, download the latest ISO using latest.properties file and start the deployment.
   1. If installer is not ISO based, the deployment is done using the latest version of the scripts located in installer repo.
3. test-projects: Jobs of test projects get triggered by installer-daily job if the deployment is successful.

As seen above, the deployment may fail, resulting in not running the testing. This is due to 2 main reasons

• Always latest versions of installers are used for deploying the platform without knowing if the latest version works
• Low number of resources causes us to have installer build and deploys done together with the tests.

Due to reasons listed above, OPNFV CI process needs to be improved, separating the installer specific build and deploy from testing.

This can be achieved by introducing simple promotion mechanism meaning that installer jobs can identify daily stable versions by running build and deploy as necessary, marking the version that is used as stable and exiting.

When it comes to testing focused CI process, the first thing that needs to be done is choosing the stable version of an installer to do the deployment. If there is a more recent stable version then the deployment can use this version to deploy the platform. If the installer(s) are unable to produce recent stable version then the deployment either can be done using the latest known stable version or can be skipped altogether if the test projects leave the SUT as they found it.

The installer part is similar to current CI process; each installers has its own daily, build, and deploy jobs. Once a stable version of installer is determined, the test focused runs will use this stable version and execute same testing against the deployed platform. So testing should be same for all 4 installers.

Below diagram shows this in high level.

Here are the steps.

installer-daily

1. installer-build: Checkout the latest version of the installer repo, build the ISO, and store the ISO on OPNFV Artifact repository as latest.
   1. If installer is not ISO based, this step does not exist.
2. installer-deploy: Checkout the latest version of the installer, download the latest ISO and start the deployment.
   1. If the deployment succeeds: Mark the installer version as stable by placing installer metadata in stable.properties file and upload this file to OPNFV Artifact Repository, overwriting the existing stable.properties file. The stable.properties file contains the installer repo SHA1 so non-ISO based installers are also covered.
   2. If the deployment fails: Do nothing.
   3. If installer is not ISO based, the deployment is done using the latest version of the scripts located in installer repo.

opnfv-daily

1. opnfv-installer-deploy: Fetch the stable.properties and initiate the deployment.
   1. If the installer is ISO based, download the stable ISO, checkout the stable version of the installer repo using the SHA1 from stable.properties file and proceed with the deployment.
   2. If the installer is not ISO based, just checkout the stable version of the installer repo using the SHA1 from stable.properties file and proceed with the deployment.
   3. As noted before, if redeployment is not necessary, this step can totally be skipped.
2. test-projects: Run the testing as usual.

Please note that the stability referred here is something that is deployable. There are many details that are not taken into account in this proposal. (such as feature addons and so on)

As expected, above proposal requires additional resources to be available in order to separate the installer specific CI from the OPNFV Platform CI that is used for testing purposes. Infra and test projects are in an agreement regarding this need and have been raising this issue as a risk to the community. It is important to get approval from TSC in order to use community labs for Brahmaputra Release Testing.

Current planning regarding the resources can be seen on below table.

As explained in the proposal, only "stable" version shall be deployed on the target platforms and the Stable row points to target labs to do this. We still need to identify where to run installer specific CI.

Another thing to highlight is that the same tests need to be executed on all PODs, guaranteeing the portability of OPNFV target platform.