====== OPNFV CI Infrastructure ======

//**
The information presented here is yet to be discussed within Octopus Team and should be seen as input to discussion unless noted otherwise.
**//

There are several tools and hardware resources that make it possible for us to develop/establish CI Framework for OPNFV. 

Most of the tools needed for CI are already provided by Linux Foundation. We need to setup/configure/utilize these tools. 

OPNFV hardware resources will be delivered to LF during coming week(s) so first CI POCs can/will be done on community donated hardware if things don't move on fast enough.

We also need to identify what other tools/resources we need with the help from wider OPNFV Community and make them available with the help from LF. 

Please note that the information on this page will be detailed/evolved while we move on.

====== Overview ======

Below diagram tries to capture some of the tools/hardware resources needed for Octopus and does not mean how the things work. 

See [[octopus:flows|this page]] for CI Flows.

{{ :octopus:basic_ci_infra.png?nolink&720 |}}

====== Tools/Resources ======

^ Tool/Resource           ^ Name    ^ Status                ^ Tasks Left to Do   ^ Notes                     ^
| CI Engine               | Jenkins | Pending configuration | See below          | [[wiki:jenkins|Link to general info]] |
| SCM                     | Git     | Pending configuration | See below          | [[:git|Link to general info]] |
| Code Review             | Gerrit  | Pending configuration | See below          | [[:git|Link to general info]] |
| Artifact Repository     | TBD     | TBD                   | See below          | [[wiki:artifactrepos|Link to general info]] |
| Build Servers           | N/A     | TBD                   | See below          | N/A |
| Test/Deployment Servers | N/A     | TBD                   | See below          | N/A |

====== Tasks ======

This section lists tasks to be handled in order to setup and configure tools and hardware resources.

Tasks that are not strikethroughed are pending.

===== Jenkins =====

^ Summary                          ^ Priority    ^ Responsible  ^ Notes ^
| Create CI Service/Functional User | High        | LF           | Will be used for connecting LF/community Jenkins slaves, running jobs, voting in Gerrit, mail notification, chat, etc. \\ Suggested id: **octopus** |
| Configure CI Service/Functional User in Jenkins | High        | LF           | SSH keys etc. setup on Jenkins. |
| Configure Jenkins master/apply configuration to Jenkins Global Configuration| High | LF | Whatever is necessary.|
| Create domains | Medium | LF | Whatever is necessary.|
| Install Jenkins Plugins and configure them | High | LF | Please see below or [[https://wiki.opnfv.org/octopus/infrastructure#list_of_plugins_to_install|click here]] for initial list of plugins.|
| Connect Jenkins to Gerrit        | High    | LF   | Need to be able to trigger jobs on Gerrit events. \\ Please see [[https://wiki.jenkins-ci.org/display/JENKINS/Gerrit+Trigger#GerritTrigger-AdministrativeSettings|here.]] |
| Connect/configure slaves         | High    | LF   | Can/will start with echoing "Hello World!" by jobs running on master. \\ And then connect/use community(Ericsson/Intel BGS) hosting until LF resources are ready.|
| Create pools/attach labels on slaves so build and test/deployment resources are properly used | High | LF/Octopus Team| N/A  |
| Configure mail notification | Low | LF | This could be handled like opt-in if people don't want to be bugged.|
| Make Jenkins talk/chat      | Low | LF   | Like it is done with #opnfv-gerrit chatroom. \\ Please see [[https://wiki.jenkins-ci.org/display/JENKINS/Instant+Messaging+Plugin|here]] and [[https://wiki.jenkins-ci.org/display/JENKINS/Jabber+Plugin|here]]. |
| Create basic jobs to evaluate what can be done without proper HW | High | Octopus Team | Create and run some jobs to see how urgent to get real HW for real Gerrit Verification flow etc.|
| Create basic jobs to verify the Gerrit connection | High | Octopus Team | Do simple commits and check if things are triggered/voted on Gerrit/mail notification sent.|
| Decide/agree which repos to monitor/trigger jobs on | High | Octopus Team/OPNFV Community | We start with the repos with some code/activity in them, such as BGS.|

===== Git/Gerrit =====

^ Summary                          ^ Priority    ^ Responsible  ^ Notes ^
| Configure CI Service/Functional User in Gerrit | High        | LF           | SSH keys etc. setup on Gerrit. \\ Also it should be granted to Gerrit stream events. \\ Please see [[https://wiki.jenkins-ci.org/display/JENKINS/Gerrit+Trigger#GerritTrigger-Gerritaccessrights|here]].  |
| Configure Mail Notification | High | LF           | This could be duplicate to Jenkins Mail notification and can be opted in. \\ But in general, people should be notified if a certain commit passed Gerrit verification so they can go and do the code review on Gerrit. |
| Enable Git/Gerrit hooks for large file check | Medium | LF/Octopus Team/OPNFV Community | This is still under discussion but it would be good if we can start with some initial setting so we don't mess up with the size of the repos right in the beginning. \\ This obviously requires discussion/agreement with other OPNFV projects who have Git repos. \\ Please see [[https://wiki.opnfv.org/git#how_to_control_file_size_in_git_commits|here]]. |
| Do simple commit/push/wait for verification/review stuff| High | Octopus Team | In order to evaluate if things are setup fine. Similar to Jenkins basic job creation task above. |

===== Artifact Repository =====

^ Summary                          ^ Priority    ^ Responsible  ^ Notes ^
| Agree on if we need artifact repository or not | High        | Octopus Team/OPNFV Community| SSH keys etc. setup on Gerrit. \\ Also it should be granted to Gerrit stream events. \\ Please see [[https://wiki.jenkins-ci.org/display/JENKINS/Gerrit+Trigger#GerritTrigger-Gerritaccessrights|here]].  |
| Decide which artifact repository to use with the guidance from LF | High | Octopus Team/LF/OPNFV Community| Swift/Artifactory/Nexus|
| Guide OPNFV Community regarding artifact repository selection | High | LF| Given that LF already hosts repositories for other OSS projects. |
| Decide and create repositories | Medium | Octopus Team/LF| We can start with "a" repo and evolve later on. |
| Decide retention policy | Medium | Octopus Team/LF| If storage becomes the bottleneck, we may need to start cleaning up based on the policy. |
| Acquire/provide storage if storage becomes the bottleneck even after enforcing the the retention policy | Low | OPNFV Management | This needs to be evaluated while we move on and raised by the community.  |

===== Hardware Resources =====

^ Summary                          ^ Priority    ^ Responsible  ^ Notes ^
| Configure/connect/evaluate Ericsson hosting until LF resources arrive | High        | LF/Octopus Team/Ericsson OPNFV Contributors| This requires discussions.  |
| Configure/connect/evaluate Intel hosting until LF resources arrive | High        | LF/Octopus Team/Intel OPNFV Contributors| This requires discussions.  |
| Assist Ericsson/Intel OPNFV Contributors to securely connect hosting to Jenkins | High        | LF| This requires discussions.  |
| Configure/connect LF resources | High        | LF/Octopus Team| N/A  |

====== Jenkins Job Builder ======

During Octopus Team Meeting on 2015-02-16, it has been suggested to give a try and perhaps use [[http://ci.openstack.org/jenkins-job-builder/|OpenStack Jenkins Job Builder]] for OPNFV CI. 

OpenDaylight also uses Jenkins Job Builder for Jenkins Jobs so this is a common way of handling Jenkins related things. As OPNFV uses OpenStack and OpenDaylight as upstream projects for the components themselves, using/doing CI in a similar way is also logical.

As a first step, LF will help us setting up some simple jobs using Jenkins Job Builder and they will be put in Octopus repo on OPNFV Gerrit. Then we will continue with the rest of the evaluation but this looks quite promising as the things stand.

Jenkins Job Builder helps us to develop/maintain/handle Jenkins job configurations like the source code; changes to jobs go through regular Gerrit review process and makes things visible to greater OPNFV Community. Having job configurations version controlled has its own benefits which is not needed to be listed here. Deploying those jobs could also be automated.

Apart from all the good things listed in previous sentence, it also relieves us from creating/modifying/maintaining Jenkins jobs from Jenkins GUI, potentially reducing chance of mistakes and annoyance of clicking things around.

Here are some links
  * Jenkins Job Builder: http://ci.openstack.org/jenkins-job-builder/
  * How OpenDaylight uses Jenkins: https://wiki.opendaylight.org/view/RelEng:Jenkins
  * OpenDaylight Builder: https://jenkins.opendaylight.org/releng/view/builder/
  * OpenDaylight Build/Release jobs on Jenkins: https://jenkins.opendaylight.org/releng/

====== List of Plugins to Install ======

This is the initial list of plugins to install and new ones will probably be added based on Octopus/Community feedback/needs.

The ones that are marked as low priority can be skipped for the timebeing to see/evaluate the load on Jenkins master when other plugins are installed. If the load looks fine, they can be installed as well.

^ Plugin  ^ Purpose  ^ Responsible  ^ Priority ^ Link ^
|SSH Slaves Plugin |Connecting slaves to Jenkins with SSH | LF | High | [[https://wiki.jenkins-ci.org/display/JENKINS/SSH+Slaves+plugin|SSH Slaves Plugin]] |
| Git Plugin |Use Git as SCM| LF| High | [[https://wiki.jenkins-ci.org/display/JENKINS/Git+Plugin|Git Plugin]] |
| Gerrit Plugin | Integrate Gerrit with Jenkins| LF| High | [[https://wiki.jenkins-ci.org/display/JENKINS/Gerrit+Plugin|Gerrit Plugin]] |
| Gerrit Trigger Plugin | Integrate Gerrit with Jenkins to trigger jobs | LF| High | [[https://wiki.jenkins-ci.org/display/JENKINS/Gerrit+Trigger|Gerrit Trigger Plugin]]|
| Parameterized Trigger Plugin | Create jobs that can trigger other jobs and take action depending on result | LF| High | [[https://wiki.jenkins-ci.org/display/JENKINS/Parameterized+Trigger+Plugin|Parameterized Trigger Plugin]] |
| Multijob Plugin |Create complex/hierarchical jobs | LF| High | [[https://wiki.jenkins-ci.org/display/JENKINS/Multijob+Plugin|Multijob Plugin]]|
| Job DSL Plugin | Create jobs programmatically | LF| Medium | [[https://wiki.jenkins-ci.org/display/JENKINS/Job+DSL+Plugin|Job DSL Plugin]] |
| Workflow Plugin | Create workflows |LF| Low |[[https://github.com/jenkinsci/workflow-plugin|Workflow Plugin]] | 
|Robot Framework Plugin| Publishes/stores Robot FW test reports| LF |High |[[https://wiki.jenkins-ci.org/display/JENKINS/Robot+Framework+Plugin|Robot Framework plugin]]|
| Throttle Concurrent Builds Plugin | Throttle concurrent builds |LF|High | [[https://wiki.jenkins-ci.org/display/JENKINS/Throttle+Concurrent+Builds+Plugin|Throttle Concurrent Builds Plugin]]|
|Email-ext Plugin | Send customized emails |LF|Medium | [[https://wiki.jenkins-ci.org/display/JENKINS/Email-ext+plugin|Email-ext Plugin]] |
|Instant Messaging/Jabber Plugin | Make Jenkins talk/chat |LF|Low | [[https://wiki.jenkins-ci.org/display/JENKINS/Instant+Messaging+Plugin|Instant Messaging Plugin]], [[https://wiki.jenkins-ci.org/display/JENKINS/Jabber+Plugin|Jabber Plugin]]|
|eXtreme Feedback Plugin | Show quick build status |LF|High | [[https://wiki.jenkins-ci.org/display/JENKINS/eXtreme+Feedback+Panel+Plugin|eXtreme Feedback Plugin]] |
|Build Pipeline Plugin | Create and visualize pipelines |LF|Medium | [[https://wiki.jenkins-ci.org/display/JENKINS/Build+Pipeline+Plugin|Build Pipeline Plugin]]|
|Green Balls | Use Green Balls for successful builds |LF|High | [[https://wiki.jenkins-ci.org/display/JENKINS/Green+Balls|Green Balls]]|
| Plot Plugin | Create plots | LF | Low | [[https://wiki.jenkins-ci.org/display/JENKINS/Plot+Plugin|Plot Plugin]] |
| Priority Sorter Plugin | Adjust priorities of jobs | LF | Medium | [[https://wiki.jenkins-ci.org/display/JENKINS/Priority+Sorter+Plugin|Priority Sorter Plugin]] |
| Dashboard View Plugin | Create dashboards | LF | Medium | [[http://wiki.jenkins-ci.org/display/JENKINS/Dashboard+View|Dashboard View]]|
| Static Code Analysis Plugin | Provide Static Code Analysis utilities | LF |High | [[https://wiki.jenkins-ci.org/display/JENKINS/Static+Code+Analysis+Plug-ins|Static Code Analysis Plugin]]|
| Artifact Deployer Plugin | Deploy artifacts from workspace to output directories | LF | Medium | [[https://wiki.jenkins-ci.org/display/JENKINS/ArtifactDeployer+Plugin|ArtifactDeployer Plugin]]|
|Promoted builds plugin | Mark builds as promoted | LF | Medium | [[https://wiki.jenkins-ci.org/display/JENKINS/Promoted+Builds+Plugin|Promoted Builds Plugin]]|
|Build Failure Analyser| Present causes of failures | LF | Low | [[https://wiki.jenkins-ci.org/display/JENKINS/Build+Failure+Analyzer|Build Failure Analyser]]|
|Build Timeout Plugin| Set timeout so builds don't get stuck| LF |High | [[https://wiki.jenkins-ci.org/display/JENKINS/Build-timeout+Plugin|Build Timeout Plugin]]|
|EnvInject Plugin| Inject env variables to builds |LF |High | [[https://wiki.jenkins-ci.org/display/JENKINS/EnvInject+Plugin|EnvInject Plugin]]|
|Git Parameter Plugin| Choose revisions or tags | LF |Medium | [[https://wiki.jenkins-ci.org/display/JENKINS/Git+Parameter+Plugin|Git Parameter Plugin]]|
|Job Config History Plugin| Track job configuration history | LF |High | [[https://wiki.jenkins-ci.org/display/JENKINS/JobConfigHistory+Plugin|JobConfigHistory Plugin]]|
|Log Parser Plugin| Parse console logs| LF |High | [[https://wiki.jenkins-ci.org/display/JENKINS/Log+Parser+Plugin|Log Parser Plugin]]|
|Mask Passwords Plugin| Mask passwords | LF |High | [[https://wiki.jenkins-ci.org/display/JENKINS/Mask+Passwords+Plugin|Mask Passwords Plugin]]|
|Post Build Task Plugin| Do post build tasks | LF |High | [[https://wiki.jenkins-ci.org/display/JENKINS/Post+build+task|Post Build Task Plugin]]|
|Show Build Parameters Plugin| Show parameters used for a certain build | LF |High | [[https://wiki.jenkins-ci.org/display/JENKINS/Show+Build+Parameters+Plugin|Show Build Parameters plugin]]|
|Timestamper Plugin|Add timestamps to console | LF |High | [[https://wiki.jenkins-ci.org/display/JENKINS/Timestamper|Timestamper Plugin]]|
|Delivery Pipeline Plugin| Visualize delivery pipeline| LF |Low | [[https://wiki.jenkins-ci.org/display/JENKINS/Delivery+Pipeline+Plugin|Delivery Pipeline Plugin]]|
|Robot Framework Plugin| Publishes/stores Robot FW test reports| LF |High |[[https://wiki.jenkins-ci.org/display/JENKINS/Robot+Framework+Plugin|Robot Framework plugin]]|