Draft proposal for way of working with stable branches and doing maintenance
The page is derived from https://wiki.openstack.org/wiki/StableBranch , simplified and adapted to OPNFV.

At this time only Arno release maintenance is covered.

The stable branch is intended to be a safe source of fixes for high impact bugs and security issues which have been fixed on master since a given release. It allows users of release (stable) versions to benefit from the ongoing bugfix work after the release.

Official point releases for each project are published from the branch on a per need basis, as decided by the TSC. In later stages, a regular cadence for point releases may be introduced.

It's possible to check current maintained versions in the releases page. At this time only Arno is maintained.

OPNFV's stable branch policy borrows much from prior art, in particular from OpenStack.

Only a limited class of changes are appropriate for inclusion on the stable branch.

A number of factors must be weighed when considering a change:

• The risk of regression - even the tiniest changes carry some risk of breaking something and we really want to avoid regressions on the stable branch
• The user visible benefit - are we fixing something that users might actually notice and, if so, how important is it?
• How self-contained the fix is - if it fixes a significant issue but also refactors a lot of code, it's probably worth thinking about what a less risky fix might look like
• Whether the fix is already on master - a change must be a backport of a change already merged onto master, unless the change simply does not make sense on master.

Rules to maintain multiple versions and exceptions will be added later.

The stable review team needs to balance the risk of any given patch with the value that it will provide to users of the stable branch. A large, risky patch for a major data corruption issue might make sense. As might a trivial fix for a fairly obscure error handling case.

Some types of changes are completely forbidden:

• New features
• Changes to the external APIs
• Changes to the notification definitions
• DB schema changes
• Incompatible config file changes

This excludes changes that include a version upgrade of an upstream component of OPNFV.

Support phases will be introduced at a later time

Each backported commit proposed to gerrit should be reviewed and +2ed by two Arno-stable-maint members before it is approved. Where a stable-maint member has backported a fix, a single other +2 is sufficient for approval.

If unsure about the technical details of a given fix, stable-maint members should consult with the appropriate developers from the affected projects for a more detailed technical review.

If unsure if a fix is appropriate for the stable branch, at this time the TSC will do the final decision.

Fixes for embargoed security issues receive special treatment. These should be reviewed in advance of disclosure by committers and stable-maint. At the time of coordinated public disclosure, the fix is proposed simultaneously to master and the stable branches and immediately approved.

Anyone can propose a cherry-pick to the stable-maint team.

One way is that if a bugfix on master looks like a good candidate for backporting - e.g. if it's a significant bug with the previous release - then just nominating the bug for Arno maintenance will bring it to the attention of the maintainers.

If you don't have the appropriate permissions to nominate the bug, then send an email via the user list.

The best way to get the patch merged in timely manner is to send it backported by yourself. To do so, you may try to use "Cherry Pick To" button in Gerrit UI for the original patch in master. Gerrit will take care of creating a new review, modifying commit message to include 'cherry-picked from …' line etc.

If the patch you're proposing will not cherry-pick cleanly, you can help by resolving the conflicts yourself and proposing the resulting patch. Please keep Conflicts lines in the commit message to help reviewers! You can use git-review to propose a change to the stable branch with:

$> git checkout stable/arno
$> git cherry-pick -x $master_commit_d
$> git review stable/arno

Note: cherry-pick -x option includes 'cherry-picked from …' line in the commit message which is required to avoid Gerrit bug

Failing all that, just ping one of the team and mention that you think the bug/commit is a good candidate.

When cherry-picking a commit, keep the original Change-Id and gerrit will show a separate review for the stable branch while still allowing you to use the Change-Id to see all the reviews associated with it.

Hint: Change-Id line must be in the last paragraph. Conflicts in the backport: add a new paragraph, creating a new Change-Id but you can avoid that by moving conflicts above the paragraph with Change-Id line or removing empty lines to make a single paragraph.

If you want to be notified of these patches you can create a watch on this screen:
https://gerrit.opnfv.org/gerrit/#/settings/projects
click "Watched Projects"

Project Name: All-Projects

Only If: branch:stable/arno

Then check the "Email Notifications - New Changes" checkbox. That will cause gerrit to send an email whenever a matching change is proposed, and better yet, the change shows up in your 'watched changes' list in gerrit.

I don't think we need that at the moment

Octopus team will setup separate pipeline for merge-, build-, verify-jobs.

For Arno release the jobs will be run once per day per installer (Fuel and Foreman) on stable/arno branch. Since this is in addition to the jobs for master branch and jobs have long run time, this might need re-evaluation as we go on.

Each of the 5 projects that contributed to Arno will dedicate some committers which would be in charge of reviewing backports for their project, following the stable branch policy.

Stable branches are less exercised than master branches, and they may get broken by external events.

Therefore a "Arno stable maintenance team" is tasked with specific stable branch support, making sure the branch stays in good shape and remains usable at all times. They monitor periodic jobs failures and enlist the help of others in order to fix the branches in case of breakage. They should also raise flags if for some reason they are blocked and don't receive enough support, in which case early abandon of the branch will be considered.

The Arno stable maintenance team is responsible for the enforcement of the Stable Branch policy. Initially it is composed of a release manager and one committer of each of the participating projects and will have similar organization and rights as OPNFV project teams. It will be granting exceptions for all questionable backports raised by projects, providing backports reviews help everywhere, and educating projects members on the stable branch policy.

The Arno stable maintenance team will propose to TSC to decide on point releases from the stable branch. Preparation of the point release will be described in a second step. It can also propose to TSC to decide to abandon maintenance of the release.

When a new OPNFV version is released, a separate stable branch maintenance team for that project will start. At that time, the earlier maintenance version will go to a phase with less support. Details will be defined later.

Existing committers are greatly encouraged to join the Arno stable maintenance team in order to help with reviewing backports, judging their appropriateness for the stable branch and approving them.

We're really keen to add more folks to the stable Arno stable maint team to help out with reviews.

All you really need is some time and the ability to apply the "safe source of high impact fixes" and "must be fixed on master first" policies. It mostly comes down to having a good sense of the risk vs benefit of applying a backport to the branch.

If you'd like to join the team, you can start by simply [+-]1ing stable branch reviews. It's best if you can add some brief thoughts to your review on why you think the fix is suitable for stable so that we know how you're applying the policy.

Same as in other OPNFV projects, Arno stable maintenance team will manage its organization by committer votes.