Wiki

This page is read only. You can view the source, but not change it. Ask your administrator if you think this is wrong.
**Archived Daily Standup Status**

----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------

**1/27/16 and 1/28/16** 

Countdown to Release: **TBD days**
[[https://etherpad.opnfv.org/p/steps_to_brahmaputra|__**Upcoming deadlines:**__]] under revision with proposed milestones dates until we work out details and report to TSC on 2/2/16 for their vote
**===============================================================================================================**
**Minutes from daily standup** [[http://ircbot.wl.linuxfoundation.org/meetings/opnfv-release/2016/opnfv-release.2016-01-27-14.30.html|1/27/16]]
[[http://ircbot.wl.linuxfoundation.org/meetings/opnfv-release/2016/opnfv-release.2016-01-28-14.30.html|1/28/16]]

__**Top Accomplishments & Important Updates since last report**__ 

.TSC vote in progress to move code freeze to Friday 1/29 and to extend release date (to be defined). Debra and key community members to evaluate remaining work and recommend new release date with solid schedule to achieve that.

.LSOAPI officially notified they are dropping out of Brahmaputra 1/28 see meeting minutes with announcement and justifications: http://ircbot.wl.linuxfoundation.org/meetings/opnfv-meeting/2016/opnfv-meeting.2016-01-28-17.04.html
 

.Fuel team is nearly ready to freeze. Driving hard to meet Friday deadline. 

.ARMBand is busy rebasing to Fuel 8, may not make a Friday code freeze but very close. Driving hard to finish.

.Functest much more stable



 

 **===============================================================================================================**

__**Top Issues**__

**Issue:** We are at high risk of missing release date. Test cycles for each scenario take a significant amount of time. Even if all installers & scenarios are working properly it takes 7 days to run test through them all.  
**Owner:** Debra  
**Status:** TSC voted to delay release rather than trying to phase release. Debra & key community members to identify remaining work and develop reasonably accurate schedule to propose to TSC. 

**Issue:** Functest & Yardstick do not produce stable results. Troubleshooting scenarios. Need assistance from installer & feature project teams to fault isolation and fix.
**Owner:** Morgan Richomme (Functest), Ana Cuna (Yardstick)
**Status:** All Functest scenario blue in Jenkins on Stable branch. Still troubleshooting vPing (and then vIMS). For vPing problem = floating IP not reachable. LF POD1 (apex/odl_l3) broken while running Functest (Neutron errors occured at the beginning of the first tests). One critical issue seen in CI is that after ODL L3 baremetal deploys, and functest executes - the nodes start spamming packets in the network. This does not happen on the ODL L2 or ONOS deployments. The nodes then flood our jenkins slave with 1 million packets per second, and the LF gateway router shuts the port to our VLAN.  Looks like a giant broadcast storm. This is not a POD issue. 

Yardstick:ONOS - ODL scenario troubleshooting in compass continues, for Fuel - will continue on Fuel8
Some docs ready - not all yet

**Issue:** Getting RC0 out
**Owner:** Debra
**Status:**  Installer code freeze extended due to Fuel trying to stabilize with upstream. Fuel should be ready on Friday. Apex, JOID, Compass ready to freeze. ARMBand is struggling to rebase to Fuel 8, may not be ready to freeze and as such may miss this release. Release is planning to freeze on Friday unless TSC disagrees.

**Issue:** Fuel is not ready to code freeze. Upstream Fuel is not stable.
**Owner:** Jonas Bjurel
**Status:** Excellent progress by Fuel team. It is fairly coherent and stable! Now cherry picking to stable.

**Issue:** A large number of projects had code file licensing issues (either missing licenses or non-Apache licenses).
**Owner:** Ray Paik
**Status:**No update 1/27-28 Projects have made good progress on getting patches in to fix the files. Scott is making another license scan to see what if anything is outstanding. Thank you to all who have responded to this request in such a timely manner.

**Issue:** OpenContrail does not deploy
**Owner:** Stuart Mackey
**Status:** JOID & Compass now have working plugins. Getting those into CI/build and testing. Apex still trying to get contrail to work, no update

**Issue:** Scenarios are frozen, but several not working yet, or have not been tried
**Owner:** test teams  
**Status:**  This should be closed now. Validating OpenContrail scenarios in Compass & JOID in preparation for code freeze tomorrow. 
[[https://wiki.opnfv.org/brahmaputra_testing_page#candidate_scenarios]] and [[https://wiki.opnfv.org/brahmaputra_testing_page]]


**Issue:** NFV4KVM – plugin does not work with Fuel, The NFV4KVM does install a working kernel on the compute node. 
**Owner:**Don Dugger/Raghuveer Reddy
**Status:** 1/22 no new update.  Fuel has bigger issues at the moment. We have discovered that OVS appears to require kernel modules that are not there. Working on creating a fix to the Fuel plugin creation scripts that will fix this problem, potentially will have a fix before code freeze, but certainly should be able to solve the problem for an SR1 release.
**===============================================================================================================**
__**Pod Status:**__ 

Troubleshooting Intel Pod 2 for Apex support

[[https://wiki.opnfv.org/pharos_rls_b_labs|Pharos lab status]]

**===============================================================================================================**
**Scenario Status:** Scenarios are frozen. No feature project has reported that their scenarios are missing or unsupported. Candidate scenarios are having Jenkins jobs created for them. Test teams troubleshooting scenarios and test software.
Reviewed scenarios in table and noted those that will not be implemented this release 

Apex: 5 scenarios on Jenkins: 4 work from deploy point of view odl_l2 is about 90 pass/10 fail, odl_l3 worse pass rate Not successfully deployed yet: ocl- working on it

Compass: 4 scenarios on Jenkins: pure openstack, odl_l2, onos, and ocl. From deployment point of view, 3 working, 1 not. Deployed successfully: openstack, odl_l2, onos. Once test has verified plugin is working will add ocl scenarios back in on wiki...

Fuel: Fuel@OPNF in general seems to be in a pretty good shape.
o It is fairly coherent and stable!
o All plugins/features/scenarios (except one which I expect tomorrow) have been merged
o Since Fuel upstream is not released yet, rebasing is needed from time to time
(not really problematic)
o Build system is in good shape
o Deploy system is in good shape
o Building, deploying and smoke-test fine in our internal labs

-          Issues:
o Configuration issues with CI-deploy, probably not a CI-issue but rather a Fuel8   
   configuration issue. We will put all the manpower on this after the freeze, and 
   should be resolved on Monday latest.
o ONOS L3 scenario for external connectivity requires an exemption as it’s requirements
   are not fully compatible with all of the labs (Pharos)! We will work on that next week.
-          Plan for the coming 24 hours:

o Tomorrow we cherry-pick all patches from master (related to B-release) to stable, expect 
   some intermediate flux in build/verify/merge results!
o From this point – strict cherry-picking policies will apply, policed by Jonas.

Joid: 4 scenarios on Jenkins: pure openstack, odl_l2, onos, and ocl. From deployment point of view, 3 working, 1 not activated/attempted yet. Deployed successfully: openstack, odl_l2, onos. Not successfully deployed yet: none. Once ocl scenario is validated in CI will add it back into wiki for this installer

[[https://wiki.opnfv.org/brahmaputra_testing_page#candidate_scenarios]] and [[https://wiki.opnfv.org/brahmaputra_testing_page]]
**===============================================================================================================**
__**Release Test Status:**__


Functest (Morgan Richomme)  All Functest scenario blue in Jenkins on Stable branch. No progress on vPing (and then vIMS) but troubleshooting slots mainly in the evening or at night. For vPing problem = floating IP not reachable. Functest doc update, new patch to be submitted today. LF POD1 (apex/odl_l3) broken while running Functest (Neutron errors occured at the beginning of the first tests)
Test dashboard draft ready http://testresults.opnfv.org/proto/  Should not be too difficult to add new tests/projects and comparison per scenario and/or pod. If you look at the Tempest test case, it gives a good idea of the stability. See Tim' mail yesterday one critical issue I have seen in our CI is that after ODL L3 baremetal deploys, and functest executes - the nodes start spamming packets in the network. This does not happen on the ODL L2 or ONOS deployments. The nodes then flood our jenkins slave with 1 million packets per second, and the LF gateway router shuts the port to our VLAN.  Looks like a giant broadcast storm. This is not a POD issue. As I write this, I noticed it just happened again and I need to go power off the baremetal nodes.

Yardstick:ONOS - reviewing open patches in yardstick repo to prepare for code freeze. ODL scenario troubleshooting in. Some docs ready - not all yet
**===============================================================================================================**
__**Installer Status:**__

**Apex** (Tim Rozet):  Still trying to get contrail to work, running into some openstack version issues where some contrail pieces rely on juno


**JOID** (Narinder Gupta) JOID deployed successfully on all three labs yesterday including the virtual deployment. Unfortunately functest had some issues yesterday which seems to be resolved.

**Compass4nfv** (Weidong Shao) there were some functest and other instability with the code base, and we will need some extra time in getting Compass code stable

**Fuel** (Jonas Bjurel) 
[[https://jira.opnfv.org/secure/Dashboard.jspa?selectPageId=10608|Tickets and backlog jira dashboard]]
1)      

Struggling with an autodeploy issue, hopefully solved now. Testing. Scenarios being rebased

**===============================================================================================================**
__**Project Updates:**__ (as provided)


Scenarios: 
Apex: 5 scenarios on Jenkins: 4 work from deploy point of view
odl_l2 is about 90 pass/10 fail, odl_l3 worse pass rate
Not successfully deployed yet: ocl- working on it

Compass: 4 scenarios on Jenkins: pure openstack, odl_l2, onos, and ocl.
From deployment point of view, 3 working, 1 not.
Deployed successfully: openstack, odl_l2, onos.
Now successfully deployed: ocl (removed from this installer) will add the scenario back in once CI & test are verified

Fuel: 8 scenarios on Jenkins: pure openstack, odl_l2, odl_l3, onos, bgpvpn, ovs, kvm, kvm_ovs. 
From deployment point of view, 5 working, 1 not working, 2 not activated/attempted yet.
Deployed successfully: openstack, odl_l2, odl_l3, onos, bgpvpn
Not successfully deployed yet: ovs.
Not activated/attempted: kvm, kvm_ovs.

Fuel has Fuel7 on stable/brahmaputra branch and Fuel8 uplift work is WIP on master branch.

Joid: 4 scenarios on Jenkins: pure openstack, odl_l2, onos, and ocl.
From deployment point of view, 3 working, 1 not activated/attempted yet.
Deployed successfully: openstack, odl_l2, onos.
Not successfully deployed yet: none.
Now activated/attempted: ocl (removed from this installer). Will add the scenario back in once CI & test are verified

**ARMband:** (Joe Kidder) [My opinion is that we won’t be able to freeze on Friday, as we’re still catching up to Fuel 7 capability, and the Fuel project has moved on to 8.]
Fuel 7:
We are almost complete building the set of arm64 repos needed for a Fuel deployment  and populating a public mirror (these are mainly the mos7 repo and things like that).
We are in the process of doing modifications to cobbler [this is an approximate description] to do the work of Fuel using arm64 rather than x86 (amd64).
Fuel 8:
We have started building/populating the arm64 repos (similar to Fuel 7). 
We have not started the Fuel 8 arm64 porting yet.  This may be more straightforward than with Fuel7, however.
Fuel 6.1 for HA functionality checking and preliminary Functest debug while waiting for Fuel 7/8 deployments to be ready.
We are working on solving an HA bringup issue…some missing virtual interface
  Assessment is that we won’t be able to freeze the porting (at best we’ll be close for Fuel 7, and maybe getting started with Fuel 8 by Friday, but that would be great progress).
  We’re still going to be going full steam ahead.  We want to be ready/available as soon as possible regardless of whether we make the release.


**Availability：**（Qiao Fu) HA project has got all the complete rst files merged and ready for release. 

**Bottlenecks:** (Hongbo Tian) 1, code is freezed in stable/brahmaputra branch
https://gerrit.opnfv.org/gerrit/gitweb?p=bottlenecks.git;a=shortlog;h=refs/heads/stable/brahmaputra
https://git.opnfv.org/cgit/bottlenecks/?h=stable%2Fbrahmaputra
2, bottlenecks/rubbos has been successfully integrated into fuel/compass CI pipeline
https://build.opnfv.org/ci/view/bottlenecks/job/bottlenecks-daily-fuel-rubbos-lf-master/29/
https://build.opnfv.org/ci/view/bottlenecks/job/bottlenecks-daily-fuel-rubbos-lf-master/29/console
https://build.opnfv.org/ci/view/bottlenecks/job/bottlenecks-compass-rubbos-huawei-us-deploy-bare-1-daily-brahmaputra/15/
https://build.opnfv.org/ci/view/bottlenecks/job/bottlenecks-compass-rubbos-huawei-us-deploy-bare-1-daily-brahmaputra/15/console
Test result stored in the DB

 
3,bottlenecks/vstf has been successfully integrated into fuel CI pipeline
https://build.opnfv.org/ci/view/bottlenecks/job/bottlenecks-daily-fuel-vstf-lf-master/32/
https://build.opnfv.org/ci/view/bottlenecks/job/bottlenecks-daily-fuel-vstf-lf-master/32/console
Test result stored in the DB


4, documents
http://artifacts.opnfv.org/bottlenecks/brahmaputra/bottlenecks_brahmaputra_docs/index.html
http://artifacts.opnfv.org/bottlenecks/brahmaputra/bottlenecks_brahmaputra_docs/bottlenecks_brahmaputra_docs.pdf
 

**Copper:** (Bryan Sullivan) completing & integrating documentation for post-install add on

**Doctor** (Ryota, Ildiko) 

**HA**(Fu Qaio) The HA team has successfully merged the docs for Brahmaputra release. For the convenience of documentation work, I merged all the separate section files into one .rst file. I also try to generate pdf versions based on these .rst file and seems everything is OK.
Currently, I upload these files onto our project master repo, and put them into two separate folders: R2_Requirement and R2_Scenarios, each include .rst file and .pdf file for our two release documents, that is the HA requirement doc and HA scenario analysis doc.

**IPv6:** (Bin Hu) -          

**LSOAPI** (Kevin Leuhrs) project withdraws from the OPNFV Brahmaputra release, and plans no further activities. Today’s project meeting minutes record the announcement to terminate the project and summarizes rationale:
 
Project meetings wiki page: https://wiki.opnfv.org/meetings/lsoapi?&#january_27_2016
 
Link to  minutes for today’s meeting:
http://ircbot.wl.linuxfoundation.org/meetings/opnfv-meeting/2016/opnfv-meeting.2016-01-28-17.04.html

**NFV4KVM:** (Don Dugger, Raghuveer Reddy) 
   

**ONOSFW:** (Ashlee Young) 

**OpenvSwitch** (Mark Gray) 


**OPNFVdocs** (Chris Price) 
Issues:
Docs working on central content. Working with projects on project content.  Patches outstanding on many projects.
Awaiting final scenario list to progress to completion. Working with projects on project content.  Patches outstanding on many projects. Unable to code freeze until installers freeze.
ONOSFW reviewing reviewing onosfw docs. Projects are working on docs, some feature projects are waiting for the integration to be complete to document that for the config guide

**OVNO** (Stuart Mackie) Apex still working on patch for plugin

**Parser** 

**Prediction** (Hai Liu) 

**Qtip** (Vikram) 


**SFC** (Brady Johnson) 

**SDNVPN** (Tim Irnich)


**SFQM** (Maryam Tahhan) 

**StorPerf** (Mark Beierle) 

**===============================================================================================================**
**Action Items**__Underlined Text__
**Action:** Reminder needs to be sent to all team that reservation system is set up for CI POD use so as to avoid various teams work impacting others negatively, especially since pods are now running long test runs.
**Owner:**  Debra
**Status:** Done, reminder sent out  Please visit https://wiki.opnfv.org/pharos_rls_b_labs#production_ci_resources_reservation

**Action:**
**Owner:**
**Status:** 

__**Action:**__
**Owner:**
**Status:**

------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------


**1/26/16** 

Countdown to Release: **TBD days**
[[https://etherpad.opnfv.org/p/steps_to_brahmaputra|__**Upcoming deadlines:**__]]
**===============================================================================================================**
**Minutes from daily standup** [[http://ircbot.wl.linuxfoundation.org/meetings/opnfv-release/2016/opnfv-release.2016-01-26-14.30.html|1/26/16]]

__**Top Accomplishments & Important Updates since last report**__ 

.TSC voted to move code freeze to Friday 1/29 and to extend release date (to be defined). Debra and key community members to evaluate remaining work and recommend new release date with solid schedule to achieve that.

 

 **===============================================================================================================**

__**Top Issues**__

**Issue:** We are at high risk of missing release date. Test cycles for each scenario take a significant amount of time. Even if all installers & scenarios are working properly it takes 7 days to run test through them all.  
**Owner:** Debra  
**Status:** TSC voted to delay release rather than trying to phase release. Debra & key community members to identify remaining work and develop reasonably accurate schedule to propose to TSC. 

**Issue:** Functest & Yardstick do not produce stable results. Troubleshooting scenarios. Need assistance from installer & feature project teams to fault isolation and fix.
**Owner:** Morgan Richomme (Functest), Ana Cuna (Yardstick)
**Status:** 

Yardstick:ONOS - successfully run with Compass. ODL scenario - debugging on-going, Yardstick will not run with ODL. Narrowing down subsystems involved. Working on SFC test cases. Working on influxdB+grafana

**Issue:** Getting RC0 out
**Owner:** Debra
**Status:**  Installer code freeze extended due to Fuel trying to stabilize with upstream. Compass also troubleshooting some functest and other instability with the code base, need some extra time in getting Compass code stable. Apex, JOID ready to freeze 

**Issue:** Fuel is not ready to code freeze. Upstream Fuel is not stable.
**Owner:** Jonas Bjurel
**Status:** autodeploy issue, hopefully solved now. Testing. Scenarios being rebased

**Issue:** A large number of projects had code file licensing issues (either missing licenses or non-Apache licenses).
**Owner:** Ray Paik
**Status:**Projects have made good progress on getting patches in to fix the files. Scott is making another license scan to see what if anything is outstanding. Thank you to all who have responded to this request in such a timely manner.

**Issue:** OpenContrail does not deploy
**Owner:** Stuart Mackey
**Status:** Apex still trying to get contrail to work, running into some openstack version issues where some contrail pieces rely on juno

**Issue:** Scenarios are frozen, but several not working yet, or have not been tried
**Owner:** test teams  
**Status:**  Next step is creating Jenkins jobs and testing each scenario. In process with test teams leading. [[https://wiki.opnfv.org/brahmaputra_testing_page#candidate_scenarios]] and [[https://wiki.opnfv.org/brahmaputra_testing_page]]


**Issue:** NFV4KVM – plugin does not work with Fuel, The NFV4KVM does install a working kernel on the compute node. 
**Owner:**Don Dugger/Raghuveer Reddy
**Status:** 1/22 no new update.  Fuel has bigger issues at the moment. We have discovered that OVS appears to require kernel modules that are not there. Working on creating a fix to the Fuel plugin creation scripts that will fix this problem, potentially will have a fix before code freeze, but certainly should be able to solve the problem for an SR1 release.
**===============================================================================================================**
__**Pod Status:**__ 

All pods up and running

[[https://wiki.opnfv.org/pharos_rls_b_labs|Pharos lab status]]

**===============================================================================================================**
**Scenario Status:** Scenarios are frozen. No feature project has reported that their scenarios are missing or unsupported. Candidate scenarios are having Jenkins jobs created for them. Test teams troubleshooting scenarios and test software.
Reviewed scenarios in table and noted those that will not be implemented this release 

Apex: 5 scenarios on Jenkins: 4 work from deploy point of view odl_l2 is about 90 pass/10 fail, odl_l3 worse pass rate Not successfully deployed yet: ocl- working on it

Compass: 4 scenarios on Jenkins: pure openstack, odl_l2, onos, and ocl. From deployment point of view, 3 working, 1 not. Deployed successfully: openstack, odl_l2, onos. Not successfully deployed yet: ocl (removed from this installer)

Fuel: 8 scenarios on Jenkins: pure openstack, odl_l2, odl_l3, onos, bgpvpn, ovs, kvm, kvm_ovs. From deployment point of view, 5 working, 1 not working, 2 not activated/attempted yet. Deployed successfully: openstack, odl_l2, odl_l3, onos, bgpvpn Not successfully deployed yet: ovs. Not activated/attempted: kvm, kvm_ovs.

Fuel has Fuel7 on stable/brahmaputra branch and Fuel8 uplift work is WIP on master branch.

Joid: 4 scenarios on Jenkins: pure openstack, odl_l2, onos, and ocl. From deployment point of view, 3 working, 1 not activated/attempted yet. Deployed successfully: openstack, odl_l2, onos. Not successfully deployed yet: none. Not activated/attempted: ocl (removed from this installer)



[[https://wiki.opnfv.org/brahmaputra_testing_page#candidate_scenarios]] and [[https://wiki.opnfv.org/brahmaputra_testing_page]]
**===============================================================================================================**
__**Release Test Status:**__


Functest (Morgan Richomme)  Current situation

Functest regression yesterday night due to sdnvpn integration, fixed now
.vPing/SSH Ok on manual tests with compass, not working with all scenarios yet
we defined 2 vPING: vPing SSH and vPing userdata/cloudinit
https://wiki.opnfv.org/vping_brahmaputra_page
vPing/SSH should be OK now on compass (wait for next run). Table to be updated for apex as well I think
http://testresults.opnfv.org/testapi/dashboard?project=functest&case=vPing&installer=apex and you can change the installer. wait for the test dashboard of LF...

Yardstick (Ana Cunha)  first successful run with joid yesterday (os scenario), fails on odl scenario
On-going code writing for a patch for workaround for odl scenario - remains to be tested on the labs
On-going work for grafana dashboard and documentation
Started to document results for ha

**===============================================================================================================**
__**Installer Status:**__

**Apex** (Tim Rozet):  Still trying to get contrail to work, running into some openstack version issues where some contrail pieces rely on juno


**JOID** (Narinder Gupta) JOID deployed successfully on all three labs yesterday including the virtual deployment. Unfortunately functest had some issues yesterday which seems to be resolved today.

**Compass4nfv** (Weidong Shao) there were some functest and other instability with the code base, and we will need some extra time in getting Compass code stable

**Fuel** (Jonas Bjurel) 
[[https://jira.opnfv.org/secure/Dashboard.jspa?selectPageId=10608|Tickets and backlog jira dashboard]]
1)      

Struggling with an autodeploy issue, hopefully solved now. Testing. Scenarios being rebased

**===============================================================================================================**
__**Project Updates:**__ (as provided)


Scenario view status: Reviewed scenarios in table and noted those that will not be implemented this release
Apex: 5 scenarios on Jenkins: 4 work from deploy point of view
odl_l2 is about 90 pass/10 fail, odl_l3 worse pass rate
Not successfully deployed yet: ocl- working on it

Compass: 4 scenarios on Jenkins: pure openstack, odl_l2, onos, and ocl.
From deployment point of view, 3 working, 1 not.
Deployed successfully: openstack, odl_l2, onos.
Not successfully deployed yet: ocl (removed from this installer)

Fuel: 8 scenarios on Jenkins: pure openstack, odl_l2, odl_l3, onos, bgpvpn, ovs, kvm, kvm_ovs. 
From deployment point of view, 5 working, 1 not working, 2 not activated/attempted yet.
Deployed successfully: openstack, odl_l2, odl_l3, onos, bgpvpn
Not successfully deployed yet: ovs.
Not activated/attempted: kvm, kvm_ovs.

Fuel has Fuel7 on stable/brahmaputra branch and Fuel8 uplift work is WIP on master branch.

Joid: 4 scenarios on Jenkins: pure openstack, odl_l2, onos, and ocl.
From deployment point of view, 3 working, 1 not activated/attempted yet.
Deployed successfully: openstack, odl_l2, onos.
Not successfully deployed yet: none.
Not activated/attempted: ocl (removed from this installer)

**ARMband:**

**Availability：**（Qiao Fu) HA project has got all the rst file merged and ready for release. Will upload a full set of rst file for the two release doc later today.

**Bottlenecks:** (Hongbo Tian) 

**Copper:** (Bryan Sullivan) at risk

**Doctor** (Ryota, Ildiko) 

**IPv6:** (Bin Hu) -          

**LSOAPI** (Kevin Leuhrs) at risk

**NFV4KVM:** (Don Dugger, Raghuveer Reddy) 
   

**ONOSFW:** (Ashlee Young) 

**OpenvSwitch** (Mark Gray) 


**OPNFVdocs** (Chris Price) 
Issues:
Docs working on central content. Working with projects on project content.  Patches outstanding on many projects.
Awaiting final scenario list to progress to completion. Working with projects on project content.  Patches outstanding on many projects. Unable to code freeze until installers freeze.
ONOSFW reviewing reviewing onosfw docs. Projects are working on docs, some feature projects are waiting for the integration to be complete to document that for the config guide

**OVNO** (Stuart Mackie) Apex still working on patch for plugin

**Parser** 

**Prediction** (Hai Liu) 

**Qtip** (Vikram) 


**SFC** (Brady Johnson) 

**SDNVPN** (Tim Irnich)


**SFQM** (Maryam Tahhan) 

**StorPerf** (Mark Beierle) 

**===============================================================================================================**
**Action Items**__Underlined Text__
**Action:** 
**Owner:**  
**Status:** 

**Action:**
**Owner:**
**Status:** 

__**Action:**__
**Owner:**
**Status:**

------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------

**1/25/16** 

Countdown to Release: **6 days**
[[https://etherpad.opnfv.org/p/steps_to_brahmaputra|__**Upcoming deadlines:**__]]
**===============================================================================================================**
**Minutes from daily standup** [[http://ircbot.wl.linuxfoundation.org/meetings/opnfv-release/2016/opnfv-release.2016-01-25-14.30.html|1/25/16]]

__**Top Accomplishments & Important Updates since last report**__ 

.Fuel made great strides over the weekend. ONOS scenarios working. Still unable to freeze. 

.Yardstick successfully ran ONOS scenario with Compass.

.Functest now relatively stable from CI perspective. ONOS scenarios working. 

 **===============================================================================================================**

__**Top Issues**__

**Issue:** We are at high risk of missing release date. Test cycles for each scenario take a significant amount of time. Even if all installers & scenarios are working properly it takes 7 days to run test through them all.  
**Owner:** Debra  
**Status:** Additional options being proposed: 1. Release with what we have and move everything else to C Release  2. Delay release for up to 3 weeks 3. staggered/phased release of scenario support as a potential. Phase 1: Release what we can on Feb 2; Phase 2: Release what is ready 1-2 weeks later, Phase 3: final release 1-2 weeks after phase 2. These have been brought up to the TSC via email for consideration next TSC meeting. 

**Issue:** Functest & Yardstick do not produce stable results. Troubleshooting scenarios. Need assistance from installer & feature project teams to fault isolation and fix.
**Owner:** Morgan Richomme (Functest), Ana Cuna (Yardstick)
**Status:** Functest now relatively stable from CI perspective (except just joid). Still work on the different vPing (and then vIMS) testcases (that need ssh access from the VMs). Onos OK. 

Yardstick:ONOS - successfully run with Compass. ODL scenario - debugging on-going, Yardstick will not run with ODL. Narrowing down subsystems involved. Working on SFC test cases. Working on influxdB+grafana

**Issue:** Getting RC0 out
**Owner:** Debra
**Status:**  Installer code freeze extended due to Fuel trying to stabilize with upstream. Compass also troubleshooting some functest and other instability with the code base, need some extra time in getting Compass code stable. Apex, JOID ready to freeze 

**Issue:** Fuel is not ready to code freeze. Upstream Fuel is not stable.
**Owner:** Jonas Bjurel
**Status:** Requesting code freeze delay until 1/29 and suggesting release slip 3 weeks

**Issue:** A large number of projects had code file licensing issues (either missing licenses or non-Apache licenses).
**Owner:** Ray Paik
**Status:**Projects have made good progress on getting patches in to fix the files. Scott is making another license scan to see what if anything is outstanding. Thank you to all who have responded to this request in such a timely manner.

**Issue:** OpenContrail does not deploy
**Owner:** Stuart Mackey
**Status:** Apex still trying to get contrail to work, running into some openstack version issues where some contrail pieces rely on juno

**Issue:** Scenarios are frozen, but several not working yet, or have not been tried
**Owner:** test teams  
**Status:**  Next step is creating Jenkins jobs and testing each scenario. In process with test teams leading. [[https://wiki.opnfv.org/brahmaputra_testing_page#candidate_scenarios]] and [[https://wiki.opnfv.org/brahmaputra_testing_page]]


**Issue:** NFV4KVM – plugin does not work with Fuel, The NFV4KVM does install a working kernel on the compute node. 
**Owner:**Don Dugger/Raghuveer Reddy
**Status:** 1/22 no new update.  Fuel has bigger issues at the moment. We have discovered that OVS appears to require kernel modules that are not there. Working on creating a fix to the Fuel plugin creation scripts that will fix this problem, potentially will have a fix before code freeze, but certainly should be able to solve the problem for an SR1 release.
**===============================================================================================================**
__**Pod Status:**__ 

All pods up and running

[[https://wiki.opnfv.org/pharos_rls_b_labs|Pharos lab status]]

**===============================================================================================================**
**Scenario Status:** Scenarios are frozen. No feature project has reported that their scenarios are missing or unsupported. Candidate scenarios are having Jenkins jobs created for them. Test teams troubleshooting scenarios and test software.


[[https://wiki.opnfv.org/brahmaputra_testing_page#candidate_scenarios]] and [[https://wiki.opnfv.org/brahmaputra_testing_page]]
**===============================================================================================================**
__**Release Test Status:**__


Functest (Morgan Richomme)  Current situation
*****************
Functest Functest tests more stable, Some tests run in CI over the week end, Functest now relatively stable from CI perspective (except just joid I had to cancel). Still work on the different vPing (and then vIMS) testcases (that need ssh access from the VMs). Onos OK. ODL, maybe an update to be done, but we are integrating upstream suite, and it is not fully clear that the suite has been adapted to new version of ODL. 
https://wiki.opnfv.org/doku.php?id=vPing_brahmaputra_page
https://wiki.opnfv.org/doku.php?id=vIMS_brahmaputra_page
Test DB migrated this morning (9 UTC) http://testresults.opnfv.org/testapi/pods
Still work ahead to tune some cases, and collect the error causes

Yardstick (Ana Cunha)  ONOS first succesfull run on Compass, continue troubleshooting ODL scenarios


**===============================================================================================================**
__**Installer Status:**__

**Apex** (Tim Rozet):  Still trying to get contrail to work, running into some openstack version issues where some contrail pieces rely on juno


**JOID** (Narinder Gupta) JOID we are ready for code freeze as last pending issue related to Orange pod has been fixed

**Compass4nfv** (Weidong Shao) there were some functest and other instability with the code base, and we will need some extra time in getting Compass code stable

**Fuel** (Jonas Bjurel) 
[[https://jira.opnfv.org/secure/Dashboard.jspa?selectPageId=10608|Tickets and backlog jira dashboard]]
1)      Build

The building issues have been resolved to the point that it builds flawlessly in the
OPNFV CI engine. Daily masters are produced every night since Friday. Work ongoing
with getting it to build inside Ericsson corporate network. Stefan probably
nailed it as it seems to be an issue with the 3.13.0-66 kernel.

2)      Deployment. 
I have successfully deployed the daily master image from Friday CI build: 
http://artifacts.opnfv.org/fuel/opnfv-2016-01-22_16-05-16.iso
It works more or less flawlessly, a few minor comments:
- The ssh service is not started as early as it used to do, it is not started when Fuelmenu
  pops up which will be an issue for auto deploy. # service sshd start fixes it
- Fuelmenu will sometimes tell you that the URLs for the bootimage cannot be found,
  if you make sure you can reach url resources eg. Through open a consol from fuelmenu
  and type nslookup www.google.com – just don’t care, select “save & continue” and the
  “quit without saving” you should be good.
- Selecting centos boot image didn’t work for me

3)      Early this week we need to prioritize getting autodeploy to work.

 
But to my mind, this seems to be stable enough for plugin work. Please start commit rebased plugins to
Master.

-          As I have indicated to TSC (Chris and Frank), my judgement is that we need another 3 week after Feb 2 to complete the things and have them in a good shape. The other option is to release with Fuel7/Kilo which is quite stable and a good veichle for developing stabilizing the scenarios  and the test pipe-line.

-          Issue with Jenkins server caused delays on Sunday but were resolved and builds are working again.

**===============================================================================================================**
__**Project Updates:**__ (as provided)

**ARMband:**

**Bottlenecks:** (Hongbo Tian) 

**Copper:** (Bryan Sullivan) at risk

**Doctor** (Ryota, Ildiko) Apex patch for AODH is merged. Doctor will make the release.

**IPv6:** (Bin Hu) -          

**LSOAPI** (Kevin Leuhrs) at risk

**NFV4KVM:** (Don Dugger, Raghuveer Reddy) 
   

**ONOSFW:** (Ashlee Young) 

**OpenvSwitch** (Mark Gray) 


**OPNFVdocs** (Chris Price) 
Issues:
Docs working on central content. Working with projects on project content.  Patches outstanding on many projects.
Awaiting final scenario list to progress to completion. Working with projects on project content.  Patches outstanding on many projects. Unable to code freeze until installers freeze.
ONOSFW reviewing reviewing onosfw docs. Projects are working on docs, some feature projects are waiting for the integration to be complete to document that for the config guide

**OVNO** (Stuart Mackie) Apex 

**Parser** 

**Prediction** (Hai Liu) 

**Qtip** (Vikram) 


**SFC** (Brady Johnson) Apex patch is merged, SFC scenario is supported in the release

**SDNVPN** (Tim Irnich)


**SFQM** (Maryam Tahhan) 

**StorPerf** (Mark Beierle) 

**===============================================================================================================**
**Action Items**__Underlined Text__
**Action:** 
**Owner:**  
**Status:** 

**Action:**
**Owner:**
**Status:** 

__**Action:**__
**Owner:**
**Status:**

------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
**1/22/16** 

Countdown to Release: **11 days**
[[https://etherpad.opnfv.org/p/steps_to_brahmaputra|__**Upcoming deadlines:**__]]
**===============================================================================================================**
**Minutes from daily standup** [[http://ircbot.wl.linuxfoundation.org/meetings/opnfv-release/2016/opnfv-release.2016-01-22-14.30.html|1/22/16]]

__**Top Accomplishments & Important Updates since last report**__ 

.Fuel unable to freeze due to upstream instability, requests delay of freeze until 1/29 and suggests 3 week delay to release or else fall back to previous upstream release of Fuel which is quite stable but that also requires fallback to OpenStack Kilo.

.BREAKING NEWS: Successful run of bgpvpn noha scenario with Fuel with stable/Brahmaputra on LF-POD2

.Apex got SFC and Aodh (Doctor) support merged

.Functest stabilizing, successful runs 11/11 with ONOS, Apex, and Compass

.Yardstick had first successful run with ONOS and Compass
  
 **===============================================================================================================**

__**Top Issues**__

**Issue:** We are at high risk of missing release date. Test cycles for each scenario take a significant amount of time. Even if all installers & scenarios are working properly it takes 7 days to run test through them all.  
**Owner:** Debra  
**Status:** Additional options being proposed: 1. Release with what we have and move everything else to C Release  2. Delay release for up to 3 weeks 3. staggered/phased release of scenario support as a potential. Phase 1: Release what we can on Feb 2; Phase 2: Release what is ready 1-2 weeks later, Phase 3: final release 1-2 weeks after phase 2. These have been brought up to the TSC via email for consideration next TSC meeting. 

**Issue:** Functest & Yardstick do not produce stable results. Troubleshooting scenarios. Need assistance from installer & feature project teams to fault isolation and fix.
**Owner:** Morgan Richomme (Functest), Ana Cuna (Yardstick)
**Status:** Functest tests more stable, several successfull run done since 24h, ONOS scenarios first successfull runs with Apex and Compass: 11/11  runs. Some concerns on the config of some PoDs, we were able to run the same tempest suite on 3 different PoDs with the same scenario (very inconsistent results)

Yardstick: Continuing to troubleshoot troubleshooting ODL and ONOS scenarios. Not possible to run yardstick on ODL, ONOS yet - debugging Yardstick software.
.Integration with SFC started
.Results documentation for on-demand test cases started

**Issue:** Functest & Yardstick have difficulty troubleshooting issues when underlying configurations are changed as installers are not frozen. They start troubleshooting one issue and then when they come back the next day the Installer code has been updated and they have to start from scratch. 
**Owner:** All teams
**Status:** Fatih started a wiki for reservation of PODs so that teams are not stomping over each others work. Better communication & coordination is needed with regard to lab use considering the time it takes to set up and troubleshoot. All teams please adhere to the requested schedule or if the proposed schedule doesn't work for your team then please contact Fatih to work out the issues.

**Issue:** Getting RC0 out
**Owner:** Debra
**Status:**  Installer code freeze extended again for 1 day due to Fuel trying to stabilize with upstream. 1/22 installer/documents freeze date delayed. Compass: There were some functest and other instability with the code base, and we will need some extra time in getting Compass code stable. Apex, JOID ready to freeze 

**Issue:** Fuel is not ready to code freeze. Upstream Fuel is not stable.
**Owner:** Jonas Bjurel
**Status:** Requesting code freeze delay until 1/29 and suggesting release slip 3 weeks

**Issue:** OpenContrail does not deploy
**Owner:** Stuart Mackey
**Status:** Components of the needed patch for OpenContrail/Liberty are identifed and being scripted, This is one of the patches Apex is waiting for before freezing their code. They could freeze today if Fuel were ready but that would drop OpenContrail support.

**Issue:** Scenarios are frozen, but need to be planned out or mapped to lab/test cycle
**Owner:** test teams  
**Status:**  Next step is creating Jenkins jobs and testing each scenario. In process with test teams leading. [[https://wiki.opnfv.org/brahmaputra_testing_page#candidate_scenarios]] and [[https://wiki.opnfv.org/brahmaputra_testing_page]]

**Issue:** Doctor install requirements - Aodh as critical component
**Owner:** Ildiko, Ryota
**Status:** Apex patch for AODH is working, getting it integrated since freeze is delayed. Doctor will make the release.

**Issue:** NFV4KVM – plugin does not work with Fuel, The NFV4KVM does install a working kernel on the compute node. 
**Owner:**Don Dugger/Raghuveer Reddy
**Status:** 1/22 no new update.  Fuel has bigger issues at the moment. We have discovered that OVS appears to require kernel modules that are not there. Working on creating a fix to the Fuel plugin creation scripts that will fix this problem, potentially will have a fix before code freeze, but certainly should be able to solve the problem for an SR1 release.
**===============================================================================================================**
__**Pod Status:**__ 

All pods up and running

[[https://wiki.opnfv.org/pharos_rls_b_labs|Pharos lab status]]

**===============================================================================================================**
**Scenario Status:** Scenarios are frozen. No feature project has reported that their scenarios are missing or unsupported. Candidate scenarios are having Jenkins jobs created for them. Test teams troubleshooting scenarios and test software.


[[https://wiki.opnfv.org/brahmaputra_testing_page#candidate_scenarios]] and [[https://wiki.opnfv.org/brahmaputra_testing_page]]
**===============================================================================================================**
__**Release Test Status:**__


Functest (Morgan Richomme)  Current situation
*****************
Functest Functest tests more stable, several successfull run done since 24h, all tables updated.
ONOS first successfull run on apex and compass 11/11
Some concerns on the config of some PoDs, we were able to run the same tempest suite on 3 different PoDs with the same scenario. We got very different results..on Orange Pod 91% OK, On Intel POD5 35%, Intel POD5 14ù, Intel POD6 34%
Here it is a question of network config of the POD. Other concern it looks like ODL suite may be provide different results according to ODL version...looks like all the installers do not use the same version, however all have verified they use Lithium 3 https://wiki.opnfv.org/tempest_brahmaputra_page https://wiki.opnfv.org/onos_brahmaputra_page 
Test DB ready for migration, mail sent for transfer to LF on Monday

Yardstick (Ana Cunha)  ONOS first succesfull run on Compass, continue troubleshooting ODL scenarios


**===============================================================================================================**
__**Installer Status:**__

**Apex** (Tim Rozet):  SFC and Aodh (Doctor) support merged


**JOID** (Narinder Gupta) JOID we are ready for code freeze as last pending issue related to Orange pod has been fixed

**Compass4nfv** (Weidong Shao) there were some functest and other instability with the code base, and we will need some extra time in getting Compass code stable

**Fuel** (Jonas Bjurel) 
[[https://jira.opnfv.org/secure/Dashboard.jspa?selectPageId=10608|Tickets and backlog jira dashboard]]
-          Upstream continues to give us surprises, but stability has improve significantly the last few days.

-          Some good news. Fatih ran bgpvpn noha scenario with Fuel with stable/Brahmaputra on LF-POD2 and it deployed like a charm! See: https://build.opnfv.org/ci/job/fuel-deploy-opnfv-jump-2-daily-brahmaputra/43/console. Next stage in advancement is to run the bgpvpn ha scenario (it is already developed and awaiting testing!). Super great work from Nikolas, Tim and Prem (and Fatih of course)

-          We now believe that we could code-freeze Fuel@OPNFV Friday next week, with a few more days needed for the features/plug-ins.

-          As I have indicated to TSC (Chris and Frank), my judgement is that we need another 3 week after Feb 2 to complete the things and have them in a good shape. The other option is to release with Fuel7/Kilo which is quite stable and a good veichle for developing stabilizing the scenarios  and the test pipe-line.

**===============================================================================================================**
__**Project Updates:**__ (as provided)

**ARMband:**

**Bottlenecks:** (Hongbo Tian) 

**Copper:** (Bryan Sullivan) at risk

**Doctor** (Ryota, Ildiko) Apex patch for AODH is merged. Doctor will make the release.

**IPv6:** (Bin Hu) -          

**LSOAPI** (Kevin Leuhrs) at risk

**NFV4KVM:** (Don Dugger, Raghuveer Reddy) 
   

**ONOSFW:** (Ashlee Young) 

**OpenvSwitch** (Mark Gray) 


**OPNFVdocs** (Chris Price) 
Issues:
The code freeze date will not be met given the current state of Brahmaputra documentation.

**OVNO** (Stuart Mackie) Apex getting close to supporting OpenContrail, continuing to work on it since Fuel has delayed code freeze. Need to integrate new neutron plugin for Liberty.

**Parser** 

**Prediction** (Hai Liu) 

**Qtip** (Vikram) 


**SFC** (Brady Johnson) Apex patch is merged, SFC scenario is supported in the release

**SDNVPN** (Tim Irnich)


**SFQM** (Maryam Tahhan) 

**StorPerf** (Mark Beierle) 

**===============================================================================================================**
**Action Items**__Underlined Text__
**Action:** 
**Owner:**  
**Status:** 

**Action:**
**Owner:**
**Status:** 

__**Action:**__
**Owner:**
**Status:**

------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------

**1/21/16** 

Countdown to Release: **12 days**
[[https://etherpad.opnfv.org/p/steps_to_brahmaputra|__**Upcoming deadlines:**__]]
**===============================================================================================================**
**Minutes from daily standup** [[http://ircbot.wl.linuxfoundation.org/meetings/opnfv-release/2016/opnfv-release.2016-01-21-14.30.html|1/21/16]]

__**Top Accomplishments & Important Updates since last report**__ 

.Fuel unable to freeze today. Will be better prepared to estimate additional time needed by tomorrow
  
.Apex will be ready to freeze this evening

.JOID and Compass ready to freeze

.Apex very close to having SFC and AODH added, so sfc scenario will be working, and Doctor support will be there. Great work by the team.

.
.
  
 **===============================================================================================================**

__**Top Issues**__

**Issue:** We are at high risk of missing release date. Test cycles for each scenario take a significant amount of time. Even if all scenarios are working properly it takes 7 days to run test through them all.  
**Owner:** Debra  
**Status:** Proposing we look at staggered/phased release of scenario support as a potential. If we prioritize scenarios, let one or two highest priority scenarios out ASAP, preferably Feb 2; we could then take additional time to test more scenarios and release them 2 weeks later, and if necessary release more in another 2 weeks. This could potentially allow additional time for test projects and allow focus on smaller set of issues. Would need TSC discussion/decision on this approach. Test team are not sure if this will relieve their issues or not. It depends on the nature of the issues they identify with their tests suites. Debra to bring this idea up for consideration by TSC. We should know better very soon if this is a viable option to relieve test efforts.

**Issue:** Functest & Yardstick do not produce stable results. Troubleshooting scenarios. Need assistance from installer & feature project teams to fault isolation and fix.
**Owner:** Morgan Richomme (Functest), Ana Cuna (Yardstick)
**Status:** Functest running again, back to Tuesday's status of investigating lack stability in test results.  

Yardstick: Continuing to troubleshoot troubleshooting ODL and ONOS scenarios. Not possible to run yardstick on ODL, ONOS yet - debugging Yardstick software.
.Integration with SFC started
.Results documentation for on-demand test cases started

**Issue:** Functest & Yardstick have difficulty troubleshooting issues when underlying configurations are changed as installers are not frozen. They start troubleshooting one issue and then when they come back the next day the Installer code has been updated and they have to start from scratch. 
**Owner:** All teams
**Status:** Fatih started a wiki for reservation of PODs so that teams are not stomping over each others work. Better communication & coordination is needed with regard to lab use considering the time it takes to set up and troubleshoot. All teams please adhere to the requested schedule or if the proposed schedule doesn't work for your team then please contact Fatih to work out the issues.

**Issue:** Getting RC0 out
**Owner:** Debra
**Status:**  Installer code freeze extended again for 1 day due to Fuel trying to stabilize with upstream. New installer/documents freeze date 1/22. Apex, JOID & Compass all ready to freeze 

**Issue:** Fuel is not ready to code freeze. Upstream pip issue has a workaround but waiting merge iupstream.
**Owner:** Jonas Bjurel
**Status:** Upstream pip issue has a workaround but waiting merge in upstream.

**Issue:** OpenContrail does not deploy
**Owner:** Stuart Mackey
**Status:** Components of the needed patch for OpenContrail/Liberty are identifed and being scripted, This is one of the patches Apex is waiting for before freezing this evening.

**Issue:** Scenarios are frozen, but need to be planned out or mapped to lab/test cycle
**Owner:** test teams  
**Status:**  Next step is creating Jenkins jobs and testing each scenario. In process with test teams leading. [[https://wiki.opnfv.org/brahmaputra_testing_page#candidate_scenarios]] and [[https://wiki.opnfv.org/brahmaputra_testing_page]]

**Issue:** Doctor install requirements - Aodh as critical component
**Owner:** Ildiko, Ryota
**Status:** Apex patch for AODH is working, getting it integrated. Doctor will make the release.

**Issue:** NFV4KVM – plugin does not work with Fuel, The NFV4KVM does install a working kernel on the compute node. 
**Owner:**Don Dugger/Raghuveer Reddy
**Status:** 1/20 no new update.  We have discovered that OVS appears to require kernel modules that are not there. Working on creating a fix to the Fuel plugin creation scripts that will fix this problem, potentially will have a fix before code freeze, but certainly should be able to solve the problem for an SR1 release.
**===============================================================================================================**
__**Pod Status:**__ 

All pods up and running

[[https://wiki.opnfv.org/pharos_rls_b_labs|Pharos lab status]]

**===============================================================================================================**
**Scenario Status:** Scenarios are frozen. No feature project has reported that their scenarios are missing or unsupported. Candidate scenarios are having Jenkins jobs created for them. Test teams troubleshooting scenarios and test software.


[[https://wiki.opnfv.org/brahmaputra_testing_page#candidate_scenarios]] and [[https://wiki.opnfv.org/brahmaputra_testing_page]]
**===============================================================================================================**
__**Release Test Status:**__


Functest (Morgan Richomme)  Current situation
*****************
Functest running again, back to Tuesday's status of investigating lack stability in test results.  Rally will create a new version on Monday, but we applied the temporary patch in order to work, we force the use of a pip version < 8. New vPing seems OK (tested on apex, fuel and joid). Integration functest/promise initiated. Results are still not stable. We can see that the integration of ODL lead to a significant degradation of tempest results (which we may understand) but it is not the same on the different installers. So we have to understand the errors. Without controller almost 100% OK, with other scenario different figures and different errors. Looking into potential of a scenario config issue with Fuel for ODL-L3.  https://wiki.opnfv.org/tempest_brahmaputra_page

Yardstick (Ana Cunha)  troubleshooting Joid, Compass, Fuel and Apex, working on integrated test result db.
Requested help with SSH issues.


**===============================================================================================================**
__**Installer Status:**__

**Apex** (Tim Rozet): Integrating patches and plan to freeze this evening 


**JOID** (Narinder Gupta) ready to freeze, working on docs

**Compass4nfv** (Weidong Shao) ready to freeze

**Fuel** (Jonas Bjurel) 
[[https://jira.opnfv.org/secure/Dashboard.jspa?selectPageId=10608|Tickets and backlog jira dashboard]]
Not read to freeze, working with upstream problems. Upstream Fuel not yet entirely stable, were working on it and hope to have better answers tomorrow

**===============================================================================================================**
__**Project Updates:**__ (as provided)

**ARMband:**

**Bottlenecks:** (Hongbo Tian) 

**Copper:** (Bryan Sullivan) at risk

**Doctor** (Ryota, Ildiko) Apex patch for AODH is working, getting it integrated. Doctor will make the release.

**IPv6:** (Bin Hu) -          

**LSOAPI** (Kevin Leuhrs) at risk

**NFV4KVM:** (Don Dugger, Raghuveer Reddy) 
   

**ONOSFW:** (Ashlee Young) 

**OpenvSwitch** (Mark Gray) 


**OPNFVdocs** (ChrisPriceAB:) 
Issues:
The code freeze date will not be met given the current state of Brahmaputra documentation.

**Parser** 

**Prediction** (Hai Liu) 

**Qtip** (Vikram) 


**SFC** (Brady Johnson) Apex integrating a patch that will have this scenario running

**SDNVPN** (Tim Irnich)


**SFQM** (Maryam Tahhan) 

**StorPerf** (Mark Beierle) 

**===============================================================================================================**
**Action Items**__Underlined Text__
**Action:** 
**Owner:**  
**Status:** 

**Action:**
**Owner:**
**Status:** 

__**Action:**__
**Owner:**
**Status:**

------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------**1/20/16** 

Countdown to Release: **13 days**
[[https://etherpad.opnfv.org/p/steps_to_brahmaputra|__**Upcoming deadlines:**__]]
**===============================================================================================================**
**Minutes from daily standup** [[http://ircbot.wl.linuxfoundation.org/meetings/opnfv-release/2016/opnfv-release.2016-01-20-14.30.html|1/20/16]]

__**Top Accomplishments & Important Updates since last report**__ 

.Unless there is significant progress by 1/21, we are at a tipping point where we need to consider the high likelihood of schedule slip. We cannot release without a basic level of Functest and Yardstick. If these were working perfectly today it would still require a minimum of 7 days to validate the test scenarios to provide a baseline for beginning the test cycle, however, each scenario is requiring significant time to troubleshoot & fix. Alternatives to look at may include prioritize & focus on smaller set of scenarios, and follow up with others in a two or three phase release.
  
.JOID reached RC0

.LSOAPI: the Cablelabs team is following up on last action items. LSOAPI is running in a local lab, they will continue to pursue running in a Pharos lab. They will also work on documentation. They are using a post install deployment option. They are at high risk for missing this release.
  
.Installer/documentation code freeze extended to 1/21. This potentially delays test work although they have started with those installers that are able to lockdown. (as of today that is Apex, JOID, Compass4nfv)
  
  
 **===============================================================================================================**

__**Top Issues**__

**Issue:** Functest & Yardstick do not produce stable results. Troubleshooting scenarios. Need assistance from installer & feature project teams to fault isolation and fix.
**Owner:** Morgan Richomme (Functest), Ana Cuna (Yardstick)
**Status:** Functest all tests failed due to problem with pip (we should not the only ones impacted...). Installation of rally is not possible...but it is not Rally's fault. https://en.wikipedia.org/wiki/Pip_(package_manager)
Before this, we updated the different tables https://wiki.opnfv.org/vping_brahmaputra_page

Continuing to troubleshoot troubleshooting ODL and ONOS scenarios. With Yardstick, cleaning up one issue uncovered another. Functest tests suite runnable from CI perspective, heterogeneous and not reproducible results (due to lack of stability). 


**Issue:** Getting RC0 out
**Owner:** Debra
**Status:**  Installer code freeze extended for 2 days due to Pods being down this past weekend. New installer/documents freeze date 1/21. Apex & JOID scenario test/troubleshooting in progress; Fuel validation of upstream integration in progress. Compass is ready to freeze aside from test trying to run and then troubleshooting scenarios (like Apex & JOID).

**Issue:** Fuel is not ready to code freeze. Pods down over weekend prevented testing/progress with validation of integrated upstream code. Ran into issue in OpenStack PIP. OpenStack community is looking into this.
**Owner:** Jonas Bjurel
**Status:** see above issue status getting RC0 out

**Issue:** OpenContrail does not deploy
**Owner:** Stuart Mackey
**Status:** no update for 1/20 Compass got vrouter and GUI up, Apex has a build and will test, JOID waiting for a patch

**Issue:** Scenarios are not fully known, planned out or mapped to lab/test cycle
**Owner:** Debra  
**Status:**  Next step is creating Jenkins jobs and testing each scenario. In process with test teams leading. Assistance needed from both Installer teams and Feature teams to troubleshoot & close issues. Closed. Scenarios are frozen. No feature projects which are able to deploy with installers have reported that their scenarios are not supported. [[https://wiki.opnfv.org/brahmaputra_testing_page#candidate_scenarios]] and [[https://wiki.opnfv.org/brahmaputra_testing_page]]

**Issue:** Doctor install requirements - Aodh as critical component
**Owner:** Ildiko, Ryota
**Status:** 1/20 Still at high risk. Apex has a patch up but not working. Fuel patch is not completed. Looking now at post-install integration. May not make this release.

**Issue:** NFV4KVM – plugin does not work with Fuel, The NFV4KVM does install a working kernel on the compute node. 
**Owner:**Don Dugger/Raghuveer Reddy
**Status:** 1/20 no new update.  We have discovered that OVS appears to require kernel modules that are not there. Working on creating a fix to the Fuel plugin creation scripts that will fix this problem, potentially will have a fix before code freeze, but certainly should be able to solve the problem for an SR1 release.
**===============================================================================================================**
__**Pod Status:**__ 

All pods back up and running

[[https://wiki.opnfv.org/pharos_rls_b_labs|Pharos lab status]]

**===============================================================================================================**
**Scenario Status:** Scenarios are frozen. No feature project has reported that their scenarios are missing or unsupported


[[https://wiki.opnfv.org/brahmaputra_testing_page#candidate_scenarios]] and [[https://wiki.opnfv.org/brahmaputra_testing_page]]
**===============================================================================================================**
__**Release Test Status:**__


Functest (Morgan Richomme)  Current situation
*****************
as you can see from a CI perspective, we were able to run successfully Functest 
https://build.opnfv.org/ci/view/functest/

the following scenario were "runnable" on Stable/Brahamputra branch: 
- compas/odl_l2-nofeature-ha
- fuel/odl_l2-nofeature-ha
- joid-odl_2-nofeature-ha

and on Master (because not available 
- apex/onos-nofeature-ha

but it is only from a CI point of view
when we dive into the details, the results are not at the same level and it is difficult to get stable results 
=> different  from one run to another, we are far from a stable run as defined during le summit (and as far as I understood there are still some major patchs on some installer side)

see details on
- https://wiki.opnfv.org/vping_brahmaputra_page
- https://wiki.opnfv.org/tempest_brahmaputra_page
- https://wiki.opnfv.org/vims_brahmaputra_page

for odl tests some are 18/18 OK some 0 and it is not stable

feature projects
*****************
regarding the integration of feature projects
we are ready for:
- doctor
- promise
- ovno
- onos

but as far as I know now, none of these tests have been run from CI
fist manual test done on onos

docs
*****
3 docs have been initiated
- a user guide => http://artifacts.opnfv.org/functest/docs/userguide/index.html
- a config/installation guide => http://artifacts.opnfv.org/functest/docs/configguide/index.html
- a developper guide => http://artifacts.opnfv.org/functest/docs/devguide/index.html

work in progress

as a conclusion
I would say that the status is Yellow/Red due 
- to stability issues
- late integration of feature projects
- difficulty to get env to work (no system available on Monday)

some work could be postponed to RC1 (if we follow apex approach, we could do something similar) 


Yardstick (Ana Cunha)  troubleshooting Joid, Compass, Fuel and Apex, working on integrated test result db.
Requested help with SSH issues.


**===============================================================================================================**
__**Installer Status:**__

**Apex** (Tim Rozet): no new status for 1/19 see 1/18 status below 


**JOID** (Narinder Gupta) Intel pod6 passed with working scenario. We have working B release. Still working on documentation and debugging orange pod2

**Compass4nfv** (Weidong Shao) no new status for 1/20 see 1/18 status below

**Fuel** (Jonas Bjurel) 
[[https://jira.opnfv.org/secure/Dashboard.jspa?selectPageId=10608|Tickets and backlog jira dashboard]]
Still working on rebasing Fuel plugins. We will start running master scenarios on Ericsson POD 2 now that it is up
Working on problems with master build. Some issues with upstream yet.

**===============================================================================================================**
__**Project Updates:**__ (as provided)

**ARMband:**

**Bottlenecks:** (Hongbo Tian) 

**Copper:** (Bryan Sullivan) 

**Doctor** (Ryota, Ildiko) There is not good progress in aodh support in apex and the upstream project [1,2].
Being complete by tomorrow is not realistic...

I think we can create post-installer, but the concern is whether we can complete the development by the deadline.

[1] https://review.openstack.org/#/c/241408/
[2] https://gerrit.opnfv.org/gerrit/#/c/6631/

**IPv6:** (Bin Hu) -          

**LSOAPI** (Kevin Leuhrs) at risk

**NFV4KVM:** (Don Dugger, Raghuveer Reddy) 
   

**ONOSFW:** (Ashlee Young) 

**OpenvSwitch** (Mark Gray) 


**OPNFVdocs** (ChrisPriceAB:) no new update for 1/20
Issues:
The code freeze date will not be met given the current state of Brahmaputra documentation.

**Parser** 

**Prediction** (Hai Liu) Prediction project has completed requirements document. Code is under development, not ready yet. It is not  integrated to a installer now.

**Qtip** (Vikram) 


**SFC** (Brady Johnson) 

**SDNVPN** (Tim Irnich)
first successful deployment in CI happened yesterday, thanks to Fatih for setting up Jenkins.
 
https://build.opnfv.org/ci/job/fuel-deploy-virtual-daily-brahmaputra/8/console
 
We will now rebase our plugins to Fuel8 and as soon as we can deploy ODL Be we can start running our E2E tests in CI as well.
 
Big thanks to all contributors!

**SFQM** (Maryam Tahhan) 

**StorPerf** (Mark Beierle) Just this week I found out I have Intel POD 9 in Pharos:  https://wiki.opnfv.org/get_started/intel_pod9 I did not realize that we were going to get any resources, so I am pleasantly surprised.

There will be documentation (currently in the wiki) that will be moved into .rst files so that it can be picked up by the documentation toolchain.  This will then be published by releng to the same location as the other documentation, and I think I might look into having it part of the StorPerf docker image too.

**===============================================================================================================**
**Action Items**__Underlined Text__
**Action:** 
**Owner:**  
**Status:** 

**Action:**
**Owner:**
**Status:** 

__**Action:**__
**Owner:**
**Status:**

------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------**1/19/16** 

Countdown to Release: **14 days**
[[https://etherpad.opnfv.org/p/steps_to_brahmaputra|__**Upcoming deadlines:**__]]
**===============================================================================================================**
**Minutes from daily standup** [[http://ircbot.wl.linuxfoundation.org/meetings/opnfv-release/2016/opnfv-release.2016-01-19-14.30.html|1/19/16]]

__**Top Accomplishments & Important Updates since last report**__ 

  .LSOAPI PTL is no longer employed at Cablelabs, work is ongoing to connect with him & identify path forward, if he is 
  not planning to participate in the project as a private contributor then the community will need to vote on a replacement.
  .Installer/documentation code freeze extended to 1/21. This potentially delays test work although they have started with those installers that are able to lockdown.
  
  
 **===============================================================================================================**

__**Top Issues**__

**Issue:** Functest & Yardstick do not produce stable results. Troubleshooting scenarios. Need assistance from installer & feature project teams to fault isolation and fix.
**Owner:** Morgan Richomme (Functest), Ana Cuna (Yardstick)
**Status:** wip, Ana put out a call for help with SSH issues during Release team meeting. Functest tests suite runnable from CI perspective, heterogeneous and not reproducible results (due to lack of stability). Need installers to stabilize then can troubleshoot scenario issues.

**Issue:** Getting RC0 out
**Owner:** Debra
**Status:**  Installer code freeze extended for 2 days due to Pods being down this past weekend. New installer/documents freeze date 1/21. Apex & JOID scenario test/troubleshooting in progress; Fuel validation of upstream integration in progress. Compass work in progress.

**Issue:** Fuel is not ready to code freeze. Pods down over weekend prevented testing/progress with validation of integrated upstream code.
**Owner:** Jonas Bjurel
**Status:** see above issue status

**Issue:** OpenContrail does not deploy
**Owner:** Stuart Mackey
**Status:** Compass got vrouter and GUI up, Apex has a build and will test, JOID waiting for a patch

**Issue:** Scenarios are not fully known, planned out or mapped to lab/test cycle
**Owner:** Debra  
**Status:**  Closed. Scenarios are frozen. No feature projects which are able to deploy with installers have reported that their scenarios are not supported. [[https://wiki.opnfv.org/brahmaputra_testing_page#candidate_scenarios]] and [[https://wiki.opnfv.org/brahmaputra_testing_page]]

**Issue:** Doctor install requirements - Aodh as critical component
**Owner:** Ildiko, Ryota
**Status:** Apex has a patch up but not working. Fuel patch is not completed. Looking now at post-install integration. May not make this release.

**Issue:** NFV4KVM – plugin does not work with Fuel, The NFV4KVM does install a working kernel on the compute node. 
**Owner:**Don Dugger/Raghuveer Reddy
**Status:** 1/15/16  We have discovered that OVS appears to require kernel modules that are not there. Working on creating a fix to the Fuel plugin creation scripts that will fix this problem, potentially will have a fix before next Tues installer code freeze, but certainly should be able to solve the problem for an SR1 release.
**===============================================================================================================**
__**Pod Status:**__ 

-Intel POD2 is back up

-CI PODs for Fuel back online

LF-POD2: is up

Ericsson-POD2: is bac up
[[https://wiki.opnfv.org/pharos_rls_b_labs|Pharos lab status]]

**===============================================================================================================**
**Scenario Status:** Scenarios are frozen. No feature project has reported that their scenarios are missing or unsupported


[[https://wiki.opnfv.org/brahmaputra_testing_page#candidate_scenarios]] and [[https://wiki.opnfv.org/brahmaputra_testing_page]]
**===============================================================================================================**
__**Release Test Status:**__


Functest (Morgan Richomme)  Current situation
*****************
as you can see from a CI perspective, we were able to run successfully Functest 
https://build.opnfv.org/ci/view/functest/

the following scenario were "runnable" on Stable/Brahamputra branch: 
- compas/odl_l2-nofeature-ha
- fuel/odl_l2-nofeature-ha
- joid-odl_2-nofeature-ha

and on Master (because not available 
- apex/onos-nofeature-ha

but it is only from a CI point of view
when we dive into the details, the results are not at the same level and it is difficult to get stable results 
=> different  from one run to another, we are far from a stable run as defined during le summit (and as far as I understood there are still some major patchs on some installer side)

see details on
- https://wiki.opnfv.org/vping_brahmaputra_page
- https://wiki.opnfv.org/tempest_brahmaputra_page
- https://wiki.opnfv.org/vims_brahmaputra_page

for odl tests some are 18/18 OK some 0 and it is not stable

feature projects
*****************
regarding the integration of feature projects
we are ready for:
- doctor
- promise
- ovno
- onos

but as far as I know now, none of these tests have been run from CI
fist manual test done on onos

docs
*****
3 docs have been initiated
- a user guide => http://artifacts.opnfv.org/functest/docs/userguide/index.html
- a config/installation guide => http://artifacts.opnfv.org/functest/docs/configguide/index.html
- a developper guide => http://artifacts.opnfv.org/functest/docs/devguide/index.html

work in progress

as a conclusion
I would say that the status is Yellow/Red due 
- to stability issues
- late integration of feature projects
- difficulty to get env to work (no system available on Monday)

some work could be postponed to RC1 (if we follow apex approach, we could do something similar) 


Yardstick (Ana Cunha)  troubleshooting Joid, Compass, Fuel and Apex, working on integrated test result db.
Requested help with SSH issues.


**===============================================================================================================**
__**Installer Status:**__

**Apex** (Tim Rozet): no new status for 1/19 see 1/18 status below 


**JOID** (Narinder Gupta) JOID team is working on documentation part of it and most likely would be completed by tomorrow. From functionality wise it is complete but documentation need some work to finish.


**Compass4nfv** (Weidong Shao) no new status for 1/19 see 1/18 status below



**Fuel** (Jonas Bjurel) no new status for 1/19 see 1/18 status below
[[https://jira.opnfv.org/secure/Dashboard.jspa?selectPageId=10608|Tickets and backlog jira dashboard]]
Pod downtime caused severe damage to week-end work. Had planned to test upstream Fuel for our code freeze. 
**===============================================================================================================**
__**Project Updates:**__ (as provided)

**ARMband:**

**Bottlenecks:** (Hongbo Tian) 

**Copper:** (Bryan Sullivan) Call out for helping hands if anyone is available

**Doctor** (Ryota, Ildiko) There is not good progress in aodh support in apex and the upstream project [1,2].
Being complete by today is not realistic...

I think we can create post-installer, but the concern is whether we can complete the development by the deadline.

[1] https://review.openstack.org/#/c/241408/
[2] https://gerrit.opnfv.org/gerrit/#/c/6631/

**IPv6:** (Bin Hu) -          

**LSOAPI** (Kevin Leuhrs) at risk, potential loss of PTL

**NFV4KVM:** (Don Dugger, Raghuveer Reddy) 
   

**ONOSFW:** (Ashlee Young) 

**OpenvSwitch** (Mark Gray) 


**OPNFVdocs** (ChrisPriceAB:) 
Issues:
The code freeze date will not be met given the current state of Brahmaputra documentation.

**Parser** 

**Prediction** (Hai Liu) Prediction project has completed requirements document. Code is under development, not ready yet. It is not  integrated to a installer now.

**Qtip** (Vikram) 


**SFC** (Brady Johnson) 

**SDNVPN** (Tim Irnich)
first successful deployment in CI happened yesterday, thanks to Fatih for setting up Jenkins.
 
https://build.opnfv.org/ci/job/fuel-deploy-virtual-daily-brahmaputra/8/console
 
We will now rebase our plugins to Fuel8 and as soon as we can deploy ODL Be we can start running our E2E tests in CI as well.
 
Big thanks to all contributors!

**SFQM** (Maryam Tahhan) For SFQM all the contributions went upstream to DPDK – DPDK won’t be integrated with an installer for release B – it was more a case of getting the features in place to allow us to integrate with an installer in the next release – do we will only be releasing documentation as part of rel B

**StorPerf** 

**===============================================================================================================**
**Action Items**__Underlined Text__
**Action:** add Contrail scenarios to scenario staging page on wiki
**Owner:**  Stuart Mackie
**Status:** done

**Action:**Review Scenarios and make sure your test case configuration requirements are covered (details in Scenario status)
**Owner:**Feature project PTLs
**Status:** Done

__**Action:**__Send out a reminder to feature projects to finalize scenarios
**Owner:**Debra
**Status:** done

------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------**1/18/16** 

Countdown to Release: **15 days**
[[https://etherpad.opnfv.org/p/steps_to_brahmaputra|__**Upcoming deadlines:**__]]
**===============================================================================================================**
**Minutes from daily standup** [[http://ircbot.wl.linuxfoundation.org/meetings/opnfv-release/2016/opnfv-release.2016-01-18-14.30.html|1/18/16]]

__**Top Accomplishments & Important Updates since last report**__ 

  .Scenarios are frozen. Soliciting updates from all feature projects to see which if any will be dropped due to no test scenario. 
  .ONOS code is fully merged and works in virtual deployments with Apex
  .Apex will cut an RC1 later this week
  .KVM4NFV has an updated script to create the KVM Fuel plugin with OVS support, trying to integrate before freeze
  .PolicyTest project will not release with Brahmaputra, will target C release
  .Copper, LSOAPI will pursue post-install deployment options
  .Doctor still working on potential installer deployment but likely will pursue post-install deployment
  
 **===============================================================================================================**

__**Top Issues**__

**Issue:** Getting RC0 out
**Owner:** Debra
**Status:** dependencies to be closed on labs, CI, tests, and installers
Apex has RC0, other installers are working on final integration & bugs. Installer code freeze Jan 19. 

**Issue:** Fuel is not ready to code freeze. Pods down over weekend prevented testing/progress with validation of integrated upstream code.
**Owner:** Jonas Bjurel
**Status:** LF POD 2 is back up but waiting full deployment to validate everything is working. Ericsson POD 2 had hardware issue with switch. Working to bring it back up.

**Issue:** OpenContrail does not deploy
**Owner:** Stuart Mackey
**Status:** Compass got vrouter and GUI up, Apex has a build and will test, JOID waiting for a patch

**Issue:** Scenarios are not fully known, planned out or mapped to lab/test cycle
**Owner:** Debra  
**Status:**  [[https://wiki.opnfv.org/brahmaputra_testing_page#candidate_scenarios]] and [[https://wiki.opnfv.org/brahmaputra_testing_page]]

**Issue:** Doctor install requirements - Aodh as critical component
**Owner:** Ildiko, Ryota
**Status:** Apex has a patch up, we are trying to get it to work.

**Issue:** NFVKVM – plugin does not work with Fuel, The NFV4KVM does install a working kernel on the compute node. 
**Owner:**Don Dugger/Raghuveer Reddy
**Status:** New 1/15/16  We have discovered that OVS appears to require kernel modules that are not there. Working on creating a fix to the Fuel plugin creation scripts that will fix this problem, potentially will have a fix before next Tues installer code freeze, but certainly should be able to solve the problem for an SR1 release.
**===============================================================================================================**
__**Pod Status:**__ 

-Intel POD2 is not up, so there is only 1 POD for Apex

-all CI PODs for Fuel has been down since Thursday last week. Just coming back online on Monday.

LF-POD2: A deployment is running on it at the moment to see if Peter Bandzi's fixes solved the issue
1. we could not get to jumphost via SSH, once we turned off other nodes from pod2 then we were able to access
2. issue we have wrongly configured PXE bridge on jumphost, after reconfig on linux nbow it looks that it is ok.... at least deployment still in progrees (but nodes were recognized by fuel now). LF POD2 status will be updated when/once the deployment succeeds

Ericsson-POD2: Reason HW fault we had problem with switch - WIP. Contact person Erik Reukers who have been working together with the vendor of the failed equipment over the week-end.

[[https://wiki.opnfv.org/pharos_rls_b_labs|Pharos lab status]]

**===============================================================================================================**
**Scenario Status:** Scenarios are frozen

[[https://wiki.opnfv.org/brahmaputra_testing_page#candidate_scenarios]] and [[https://wiki.opnfv.org/brahmaputra_testing_page]]
**===============================================================================================================**
__**Release Test Status:**__

working on common database for test results: ·       We’re consolidating all test result aggregation/visualization on a single server:

o   Testresults.opnfv.org (also testdashboard.opnfv.org)

·       The sever will have

o   Test results data base (MongoDB) – which Morgan just installed

o   Yardstick specific data base (InfluxDB) – Ana is about to bring this up

o   ELK stack for test dashboard (already installed – needs integration with MongoDB) – Peter is the PoC for ELK

o   Grafana (for Yardstick visualization) – Ana completed InfluxdB and Grafana installed, port assignment updated: https://wiki.opnfv.org/opnfv_test_dashboard#test_results_server
   
o   Morgan created  cronjob calling (every hour) a script consuming the  API locally
it generates results int he following folder (based on the work done by the project to consolidate their results)

├── res
│   ├── bottleneck
│   ├── functest
│   ├── qtip
│   ├── vsperf
│   └── yardstick

and if we go in one particular folder, we got json results

│   ├── functest
│   │   ├── res_functest_Tempest_apex.json
│   │   ├── res_functest_Tempest_compass.json
│   │   ├── res_functest_Tempest_fuel.json
│   │   ├── res_functest_Tempest_joid.json
│   │   ├── res_functest_Tempest.json
│   │   ├── res_functest_vIMS_apex.json
│   │   ├── res_functest_vIMS_compass.json
│   │   ├── res_functest_vIMS_fuel.json
│   │   ├── res_functest_vIMS_joid.json
│   │   ├── res_functest_vIMS.json
│   │   ├── res_functest_vPing_apex.json
│   │   ├── res_functest_vPing_compass.json
│   │   ├── res_functest_vPing_fuel.json
│   │   ├── res_functest_vPing_joid.json
│   │   └── res_functest_vPing.json

so we have post processed results that should be easily "graphable"

Functest (Morgan Richomme)  see above, working on integrated test result db


Yardstick (Ana Cunha)  see above, no new update still troubleshooting Joid, Compass, Fuel and Apex, working on integrated test result db


**===============================================================================================================**
__**Installer Status:**__

**Apex** (Tim Rozet): 
we will build another candidate (RC1) sometime within the next few days.

Progress since last update:
1.  Awesome work by Dan and Michael to get the RPM split done, our RPMs are now split into 3:
apex/opnfv-apex-2.1-20160117.noarch.rpm
apex/opnfv-apex-common-2.1-20160117.noarch.rpm
apex/opnfv-apex-undercloud-2.1-20160117.noarch.rpm
2.  ONOS code is fully merged and works in virtual deployments (baremetal and functest results pending) - thanks Bob and ONOS team for all the help integrating
3.  Scenarios were all added Friday to Apex Jenkins: https://build.opnfv.org/ci/view/apex/ (ONOS and ODL L2 are now part of our CI verify and daily)
4.  ODL L3 patch has been pushed and passed initial round of CI: https://gerrit.opnfv.org/gerrit/#/c/6995/6

TODO:
1.  Dan is working on OpenContrail integration.
2.  Michael is working on getting AODH support for Doctor into Apex.  Patch WIP: https://gerrit.opnfv.org/gerrit/#/c/6631/
3.  I'm working on getting SFC support.  Plan is to build Apex RPM with Be version of ODL included, and I'm writing up some ansible modules to update kernel, install NSH OVS, and Tacker.
4.  Yardstick was integrated into Apex, but is failing during run.  Will debug that with Yardstick team.

For our RC1 build my goal is to include ONOS and ODL L3.  As soon as we feel those are both stable, with good functest results, we will build RC1.


**JOID** (Narinder Gupta) 


**Compass4nfv** (Weidong Shao) 



**Fuel** (Jonas Bjurel)
[[https://jira.opnfv.org/secure/Dashboard.jspa?selectPageId=10608|Tickets and backlog jira dashboard]]
Pod downtime caused severe damage to week-end work. Had planned to test upstream Fuel for our code freeze. Now may need to request additional days before we freeze. Almost all scenarios run fine virtual on stable.
We have some more rebasment work to do, but we are close! since tonight we’re back with one lab running stable, we’re still without the lab for master and cannot test re-basements or bug-fixes L
Upstream is no longer broken, but at this point say it is reliable is a stretch!
From stable we’re successfully running most of the 6x2 scenarios (x2 because BM and virtual).
Although a lot of results point to the positive quarter of the pie, I’m reluctant to declare victory just yet – not the least because our week-end engagement was 
a complete turn-down (little can be achieved without labs).
What is done (some still waiting final verification on master due to lab outage):
-          Documentation 95% ready

-          Fuel8/Liberty uplift ready

-          Fuel8 rebasing of ODL + NSH, GPB + Berryllium ready

-          ONOS uplift to Fuel 7

-          Almost all of the 6x2 scenarios are running on virtual deployment and stable/Brahmaputra

What is left till code-freeze:
-          Be able to verify un picked changes on master to stable – pending lab availability (HW replacement is ongoing in Ericsson POD-2)

-          Re-basement of following OPNFV collaborative project plug-ins:

o   VSPERF

o   VFVOVS

o   NFVKVM

o   BGP-VPN

o   SFC-OVS

Work on this is ongoing, but is to a large degree outside the Fuel project boundaries – although we want them to succeed – and do everything in our power to help them (remember our abilities are constrained by our relatively small work-force).
We believe we can declare code-freeze end this week/beginning next, meaning the installer integration code is frozen for development, however – we need to be flexible to changes from upstream and side-stream – that will always be our
Challenge.

**===============================================================================================================**
__**Project Updates:**__ (as provided)

**ARMband:**
Fuel Port for arm64 support in Fuel 7 (and then Fuel 8) is two pieces:
Setting up public mirror containing arm64-based packages needed to build target images for controller and compute nodes
This work is in progress – we have our mirror set up and are now building and populating.
We hope to have the mirror and repos with packages available mid-week.
Modifying the code in Fuel to support arm64, where it needs to change
This work is also in progress.
I am not exactly sure when this will be up and working, but it should be pretty soon.
Functest work currently being targeted at running against Arno SR1.
Currently getting our 5-node HA pod up and running Arno SR1 (using Fuel 6.1 in this case).  
When this works and when our Fuel 7 porting work is done, we will shift this to run Fuel 7 (and then Fuel 8) and 
  Our plan for release testing is to integrate our lab with Jenkins when we have Functest running against Fuel 7/8 (Brahmaputra).  At that point we hope to run as many scenarios as can be targeted at a Fuel-based environment – some of these may not run on arm64, and we’ll either debug or document those.

  Our documentation plan is fuzzy at the moment.  We will likely base it on the Fuel project's documentation for how to set up, configure, and install OPNFV, with arm64 exceptions and specifics.

**Bottlenecks:** (Hongbo Tian) Bottlenecks has two test tools named rubbos and vstf.
Rubbos runs successfully on fuel  Brahmaputra branch, compass  Brahmaputra branch.
Vstf runs successfully on fuel Brahmaputra branch.
Bottlenecks tests are performed in LF POD2(NOSDN scenario) and Huawei  POD(NOSDN scenario)
 
By now, for SDN scenarios both pods installed openstack have network ssh connect failure issues, it’s the problem caused by the installers when they want to integrate openstack and ODL, they are working to solve this.

**Copper:** (Bryan Sullivan) Planning to check out a new hosting service for OPNFV testbeds provided by Ravello, that Iben has been arranging. Have a post-install script working, but not fully scripted/automated yet… so I would not call it per the goal. I do have a smoke test scenario I have also scripted, and it works.
 
If I can get one of the RC builds running there it may accelerate test/development of my B release goals. But for now I'm at the same state as of New Years. Still trying to get project supporters to be more active, not sure why that isn't happening (other than that this is complex stuff, and/or they are busy on other projects).

For now, my B release goal is to have at least one installer completed and ready to demo at the March Hackfest.

**Doctor** (Ryota, Ildiko) I have update only on Fuel, which is that we will need to write a plug-in to deploy Aodh, which is the alarming component we need. Ryota can give you an update on Apex.

If we will not be finished with the installer coverage I think we will shoot for the post install script.

**IPv6:** (Bin Hu) -          Major accomplishment:

o   CI integration with Huawei’s Pharos Lab in Santa Clara is completed.

o   Worked with Apex, Compass Fuel and Joid for installation scenarios and command line parameters, and related sourcing credential methods. Currently it is up to date.

o   Complete content transfer to opnfvdoc template so that toolchain can pick it up into an integrated document

o   Draft Preview Version of Release Document (IPv6): http://artifacts.opnfv.org/ipv6/docs/reldoc/index.html

-          To-Do:

o   Continue testing and debugging auto script for ODL-L2 scenario for Yardstick

§  vPing test case failed, probably due to bugs in Compass and Fuel installer. Kubi is working on it.

o   Continue to work on / polish / clarify documentation as needed

o   Continue to work with Installers for any update of command line parameters of deployment scenarios, as needed and applicable

-          Issues:

o   None

-          Blockers:

o   None.

 
Regarding “specific scenarios that will support ipv6 testing during Brahmaputra”, see our preview release document: http://artifacts.opnfv.org/ipv6/docs/reldoc/index.html. In short:
-          Scenario 3 and 4 are supported by Apex and Fuel (OS+ODL-L2)

o   Auto testing script (Yardstick) is being tested and debugged (see the first item of To-Do above)

-          OpenStack-only scenario is supported by Fuel, Compass and Joid.

o   Auto testing script (Yardstick) is working

o   This scenario is not officially listed as B release (because it is OpenStack only), but already supported by 3 installers

**LSOAPI** (Kevin Leuhrs) 

**NFV4KVM:** (Don Dugger, Raghuveer Reddy) have an updated script to create the KVM Fuel plugin with OVS support, don’t know if it’ll be integrated by tomorrow, we’re still hoping to get it in.
 
Raghu has been looking into scenarios
   

**ONOSFW:** (Ashlee Young) 

**OpenvSwitch** (Mark Gray) We have code upstreamed to stable/Brahmaputra and are running it against the CI. I guess this is the scenario that we will support

https://build.opnfv.org/ci/job/fuel-os-nosdn-ovs-noha-opnfv-jump-2-daily-brahmaputra/

It is currently queued for execution.


**OPNFVdocs** (ChrisPriceAB:) The toolchain is now in place and he have a pending patch which outlines the schema and output for the composite files.
https://gerrit.opnfv.org/gerrit/#/c/7017/ 
http://artifacts.opnfv.org/review/7017/configguide/index.html 
http://artifacts.opnfv.org/review/7017/userguide/index.html 
When integrated projects update their composite content on master they should see it popping up in the above links.  Un-integrated projects should reach out to Sofia or I to get their content in place asap.

Current focus:
Ensure the templates are being filled by projects now that we can publish them here.
Editorials on text and structure for the incoming documentation
Cherry-picking our patch backlog to stable. 

Issues:
The code freeze date will be difficult (nigh impossible) to meet given the current state of Brahmaputra documentation.

**Parser** completed testing work with yardstick, and I think yardstick team could provide you which scenario would support Parser test.
 
From installer side, Compass would be able to install Parser’s yang2tosca module, and Parser’s heat2tosca is already merged in Heat therefore all the installers would be able to install heat2tosca when they install Heat. Parser also support standalone installation should user decide to do so.
 
One work I am doing now is to try to cherry pick our latest docs update to B release stable branch.

**Policytest** (Keith Burns) too many tangential issues with OVS, openstack and SFC to make this a reality. We can test vanilla GBP Policy in Openstack, this has worked since lithium, but my focus has been too fractured to get this done for this release to have time to deal with all the unfamiliar JIRA process. Will target C release.

**Prediction** (Hai Liu) Prediction project has completed requirements document. Code is under development, not ready yet. It is not  integrated to a installer now.

**Qtip** (Vikram) 


**SFC** (Brady Johnson) 

**SDNVPN** (Tim Irnich)
Deployment through Fuel has been successfully tested in an Ericsson lab but not in a Pharos lab. Our main achievement last week was to get our first deployment scenario merged so that we are now ready with all we need for CI integration except having Jenkins jobs set up. I am expecting this to happen on Monday.
 
We are not intending testing deployment in a Pharos lab outside CI.
 
Progress with documentation has been slowed down by the ODL code freeze deadline Thu last week which we had to prioritize (resources are overlapping with ODL VPN Service project) so I expect this to get back to speed now.
 
In parallel we have started to work with rebasing our plugins to Fuel 8.0 and are preparing for the ODL Beryllium inclusion what will happen in the context of rebasing the ODL plugin.

**StorPerf** pursuing post-install configuration. follow the StorPerf instructions on how to get the StorPerf docker image downloaded into your OPNFV lab and run it manually.

**===============================================================================================================**
**Action Items**__Underlined Text__
**Action:** add Contrail scenarios to scenario staging page on wiki
**Owner:**  Stuart Mackie
**Status:** assigned 1/15

**Action:**Review Scenarios and make sure your test case configuration requirements are covered (details in Scenario status)
**Owner:**Feature project PTLs
**Status:** Deadline Jan 15, in progress. Due close of business 5pm Pacific.

__**Action:**__Send out a reminder to feature projects to finalize scenarios
**Owner:**Debra
**Status:** done

------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------

**1/15/16** 

Countdown to Release: **18 days**
[[https://etherpad.opnfv.org/p/steps_to_brahmaputra|__**Upcoming deadlines:**__]]
**===============================================================================================================**
**Minutes from daily standup** [[http://meetbot.opnfv.org/eavesdrop/%23opnfv-release/%23opnfv-release.2016-01-15.log.html|1/15/16]]

__**Top Accomplishments since yesterday**__ 

  .Compass got vrouter and GUI up for OpenContrail
  .Apex has a build for OpenContrail and will test
 **===============================================================================================================**

__**Top Issues**__

**Issue:** Getting RC0 out
**Owner:** Debra
**Status:** dependencies to be closed on labs, CI, tests, and installers
Apex has RC0, other installers are working on final integration & bugs. Installer code freeze Jan 19. 

**Issue:** OpenContrail does not deploy
**Owner:** Stuart Mackey
**Status:** Compass got vrouter and GUI up, Apex has a build and will test, JOID waiting for a patch

**Issue:** Scenarios are not fully known, planned out or mapped to lab/test cycle
**Owner:** Debra  
**Status:** Need this in order to define test schedules: Pharos  (Trevor Cooper or Fatih Degirmenci) and Test teams (Morgan Richomme- Functest, Ana Cunha- Yardstick) participating to define. Deadline for projects to sign up for scenarios has been extended to Friday Jan 15. Still a significant # of scenarios without owners. These cannot be expected to be setup and used without a shepherd to ensure they get into the schedule and actually set up as Jenkins jobs. Debra will send out one more reminder today. Since Asia is already on weekend we'll give them until 6am Pacific time on Sunday to complete (that catches their Monday) [[https://wiki.opnfv.org/brahmaputra_testing_page#candidate_scenarios]] and [[https://wiki.opnfv.org/brahmaputra_testing_page]]

**Issue:** Doctor install requirements - Aodh as critical component
**Owner:** Ildiko, Ryota
**Status:** Apex has a patch up, we are trying to get it to work.

**Issue:** NFVKVM – plugin does not work with Fuel, The NFV4KVM does install a working kernel on the compute node. 
**Owner:**Don Dugger/Raghuveer Reddy
**Status:** New 1/15/16  We have discovered that OVS appears to require kernel modules that are not there. Working on creating a fix to the Fuel plugin creation scripts that will fix this problem, potentially will have a fix before next Tues installer code freeze, but certainly should be able to solve the problem for an SR1 release.
**===============================================================================================================**
__**Pod Status:**__ 

-LFPod2 disconnected - need troubleshooting, not possible to run fuel Stable scenario



**===============================================================================================================**
**Scenario Status:** DEADLINE Friday Jan 15 6pm Pacific (Sunday 6am Pacific for Asia)

Feature projects need to validate whether their test cases can run in an existing scenario configuration that is listed in the table or whether they need special scenario. If the former, they need to validate with the test teams that their tests are planned for the scenario & lab during the time that scenario is set up. If the latter, they need to sign up to own their unique scenario, ensure a lab is scheduled to have that scenario, and ensure their tests are integrated into CI & completed. Scenarios and tests will be mapped together & scheduled based on the information charted on these pages [[https://wiki.opnfv.org/brahmaputra_testing_page#candidate_scenarios]] and [[https://wiki.opnfv.org/brahmaputra_testing_page]]
**===============================================================================================================**
__**Release Test Status:**__

Functest (Morgan Richomme)  no new update  for 1/15 see previous status below


Yardstick (Ana Cunha) no new update for 1/15 see previous status below. 


**===============================================================================================================**
__**Installer Status:**__

**Apex** (Tim Rozet): no new update  for 1/15 see previous status below


**JOID** (Narinder Gupta) no new update  for 1/15 see previous status below


**Compass4nfv** (Weidong Shao) 
Status/process about Compass: 

    ODL Lithium L2 cluster, ONOS Emu L2/L3 cluster, OpenStack Liberty HA , Ceph storage are supported in virtual or baremetal deployment. 

Status in CI pipeline:

    Scenario name has aligned for CI pipeline.
    The CI pipeline platforms for branch(master) and branch(Bramaputra) has been set up. 
    The vitrual deployment verify is for patch verify and the baremetal deployment for daily verify.

Status about Test project:
    
    Functest and Yardstick has been integrated in CI Daily verify jobs. 

    Functest Test result: 
        vPing:  Pass
        ODL:     15 Pass/ 3 Fail
        Tempest:  207 Pass/ 3 Fail
        Rally:     8 Pass/ 1 Fail
        vIMS:    deployed success but not run testcase. in troubling shouting now.
        ONOS:  11 pass / 0 Fail

    YardStick test result:
           Run successfully! 

    Bottlenecks: 
            Working in process. 

Status about SDN Controllers:

    ODL and ONOS has been integrated as a virtual daliy job(not enough Baremetal environment)
    And OpenContrail is working in process : virtual instance just started up successfully but not integrated in CI pipeline.

We still have a lot of work to do for Bramaputra release, we are a young team and I think we will make it.

At last thanks help from you, specially to Meimei, faith, Carey, Leo, Valentin, Stuart, qi Liang, Morgan, Matthew and so on.


**Fuel** (Jonas Bjurel)
[[https://jira.opnfv.org/secure/Dashboard.jspa?selectPageId=10608|Tickets and backlog jira dashboard]]
no new update for 1/15 see previous status below

TOP-3 issues:
1)    Rebasing to Fuel 8 puts a lot of preasure to the team, it’s affiliates, etc. (Fuel 8 is an upstream project
dependency – this was always accounted for – but now we’re here with the reality)

2)    Massive number of deployment scenarios and lab permutations – creates a lot of work.

3)    Integration with Functest and Yardstick has just started – expect a lot of trouble-shooting, especially

as almost all components are new!

**===============================================================================================================**
__**Project Updates:**__ (as provided)

**ARMband:** no new update for 1/15 see previous status below

**Bottlenecks:** (Hongbo Tian) no new update 1/15 see previous status below

**Copper:** (Bryan Sullivan) no new update 1/15 see previous status below

**IPv6:** (Bin Hu) no new update 1/15 see previous status below

**LSOAPI** (Kevin Leuhrs) no update for 1/15 see previous status below

**NFV4KVM:** (Raghuveer Reddy) no update for 1/15 see previous status below
   

**ONOSFW:** (Ashlee Young) no update for 1/15 in good shape see previous status below

**OpenvSwitch** (Mark Gray)  Our main deliverable is a Fuel Plugin for OVS-DPDK. We have that pretty much complete. I am hoping to have it complete today (fingers crossed) as we already have it working with some manual configuration. We have done some smoke testing and it looks ok but we haven’t done a full Functest. There are a few small issues which we will document. The CI for this plugin is done but we have not triggered it yet. We will do this when the plugin is complete. We also need to do a rebase to the latest fuel code.

My current thoughts are that, worst case, we will have a plugin with some limitations and some noted bugs. It may not pass all functest tests. So I would say "experimental" support. However, I will have a better idea in the next day or two.

Biggest issue now is that we haven’t tested our CI integration.

**OPNFVdocs** (ChrisPriceAB:) for Docs we now have the composite toolschain working and I will be adding the ready config guide files from other repo's to the master file over the weekend. docs composite document toolchain is now ready. Work is ongoing to get feature desciption documents in place for the central release docs an ensure they reflect and refer to in a meaningful way the project specific documentation. we will likely request a docs exemption on code freeze until we have a accurate reflection of the complete functionality of the Brahmaputra release.  (including stability etc)


**Qtip** (Vikram) 1. QTIP is integrated with Jenkins and deployed on Dell POD3. Other labs which have Fuel can run QTIP now.
2. QTIP integration with releng dashboard is underway. Few bugs need to be resolved.
3. QTIP documentation is under progress - target 1/18/2016.

**SFC** (Brady Johnson) Upstream ODL SFC, GBP, and Netvirt should be ready by the end of this week
Tim Rozet is working together with Ricky Noriega on preparing the OPNFV SFC testing env
Ricky is setting up the env and working on writing the Yardstick test cases
Tim is working on preparing Tacker via post-install scripts
The first round of testing will focus on using the Apex installer, and Ana Cunha is aware of this and will make sure it will be possible
The OVS NSH patch will most likely not be ready in time for Brahmaputra, but we have a work-around in place that has been shown to work
Until NSH is implemented in OVS 2.5, we will use a private branch of OVS (the so called "Pritesh patch")

**===============================================================================================================**
**Action Items**__Underlined Text__
**Action:** add Contrail scenarios to scenario staging page on wiki
**Owner:**  Stuart Mackie
**Status:** assigned 1/15

**Action:**Review Scenarios and make sure your test case configuration requirements are covered (details in Scenario status)
**Owner:**Feature project PTLs
**Status:** Deadline Jan 15, in progress. Due close of business 5pm Pacific.

__**Action:**__Send out a reminder to feature projects to finalize scenarios
**Owner:**Debra
**Status:** done

------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------

**1/14/16** 

Countdown to Release: **19 days**
[[https://etherpad.opnfv.org/p/steps_to_brahmaputra|__**Upcoming deadlines:**__]]
**===============================================================================================================**
**Minutes from daily standup** [[http://ircbot.wl.linuxfoundation.org/meetings/opnfv-release/2016/opnfv-release.2016-01-14-14.30.html|1/14/16]]

__**Top Accomplishments since yesterday**__ 

  .Apex has RC0 artifact
  .Promise has  integrated with Fuel 
  .ODL Beryllium Beta integrated with Fuel
 **===============================================================================================================**

__**Top Issues**__

**Issue:** Getting RC0 out
**Owner:** Debra
**Status:** dependencies to be closed on labs, CI, tests, and installers
Apex has RC0, other installers are working on final integration & bugs. Installer code freeze Jan 19. 

**Issue:** OpenContrail does not deploy
**Owner:** Stuart Mackey
**Status:** deployed with Compass, found nova issues; Apex has a build- gave them details on how to modify for Liberty; Need to patch Juju charm for JOID

**Issue:** Scenarios are not fully known, planned out or mapped to lab/test cycle
**Owner:** Debra  
**Status:** Need to define test schedules: Pharos  (Trevor Cooper or Fatih Degirmenci) and Test teams (Morgan Richomme- Functest, Ana Cunha- Yardstick) participating to define. Deadline for projects to sign up for scenarios has been extended to Friday Jan 15. Feature projects need to validate whether their test cases can run in an existing scenario configuration or whether they need special scenario. If the former, they need to validate with the test teams that their tests are planned for the scenario & lab during the time that scenario is set up. If the latter, they need to sign up to own their unique scenario, ensure a lab is scheduled to have that scenario, and ensure their tests are integrated into CI & completed. Scenarios and tests will be mapped together & scheduled based on the information charted on these pages [[https://wiki.opnfv.org/brahmaputra_testing_page#candidate_scenarios]] and [[https://wiki.opnfv.org/brahmaputra_testing_page]]

**Issue:** Doctor install requirements - Aodh as critical component
**Owner:** Ildiko, Ryota
**Status:** Changes in Ceilometer for Liberty require that an Aodh plugin is developed for deployment of Doctor. Ildiko requested assistance from an installer team as she has no experience writing these type of plugin. One installer deploying is required to ensure the project remains in Brahmaputra release. Tim Rozet talked to the author of puppet-aodh, works just fine with liberty and is stable. Ryota is working with Michael Chapman on Apex team trying to get it working with Apex. Ildiko is working with Mirantis to try to get it working with Fuel. 

**Issue:** NFVKVM – plugin does not work with Fuel, The NFV4KVM does install a working kernel on the compute node. 
**Owner:**Don Dugger/Raghuveer Reddy
**Status:** New 1/15/16  We have discovered that OVS appears to require kernel modules that are not there. Working on creating a fix to the Fuel plugin creation scripts that will fix this problem, potentially will have a fix before next Tues installer code freeze, but certainly should be able to solve the problem for an SR1 release.
**===============================================================================================================**
__**Pod Status:**__ 

-Labs for CI/test stabilized, 8 working pods designated for CI/test
-additional pods coming online for dev work or backup to CI/test
-latest status always available here:
https://wiki.opnfv.org/pharos_rls_b_labs#status_of_production_ci_resources_for_release_b


**===============================================================================================================**
**Scenario Status:** DEADLINE Friday Jan 15

Feature projects need to validate whether their test cases can run in an existing scenario configuration that is listed in the table or whether they need special scenario. If the former, they need to validate with the test teams that their tests are planned for the scenario & lab during the time that scenario is set up. If the latter, they need to sign up to own their unique scenario, ensure a lab is scheduled to have that scenario, and ensure their tests are integrated into CI & completed. Scenarios and tests will be mapped together & scheduled based on the information charted on these pages [[https://wiki.opnfv.org/brahmaputra_testing_page#candidate_scenarios]] and [[https://wiki.opnfv.org/brahmaputra_testing_page]]
**===============================================================================================================**
__**Release Test Status:**__

Functest (Morgan Richomme)  
functest troubleshooting in progress
wiki pages initiated to have a summary per testcases [[https://wiki.opnfv.org/functextnexttaks]]
lots of tests failed (mainly network issues) and integration of companion projects just started (promise, onos) lacking also stability from one run to another

[[https://wiki.opnfv.org/vping_brahmaputra_page]]
[[https://wiki.opnfv.org/vims_brahmaputra_page]]
[[https://wiki.opnfv.org/tempest_brahmaputra_page]]

ONOS test case integration started, we have now onos labs so it should be ok

Yardstick (Ana Cunha) for fuel: odl scenario fails (ssh timeout), for compass: working last night, however one new fault found today, for joid: troubleshooting, for apex: started yardstick integration on baremetal with help from apex team


**===============================================================================================================**
__**Installer Status:**__

**Apex** (Tim Rozet): Accomplishments since Yesterday:
we have built an RC0 candidate artifact!  This build supports ODL Lithium 3.3, OpenStack Liberty HA, network isolation, Ceph in a virtual or baremetal environment.

[[http://artifacts.opnfv.org/index.html]]
apex/brahmaputra/opnfv-apex-2.7-brahmaputra.1.rc0.noarch.rpm
apex/brahmaputra/opnfv-apex-2.7-brahmaputra.1.rc0.src.rpm
apex/brahmaputra/opnfv-brahmaputra.1.rc0.iso
apex/brahmaputra/opnfv-brahmaputra.1.rc0.properties

Functest results on artifact:
Tempest cases: 110 Pass/8 Fail
ODL cases: 15 Pass/3 Fail
VPING: 1 Pass/0 Fail
Rally: 6 Pass/2 Fail
VIMS: Aborted

Functest caught a bug for us that we will fix post RC0: [[https://jira.opnfv.org/browse/APEX-59]], which also caused VIMS to abort.

TODO looking past RC0:
1.  Found an issue where ONOS build artifact was not being included in our build.  Patch currently up to fix that. (Waiting on #2)

2.  However, because of #1, our artifact (rpm) size grows over 4GB which is the limit of cpio, and breaks the rpm build.  Dan is working on splitting our rpm into separate rpms which will fix this problem. Patch currently being tested: [[https://gerrit.opnfv.org/gerrit/#/c/6227/]]

3.  Test ONOS and make sure it works with our integration. (Depends on #1, #2)

4.  Dan was able to get OpenContrail to build as an RPM.  Stuart has provided us configuration info.  Next step is to see if the RPM works and if we can install/configure OpenContrail.

5.  Michael is working on getting AODH support for Doctor into Apex.  Patch WIP: [[https://gerrit.opnfv.org/gerrit/#/c/6631/]]

6.  I'm working on getting SFC support.  Plan is to build Apex RPM with Be version of ODL included, and I'm writing up some ansible modules to update kernel, install NSH OVS, and Tacker.

7.  Yardstick was integrated into Apex, but is failing to run.  Will debug that with Yardstick team.

**JOID** (Narinder Gupta)
.seeing few failure which i will be fixing it today.
.onos also plugin is availble in joid
.opencontrail migration for Kilo is complete. Plugin built in joid but a few bug fix needs to be done by juniper team for liberty support. Liberty changes needed in charm.

**Compass4nfv** (Weidong Shao) No update

**Fuel** (Jonas Bjurel)
[[https://jira.opnfv.org/secure/Dashboard.jspa?selectPageId=10608|Tickets and backlog jira dashboard]]
 
Top priorities/outstanding issues:
 1) Finalize upstream Fuel 8 rebasing - tomorrow Fri
 2) Finalize plugin rebasing to Fuel 8 - next week (Tue)
 3) Getting all 25 scenarios work/deploying - next week (Fri)
 4) Implementing scenario logic for scenarios requireing different Network topologies than Pharos prescribed standard
     Network set-up (ONOS an NFVOVS) – next week (Tue)
5) Documentation ready, due Tue next week
 
Top news:
1)    OpenContrail will not be part Fuel@OPNFV Brahmaputra SR0 release

2)    Promise has  integrated with Fuel – tests to progress during next week!

3)    ODL Beryllium Beta integrated with Fuel – patch will be merged early next week!

4)    BGP-VPN added deployment scenarios – to be tested tomorrow/over the week-end!

Today integrated plug-ins and their status:
-       ODL (L2, L3, SFC, BGP-VPN, Beryllium) – OK

-       ONOS - OK

-       OVS (NSH, GPB, DPDK) - OK

-       NFVOVS (DPDK) – OK

-       VSPERF - OK

-       BGPVPN – OK

-       NFVKVM – NOK, lacking needed kernel modules for successful deployment – will not fix for the B-release

TOP-3 issues:
1)    Rebasing to Fuel 8 puts a lot of preasure to the team, it’s affiliates, etc. (Fuel 8 is an upstream project
dependency – this was always accounted for – but now we’re here with the reality)

2)    Massive number of deployment scenarios and lab permutations – creates a lot of work.

3)    Integration with Functest and Yardstick has just started – expect a lot of trouble-shooting, especially

as almost all components are new!

**===============================================================================================================**
__**Project Updates:**__ (as provided)

**ARMband:** no new update for 1/14
Port Fuel Installer to support Pharos lab populated with arm64 servers
Creating initial bootstrap
Done for Arno SR1 environment
This is done using semi-manually using tools and headstart on an arm64 build-server.
Putting together Fuel-7 (and eventually 8) mechanism 
This is done by adding arm64 support to the real Fuel7/8 mechanisms
This is being done on a b-release pod with 2x (non-HA) Applied Micro servers.
Status: We are now updating Fuel-7 as it comes from the b-release to be capable of building bootstrap 
Providing arm64 repos (primarily a gap for MOS repos – everything else expected to be upstream)
Working on creating mirror for repos that are arm64-specific and not available upstream yet.
Working within the Fuel7 context…and then Fuel8.
Porting Fuel machinery to support arm64 as well as x86
Moving forward with creating patch with changes to b-release Fuel7 (doing work first, then making patch)
Porting Functest to arm64
Constructing an HA pod running against Arno SR1 to get Functest running against that more standard environment
This is 2 SoftIron (AMD) and 1 AMD servers for the three controllers, and two Caviums for the computes.
Will then move that work forward to work against the B-release environment.
Additionally, we are investigating building an arm64-based jump server hosting a Fuel Master VM in order to simplify/streamline the parts of Fuel that need to perform binary builds (alternative is cross-compilation or remote arm64 build-server).
We have been working with an x86 jump server up until now

**Bottlenecks:** (Hongbo Tian) no new update 1/14
   .integrated  with CI pipline on the LF POD2 and fuel(installer). It runes successfully. There are no issues toward meeting the release.

**Copper:** (Bryan Sullivan) no new update 1/14
has abandoned installer integration and is focusing on post-install deployment and verification tests

**IPv6:** (Bin Hu) no new update 1/14
Completed:
   .Automated setup script for OpenStack-only scenario is completed, and merged into Yardstick
   .Automated setup script (draft) for ODL-L2 scenario  is done, and under internal code review
   .Documentation (draft) is completed
Full documentation
B Release specific (draft)
To-Do:
   .CI integration with Huawei’s Pharos Lab in Santa Clara
   .Complete code review of auto script for ODL-L2 scenario
   .Move content of B Release specific document to central document repository so that opnfvdoc project can collect and generate one document of B release
   .Get installer project’s documentation to verify the options and parameters of installer for our scenarios (OS-only and ODL-L2)
   .Once we verified the options and parameters are correct, we can lock down our documentation.

**LSOAPI** (Kevin Leuhrs) no update for 1/14 has not run on a pharos compliant lab. A colleague who has been working on the Apex project and installing OPNFV onsite (with varying levels of stability and success) may be able to get a virtual installation here that we can use. We will add scenario descriptions. We will likely need help developing a script providing post-installation for the continuous integration environment and to get the CI deploying the scenarios automatically.

**NFV4KVM:** (Raghuveer Reddy) no update for 1/14
   .We have completed the testing on locally but have not tested on the lab yet.
   .Waiting for PODs were we plan to do KVM4NFV testing are to be ready. We are hoping the POD to be ready in the next few days

**ONOSFW:** (Ashlee Young) no update for 1/14
  .integrated with CI and look to be in good shape with all of the installers, other than the issues they might be having directly. 
  .We have our scenarios up on the scenarios wiki too. 
  .Reached out to Weidong regarding physical pharos compass status, waiting to hear back. Think for the most part we're in decent shape. 
   .Saw the note from Apex that the Apex ISO is too large now, so waiting to hear back if Apex needs a new RPM from us
   
**===============================================================================================================**
**Action Items**__Underlined Text__
**Action:** Update scenario & test web pages https://wiki.opnfv.org/brahmaputra_testing_page#candidate_scenarios and https://wiki.opnfv.org/brahmaputra_testing_page
**Owner:** Morgan Richomme 
**Status:** done

**Action:**Review Scenarios and make sure your test case configuration requirements are covered (details in Scenario status)
**Owner:**Feature project PTLs
**Status:** Deadline Jan 15, in progress

__**Action:**__Send out a reminder to key projects to participate in daily standup or at least send a status if not able to attend
**Owner:**Debra
**Status:** done


------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
**1/13/16**

**Countdown to Release: 20 days**
**===============================================================================================================**
**Minutes from daily standup** [[http://ircbot.wl.linuxfoundation.org/meetings/opnfv-release/2016/opnfv-release.2016-01-13-14.30.html|1/13/16]]
**Top Accomplishments:**__Underlined Text__
**This week:** 
  .Labs needed for CI & Test were greatly stabilized 
  .Installers made progress integrating with CI & validating in lab environments 
  .Compass made progress with OpenContrail Deploy
  .ARMband making good progress on porting Fuel installer to support Pharos lab populated with arm64 servers and porting Functest to ARM
 .IPv6 Automated setup script for OpenStack-only scenario is completed, and merged into Yardstick. Automated setup script (draft) for ODL-L2 scenario  is done, and under internal code review
 .ONOSFW Integrated with CI pipeline
 
**===============================================================================================================**

__**Top Issues**__
**Issue:** Getting RC0 out
**Owner:** Debra
**Status:** dependencies to be closed on labs, CI, tests, and installers
Branch is created, labs needed for test are stabilized, installers are working on final integration & bugs. Installer code freeze moved to Jan 19. 

**Issue:** OpenContrail does not deploy
**Owner:** Stuart Mackey
**Status:** deployed with Compass, found nova issues; Apex has a build- gave them details on how to modify for Liberty; Need to patch Juju charm for JOID

**Issue:** Scenarios are not fully known, planned out or mapped to lab/test cycle 
**Owner:** Debra  
**Status:** Need to define test schedules: Pharos  (Trevor Cooper or Fatih Degirmenci) and Test teams (Morgan Richomme- Functest, Ana Cunha- Yardstick) participating to define. Deadline for projects to sign up for scenarios has been extended to Friday Jan 15. Feature projects need to validate whether their test cases can run in an existing scenario configuration or whether they need special scenario. If the former, they need to validate with the test teams that their tests are planned for the scenario & lab during the time that scenario is set up. If the latter, they need to sign up to own their unique scenario, ensure a lab is scheduled to have that scenario, and ensure their tests are integrated into CI & completed. Scenarios and tests will be mapped together & scheduled based on the information charted on these pages https://wiki.opnfv.org/brahmaputra_testing_page#candidate_scenarios and https://wiki.opnfv.org/brahmaputra_testing_page

**Issue:** Doctor install requirements - Aodh as critical component
**Owner:** Ildiko, Ryota
**Status:** Changes in Ceilometer for Liberty require that an Aodh plugin is developed for deployment of Doctor. Ildiko requested assistance from an installer team as she has no experience writing these type of plugin. One installer deploying is required to ensure the project remains in Brahmaputra release. Tim Rozet talked to the author of puppet-aodh, works just fine with liberty and is stable. Ryota is working with Michael Chapman on Apex team trying to get it working with Apex. Ildiko is working with Mirantis to try to get it working with Fuel. 

**===============================================================================================================**
__**Pod Status:**__ 
-Labs for CI/test stabilized, 8 working pods designated for CI/test
-additional pods coming online for dev work or backup to CI/test
-latest status always available here:
https://wiki.opnfv.org/pharos_rls_b_labs#status_of_production_ci_resources_for_release_b


**===============================================================================================================**
__**Scenario Status:**__ DEADLINE Friday Jan 15
Feature projects need to validate whether their test cases can run in an existing scenario configuration that is listed in the table or whether they need special scenario. If the former, they need to validate with the test teams that their tests are planned for the scenario & lab during the time that scenario is set up. If the latter, they need to sign up to own their unique scenario, ensure a lab is scheduled to have that scenario, and ensure their tests are integrated into CI & completed. Scenarios and tests will be mapped together & scheduled based on the information charted on these pages https://wiki.opnfv.org/brahmaputra_testing_page#candidate_scenarios and https://wiki.opnfv.org/brahmaputra_testing_page
**===============================================================================================================**
__**Release Test Status:**__

**Functest** (Morgan Richomme) Functest has been run accross all the installers. Troubleshooting in progress in the different internal test case; integration of companion projects just started

**Yardstick** (Ana Cunha) troubleshooting compass and joid, activation of new scenarios caused yardstick to fail in LF POD2. SFC, NFFG and SDNVPN. scripts in progress. On track for IPv6, KVM, Parser, HA, Generic NFVI test cases (Yardstick owned) . OVSNFV have decided to focus on Functest.


**===============================================================================================================**
__**Installer Status:**__
**Apex** (Tim Rozet): Accomplishments since Monday:
1.  Ceph is working and merged.
2.  Network isolation is fully working (admin, private, public, storage networks created and used in deployment).
3.  Was able to get functest to run with 118 pass/27 fail (Thanks Jose and Morgan for the help).  There was a slight error in functest where some of our keystone users got deleted, so I am hoping on the next run there will be more passes: http://213.77.62.197/results?project=functest&case=Tempest&installer=apex

Current status for RC0:
I believe all the patches are in place to provide us an RC0 artifact.  The only remaining issue is functest failed to start again on our daily last night, so we need to fix that so we can get a valid artifact.

TODO for RC0:
1.  Fix functest execution in our daily.
2.  Merge current master code to stable/b and produce RC0 candidate via daily job.

TODO looking past RC0:
1.  Found an issue where ONOS build artifact was not being included in our build.  Patch currently up to fix that.
2.  However, because of #1, our artifact (rpm) size grows over 4GB which is the limit of cpio, and breaks the rpm build.  Dan is working on splitting our rpm into separate rpms which will fix this problem.
3.  Test ONOS and make sure it works with our integration.
4.  Dan was able to get OpenContrail to build as an RPM.  Waiting on Stuart for more information on what we need to do to integrate.
5.  Michael is working on getting AODH support for Doctor into Apex.
6.  I'll work on getting SFC supported.

**JOID** (Narinder Gupta)

**Compass4nfv** (Weidong Shao)

**Fuel** (Jonas Bjurel)

**===============================================================================================================**
__**Project Updates:**__ (as provided)

**ARMband:** 
Port Fuel Installer to support Pharos lab populated with arm64 servers
Creating initial bootstrap
Done for Arno SR1 environment
This is done using semi-manually using tools and headstart on an arm64 build-server.
Putting together Fuel-7 (and eventually 8) mechanism 
This is done by adding arm64 support to the real Fuel7/8 mechanisms
This is being done on a b-release pod with 2x (non-HA) Applied Micro servers.
Status: We are now updating Fuel-7 as it comes from the b-release to be capable of building bootstrap 
Providing arm64 repos (primarily a gap for MOS repos – everything else expected to be upstream)
Working on creating mirror for repos that are arm64-specific and not available upstream yet.
Working within the Fuel7 context…and then Fuel8.
Porting Fuel machinery to support arm64 as well as x86
Moving forward with creating patch with changes to b-release Fuel7 (doing work first, then making patch)
Porting Functest to arm64
Constructing an HA pod running against Arno SR1 to get Functest running against that more standard environment
This is 2 SoftIron (AMD) and 1 AMD servers for the three controllers, and two Caviums for the computes.
Will then move that work forward to work against the B-release environment.
Additionally, we are investigating building an arm64-based jump server hosting a Fuel Master VM in order to simplify/streamline the parts of Fuel that need to perform binary builds (alternative is cross-compilation or remote arm64 build-server).
We have been working with an x86 jump server up until now

**Bottlenecks:** (Hongbo Tian) 
   .integrated  with CI pipline on the LF POD2 and fuel(installer). It runes successfully. There are no issues toward meeting the release.

**Copper:** (Bryan Sullivan) has abandoned installer integration and is focusing on post-install deployment and verification tests

**IPv6:** (Bin Hu)
Completed:
   .Automated setup script for OpenStack-only scenario is completed, and merged into Yardstick
   .Automated setup script (draft) for ODL-L2 scenario  is done, and under internal code review
   .Documentation (draft) is completed
Full documentation
B Release specific (draft)
To-Do:
   .CI integration with Huawei’s Pharos Lab in Santa Clara
   .Complete code review of auto script for ODL-L2 scenario
   .Move content of B Release specific document to central document repository so that opnfvdoc project can collect and generate one document of B release
   .Get installer project’s documentation to verify the options and parameters of installer for our scenarios (OS-only and ODL-L2)
   .Once we verified the options and parameters are correct, we can lock down our documentation.

**LSOAPI** (Kevin Leuhrs) has not run on a pharos compliant lab. A colleague who has been working on the Apex project and installing OPNFV onsite (with varying levels of stability and success) may be able to get a virtual installation here that we can use. We will add scenario descriptions. We will likely need help developing a script providing post-installation for the continuous integration environment and to get the CI deploying the scenarios automatically.

**NFV4KVM:** (Raghuveer Reddy)
   .We have completed the testing on locally but have not tested on the lab yet.
   .Waiting for PODs were we plan to do KVM4NFV testing are to be ready. We are hoping the POD to be ready in the next few days

**ONOSFW:** (Ashlee Young)
  .integrated with CI and look to be in good shape with all of the installers, other than the issues they might be having directly. 
  .We have our scenarios up on the scenarios wiki too. 
  .Reached out to Weidong regarding physical pharos compass status, waiting to hear back. Think for the most part we're in decent shape. 
   .Saw the note from Apex that the Apex ISO is too large now, so waiting to hear back if Apex needs a new RPM from us

**===============================================================================================================**
__**Action Items**__

**Action:** Update scenario & test web pages https://wiki.opnfv.org/brahmaputra_testing_page#candidate_scenarios and https://wiki.opnfv.org/brahmaputra_testing_page
**Owner:** Morgan Richomme 
**Status:** assigned 1/13, due 1/14

**Action:**Review Scenarios and make sure your test case configuration requirements are covered (details in Scenario status)
**Owner:**Feature project PTLs
**Status:** Deadline Jan 15

**Action:**Send out a reminder to key projects to participate in daily standup or at least send a status if not able to attend
**Owner:**Debra
**Status:** assigned 1/13, due 1/14

**===============================================================================================================**
Wiki

User Tools

Site Tools

Page Tools
