This is an old revision of the document!
2/3/16 and 2/4/16
Countdown to Release: TBD days __**Upcoming deadlines:**__ under revision with proposed milestones dates until we work out details. TSC approved release delay but chose to wait one more week before setting date in hopes that by then some issues will be resolved allowing for better estimations of time; Etherpad currently proposed release date is 3/8; we also have a matrix of the scenarios and their pass/fail/stable status which will help drive decisions =============================================================================================================== Minutes from daily standup 2/3/16
Top Accomplishments & Important Updates since last report
.Excellent progress by test teams today in overcoming blocking issues
.Functest
- becoming more stable on scenarios that it has been able to run.They successfully identified and fixed the issue they were having with vPing and SSH on ODL scenarios.
- A full list of tests they have run per scenario, which tests have passed or failed, which features are supported by those scenarios, and which remain to be tested for the first time is available here Keep in mind that this is a static page, if you need more current information then visit the test dashboard
- according to success criteria, we have already 1 validated scenario: compass/onos
- another one just needs iterations to be validated: compass/nosdn
.Yardstick
- have a work around in place for networking issues with ODL scenarios, these issues seem to be Lithium related, troubleshooting continues
===============================================================================================================
Top Issues
Issue: have release date. Test cycles for each scenario take a significant amount of time. Even if all installers & scenarios are working properly it takes 7 days to run test through them all. Owner: Debra Status: TSC voted to delay release rather than trying to phase release. Debra & key community members to identify remaining work and develop reasonably accurate schedule to propose to TSC. On Tuesday TSC was not comfortable setting a date while such significant issues remained in troubleshooting status for test projects. Major breakthroughs have been achieved in the past 2 days making better reasoned schedule estimates possible. Current estimate is Mar 8 release
Issue: Functest & Yardstick do not produce stable results. Troubleshooting scenarios. Need assistance from installer & feature project teams to fault isolation and fix. Owner: Morgan Richomme (Functest), Ana Cuna (Yardstick) Status: -regression on Tempest suite (investigation to knwo if it is in Functest or upstream) continues.
- working through scenarios in logical order coordinated with Yardstick, deployment and feature projects as needed to allow for additional testing & troubleshooting
Yardstick: -odl scenarios - have workaround in place for vping and ssh which allows for further testing & debug
- working on bug fixes, docs and troubleshooting
Issue: A large number of projects had code file licensing issues (either missing licenses or non-Apache licenses). Owner: Ray Paik Status:Scott discussed status during TSC meeting; Chris to post instructions online for license inclusion. Projects have made good progress on correcting issues. Scott rerunning scan to see if any projects need to address additional issues
Issue: Scenarios are frozen, but significant number not working yet, or have not been tried Owner: test teams Status: Continuing to run & debug scenarios in coordinated manner
Issue: NFV4KVM – plugin does not work with Fuel, The NFV4KVM does install a working kernel on the compute node. Owner:Don Dugger/Raghuveer Reddy Status: no new update. Fuel has bigger issues at the moment. We have discovered that OVS appears to require kernel modules that are not there. Working on creating a fix to the Fuel plugin creation scripts that will fix this problem, potentially will have a fix before code freeze, but certainly should be able to solve the problem for an SR1 release. =============================================================================================================== Pod Status:
Request Jira ticket be logged for any and all issues as email responses are difficult to keep track of. We need clearer communication.
=============================================================================================================== Scenario Status: All scenarios which features they support and which have passed/failed/not run yet in functest is located https://wiki.opnfv.org/functextnexttaks#brahmaputra_summary Have asked for a similar view to be constructed for Yardstick
=============================================================================================================== Release Test Status:
Functest (Morgan Richomme)
- according to success criteria, we have already 1 validated scenario: compass/onos
- another one just needs iterations to be validated: compass/nosdn
- some are closed but the criteria of 90% of success on Tempest is not reached
- support on troubleshooting the Tempest error / scenario / installer welcome –
- morning errors on CI were not due to tests but to result pushed in artifacts (need config on jumphosts), now artifacts are pushed
- joid has better results on non CI POD (Orange POD2) regarding Tempest (92%), so just considering CI POD could be unfair for installer if PODs have some network issues
example of log pushed to artifact (here tempest log on apex)
Yardstick:ONOS -
- works for ONOS (Compass) and nosdn (Compass and Fuel7) - rebasing in progress for fuel8
- on-demand tests (not CI PODs) - HA, KVM - works
- Parser test case OK (will be run on weekly)
- vTC verified daily and weekly on Ericsson POD1
- ipv6 test case runs on Compass Huawei SC POD
- working on bug fixes, docs and troubleshooting
=============================================================================================================== Installer Status:
Apex (Tim Rozet):
JOID (Narinder Gupta)
Compass4nfv (Weidong Shao)
Fuel (Jonas Bjurel) Tickets and backlog jira dashboard
=============================================================================================================== Project Updates: (as provided) Please find current Jira dashboards for each project here https://wiki.opnfv.org/releases/brahmaputra/release_plan#current_jira_dashboards
Availability：（Qiao Fu)
Bottlenecks: (Hongbo Tian)
Copper: (Bryan Sullivan)
Doctor (Ryota, Ildiko)
HA(Fu Qaio)
IPv6: (Bin Hu) -
NFV4KVM: (Don Dugger, Raghuveer Reddy)
ONOSFW: (Ashlee Young)
OpenvSwitch (Mark Gray)
OPNFVdocs (Chris Price)
OVNO (Stuart Mackie)
Parser
Prediction (Hai Liu)
Qtip (Vikram)
SFC (Brady Johnson)
SDNVPN (Tim Irnich)
SFQM (Maryam Tahhan)
StorPerf (Mark Beierle)
=============================================================================================================== Action ItemsUnderlined Text Action: Owner: Status:
1/26/16
Countdown to Release: TBD days __**Upcoming deadlines:**__ =============================================================================================================== Minutes from daily standup 1/26/16
Top Accomplishments & Important Updates since last report
.TSC voted to move code freeze to Friday 1/29 and to extend release date (to be defined). Debra and key community members to evaluate remaining work and recommend new release date with solid schedule to achieve that.
===============================================================================================================
Top Issues
Issue: We are at high risk of missing release date. Test cycles for each scenario take a significant amount of time. Even if all installers & scenarios are working properly it takes 7 days to run test through them all. Owner: Debra Status: TSC voted to delay release rather than trying to phase release. Debra & key community members to identify remaining work and develop reasonably accurate schedule to propose to TSC.
Issue: Functest & Yardstick do not produce stable results. Troubleshooting scenarios. Need assistance from installer & feature project teams to fault isolation and fix. Owner: Morgan Richomme (Functest), Ana Cuna (Yardstick) Status:
Yardstick:ONOS - successfully run with Compass. ODL scenario - debugging on-going, Yardstick will not run with ODL. Narrowing down subsystems involved. Working on SFC test cases. Working on influxdB+grafana
Issue: Getting RC0 out Owner: Debra Status: Installer code freeze extended due to Fuel trying to stabilize with upstream. Compass also troubleshooting some functest and other instability with the code base, need some extra time in getting Compass code stable. Apex, JOID ready to freeze
Issue: Fuel is not ready to code freeze. Upstream Fuel is not stable. Owner: Jonas Bjurel Status: autodeploy issue, hopefully solved now. Testing. Scenarios being rebased
Issue: A large number of projects had code file licensing issues (either missing licenses or non-Apache licenses). Owner: Ray Paik Status:Projects have made good progress on getting patches in to fix the files. Scott is making another license scan to see what if anything is outstanding. Thank you to all who have responded to this request in such a timely manner.
Issue: OpenContrail does not deploy Owner: Stuart Mackey Status: Apex still trying to get contrail to work, running into some openstack version issues where some contrail pieces rely on juno
Issue: Scenarios are frozen, but several not working yet, or have not been tried Owner: test teams Status: Next step is creating Jenkins jobs and testing each scenario. In process with test teams leading. https://wiki.opnfv.org/brahmaputra_testing_page#candidate_scenarios and https://wiki.opnfv.org/brahmaputra_testing_page
Issue: NFV4KVM – plugin does not work with Fuel, The NFV4KVM does install a working kernel on the compute node. Owner:Don Dugger/Raghuveer Reddy Status: 1/22 no new update. Fuel has bigger issues at the moment. We have discovered that OVS appears to require kernel modules that are not there. Working on creating a fix to the Fuel plugin creation scripts that will fix this problem, potentially will have a fix before code freeze, but certainly should be able to solve the problem for an SR1 release. =============================================================================================================== Pod Status:
All pods up and running
=============================================================================================================== Scenario Status: Scenarios are frozen. No feature project has reported that their scenarios are missing or unsupported. Candidate scenarios are having Jenkins jobs created for them. Test teams troubleshooting scenarios and test software. Reviewed scenarios in table and noted those that will not be implemented this release
Apex: 5 scenarios on Jenkins: 4 work from deploy point of view odl_l2 is about 90 pass/10 fail, odl_l3 worse pass rate Not successfully deployed yet: ocl- working on it
Compass: 4 scenarios on Jenkins: pure openstack, odl_l2, onos, and ocl. From deployment point of view, 3 working, 1 not. Deployed successfully: openstack, odl_l2, onos. Not successfully deployed yet: ocl (removed from this installer)
Fuel: 8 scenarios on Jenkins: pure openstack, odl_l2, odl_l3, onos, bgpvpn, ovs, kvm, kvm_ovs. From deployment point of view, 5 working, 1 not working, 2 not activated/attempted yet. Deployed successfully: openstack, odl_l2, odl_l3, onos, bgpvpn Not successfully deployed yet: ovs. Not activated/attempted: kvm, kvm_ovs.
Fuel has Fuel7 on stable/brahmaputra branch and Fuel8 uplift work is WIP on master branch.
Joid: 4 scenarios on Jenkins: pure openstack, odl_l2, onos, and ocl. From deployment point of view, 3 working, 1 not activated/attempted yet. Deployed successfully: openstack, odl_l2, onos. Not successfully deployed yet: none. Not activated/attempted: ocl (removed from this installer)
https://wiki.opnfv.org/brahmaputra_testing_page#candidate_scenarios and https://wiki.opnfv.org/brahmaputra_testing_page =============================================================================================================== Release Test Status:
Functest (Morgan Richomme) Current situation
Functest regression yesterday night due to sdnvpn integration, fixed now .vPing/SSH Ok on manual tests with compass, not working with all scenarios yet we defined 2 vPING: vPing SSH and vPing userdata/cloudinit https://wiki.opnfv.org/vping_brahmaputra_page vPing/SSH should be OK now on compass (wait for next run). Table to be updated for apex as well I think http://testresults.opnfv.org/testapi/dashboard?project=functest&case=vPing&installer=apex and you can change the installer. wait for the test dashboard of LF…
Yardstick (Ana Cunha) first successful run with joid yesterday (os scenario), writing for a patch for workaround for odl scenario - remains to be tested on the labs On-going work for grafana dashboard and documentation Started to document results for ha
=============================================================================================================== Installer Status:
Apex (Tim Rozet): Still trying to get contrail to work, running into some openstack version issues where some contrail pieces rely on juno
JOID (Narinder Gupta) JOID deployed successfully on all three labs yesterday including the virtual deployment. Unfortunately functest had some issues yesterday which seems to be resolved today.
Compass4nfv (Weidong Shao) there were some functest and other instability with the code base, and we will need some extra time in getting Compass code stable
Fuel (Jonas Bjurel) Tickets and backlog jira dashboard 1)
Struggling with an autodeploy issue, hopefully solved now. Testing. Scenarios being rebased
=============================================================================================================== Project Updates: (as provided)
Scenario view status: Reviewed scenarios in table and noted those that will not be implemented this release Apex: 5 scenarios on Jenkins: 4 work from deploy point of view odl_l2 is about 90 pass/10 fail, odl_l3 worse pass rate Not successfully deployed yet: ocl- working on it
Compass: 4 scenarios on Jenkins: pure openstack, odl_l2, onos, and ocl. From deployment point of view, 3 working, 1 not. Deployed successfully: openstack, odl_l2, onos. Not successfully deployed yet: ocl (removed from this installer)
Fuel: 8 scenarios on Jenkins: pure openstack, odl_l2, odl_l3, onos, bgpvpn, ovs, kvm, kvm_ovs. From deployment point of view, 5 working, 1 not working, 2 not activated/attempted yet. Deployed successfully: openstack, odl_l2, odl_l3, onos, bgpvpn Not successfully deployed yet: ovs. Not activated/attempted: kvm, kvm_ovs.
Fuel has Fuel7 on stable/brahmaputra branch and Fuel8 uplift work is WIP on master branch.
Joid: 4 scenarios on Jenkins: pure openstack, odl_l2, onos, and ocl. From deployment point of view, 3 working, 1 not activated/attempted yet. Deployed successfully: openstack, odl_l2, onos. Not successfully deployed yet: none. Not activated/attempted: ocl (removed from this installer)
ARMband:
Availability：（Qiao Fu) HA project has got all the rst file merged and ready for release. Will upload a full set of rst file for the two release doc later today.
Bottlenecks: (Hongbo Tian)
Copper: (Bryan Sullivan) at risk
Doctor (Ryota, Ildiko)
IPv6: (Bin Hu) -
LSOAPI (Kevin Leuhrs) at risk
NFV4KVM: (Don Dugger, Raghuveer Reddy)
ONOSFW: (Ashlee Young)
OpenvSwitch (Mark Gray)
OPNFVdocs (Chris Price) Issues: Docs working on central content. Working with projects on project content. Patches outstanding on many projects. Awaiting final scenario list to progress to completion. Working with projects on project content. Patches outstanding on many projects. Unable to code freeze until installers freeze. ONOSFW reviewing reviewing onosfw docs. Projects are working on docs, some feature projects are waiting for the integration to be complete to document that for the config guide
OVNO (Stuart Mackie) Apex still working on patch for plugin
Parser
Prediction (Hai Liu)
Qtip (Vikram)
SFC (Brady Johnson)
SDNVPN (Tim Irnich)
SFQM (Maryam Tahhan)
StorPerf (Mark Beierle)
=============================================================================================================== Action ItemsUnderlined Text Action: Owner: Status:
Action: Owner: Status:
Action: Owner: Status:
1/25/16
Countdown to Release: 6 days __**Upcoming deadlines:**__ =============================================================================================================== Minutes from daily standup 1/25/16
Top Accomplishments & Important Updates since last report
.Fuel made great strides over the weekend. ONOS scenarios working. Still unable to freeze.
.Yardstick successfully ran ONOS scenario with Compass.
.Functest now relatively stable from CI perspective. ONOS scenarios working.
===============================================================================================================
Top Issues
Issue: We are at high risk of missing release date. Test cycles for each scenario take a significant amount of time. Even if all installers & scenarios are working properly it takes 7 days to run test through them all. Owner: Debra Status: Additional options being proposed: 1. Release with what we have and move everything else to C Release 2. Delay release for up to 3 weeks 3. staggered/phased release of scenario support as a potential. Phase 1: Release what we can on Feb 2; Phase 2: Release what is ready 1-2 weeks later, Phase 3: final release 1-2 weeks after phase 2. These have been brought up to the TSC via email for consideration next TSC meeting.
Issue: Functest & Yardstick do not produce stable results. Troubleshooting scenarios. Need assistance from installer & feature project teams to fault isolation and fix. Owner: Morgan Richomme (Functest), Ana Cuna (Yardstick) Status: Functest now relatively stable from CI perspective (except just joid). Still work on the different vPing (and then vIMS) testcases (that need ssh access from the VMs). Onos OK.
Yardstick:ONOS - successfully run with Compass. ODL scenario - debugging on-going, Yardstick will not run with ODL. Narrowing down subsystems involved. Working on SFC test cases. Working on influxdB+grafana
Issue: Getting RC0 out Owner: Debra Status: Installer code freeze extended due to Fuel trying to stabilize with upstream. Compass also troubleshooting some functest and other instability with the code base, need some extra time in getting Compass code stable. Apex, JOID ready to freeze
Issue: Fuel is not ready to code freeze. Upstream Fuel is not stable. Owner: Jonas Bjurel Status: Requesting code freeze delay until 1/29 and suggesting release slip 3 weeks
Issue: A large number of projects had code file licensing issues (either missing licenses or non-Apache licenses). Owner: Ray Paik Status:Projects have made good progress on getting patches in to fix the files. Scott is making another license scan to see what if anything is outstanding. Thank you to all who have responded to this request in such a timely manner.
Issue: OpenContrail does not deploy Owner: Stuart Mackey Status: Apex still trying to get contrail to work, running into some openstack version issues where some contrail pieces rely on juno
Issue: Scenarios are frozen, but several not working yet, or have not been tried Owner: test teams Status: Next step is creating Jenkins jobs and testing each scenario. In process with test teams leading. https://wiki.opnfv.org/brahmaputra_testing_page#candidate_scenarios and https://wiki.opnfv.org/brahmaputra_testing_page
Issue: NFV4KVM – plugin does not work with Fuel, The NFV4KVM does install a working kernel on the compute node. Owner:Don Dugger/Raghuveer Reddy Status: 1/22 no new update. Fuel has bigger issues at the moment. We have discovered that OVS appears to require kernel modules that are not there. Working on creating a fix to the Fuel plugin creation scripts that will fix this problem, potentially will have a fix before code freeze, but certainly should be able to solve the problem for an SR1 release. =============================================================================================================== Pod Status:
All pods up and running
=============================================================================================================== Scenario Status: Scenarios are frozen. No feature project has reported that their scenarios are missing or unsupported. Candidate scenarios are having Jenkins jobs created for them. Test teams troubleshooting scenarios and test software.
https://wiki.opnfv.org/brahmaputra_testing_page#candidate_scenarios and https://wiki.opnfv.org/brahmaputra_testing_page =============================================================================================================== Release Test Status:
Functest (Morgan Richomme) Current situation * Functest Functest tests more stable, Some tests run in CI over the week end, Functest now relatively stable from CI perspective (except just joid I had to cancel). Still work on the different vPing (and then vIMS) testcases (that need ssh access from the VMs). Onos OK. ODL, maybe an update to be done, but we are integrating upstream suite, and it is not fully clear that the suite has been adapted to new version of ODL. https://wiki.opnfv.org/doku.php?id=vPing_brahmaputra_page https://wiki.opnfv.org/doku.php?id=vIMS_brahmaputra_page Test DB migrated this morning (9 UTC) http://testresults.opnfv.org/testapi/pods Still work ahead to tune some cases, and collect the error causes
Yardstick (Ana Cunha) ONOS first succesfull run on Compass, continue troubleshooting ODL scenarios
=============================================================================================================== Installer Status:
Apex (Tim Rozet): Still trying to get contrail to work, running into some openstack version issues where some contrail pieces rely on juno
JOID (Narinder Gupta) JOID we are ready for code freeze as last pending issue related to Orange pod has been fixed
Compass4nfv (Weidong Shao) there were some functest and other instability with the code base, and we will need some extra time in getting Compass code stable
Fuel (Jonas Bjurel) Tickets and backlog jira dashboard 1) Build
The building issues have been resolved to the point that it builds flawlessly in the OPNFV CI engine. Daily masters are produced every night since Friday. Work ongoing with getting it to build inside Ericsson corporate network. Stefan probably nailed it as it seems to be an issue with the 3.13.0-66 kernel.
2) Deployment. I have successfully deployed the daily master image from Friday CI build: http://artifacts.opnfv.org/fuel/opnfv-2016-01-22_16-05-16.iso It works more or less flawlessly, a few minor comments: - The ssh service is not started as early as it used to do, it is not started when Fuelmenu
pops up which will be an issue for auto deploy. # service sshd start fixes it
- Fuelmenu will sometimes tell you that the URLs for the bootimage cannot be found,
if you make sure you can reach url resources eg. Through open a consol from fuelmenu and type nslookup www.google.com – just don’t care, select “save & continue” and the “quit without saving” you should be good.
- Selecting centos boot image didn’t work for me
3) Early this week we need to prioritize getting autodeploy to work.
But to my mind, this seems to be stable enough for plugin work. Please start commit rebased plugins to Master.
- As I have indicated to TSC (Chris and Frank), my judgement is that we need another 3 week after Feb 2 to complete the things and have them in a good shape. The other option is to release with Fuel7/Kilo which is quite stable and a good veichle for developing stabilizing the scenarios and the test pipe-line.
- Issue with Jenkins server caused delays on Sunday but were resolved and builds are working again.
=============================================================================================================== Project Updates: (as provided)
ARMband:
Bottlenecks: (Hongbo Tian)
Copper: (Bryan Sullivan) at risk
Doctor (Ryota, Ildiko) Apex patch for AODH is merged. Doctor will make the release.
IPv6: (Bin Hu) -
LSOAPI (Kevin Leuhrs) at risk
NFV4KVM: (Don Dugger, Raghuveer Reddy)
ONOSFW: (Ashlee Young)
OpenvSwitch (Mark Gray)
OPNFVdocs (Chris Price) Issues: Docs working on central content. Working with projects on project content. Patches outstanding on many projects. Awaiting final scenario list to progress to completion. Working with projects on project content. Patches outstanding on many projects. Unable to code freeze until installers freeze. ONOSFW reviewing reviewing onosfw docs. Projects are working on docs, some feature projects are waiting for the integration to be complete to document that for the config guide
OVNO (Stuart Mackie) Apex
Parser
Prediction (Hai Liu)
Qtip (Vikram)
SFC (Brady Johnson) Apex patch is merged, SFC scenario is supported in the release
SDNVPN (Tim Irnich)
SFQM (Maryam Tahhan)
StorPerf (Mark Beierle)
=============================================================================================================== Action ItemsUnderlined Text Action: Owner: Status:
Action: Owner: Status:
Action: Owner: Status:
