This is an old revision of the document!
2/11/16
Countdown to Release: 15 days __**Upcoming deadlines:**__ TSC agreed upon a phased release with primary release date of 2/26/16. All scenarios that are stable and tested ready to release on that date will be released. Any currently in-plan scenarios that are not ready on 2/26/16 will be allowed to join a follow-on release on March 28th; if deemed necessary a 3rd release could follow in late-April =============================================================================================================== Minutes from daily standup 2/11/16
Top Accomplishments & Important Updates since last report
.JOID: using orange pod2 we have mutiple successful run now for install/functest/yardstick .Still no scenarios that reach quality goal of 4 successful runs functest + yardstick ===============================================================================================================
Top Issues
Issue: Functest & Yardstick do not produce stable results. Troubleshooting scenarios. Need assistance from installer & feature project teams to fault isolation and fix. Owner: Morgan Richomme (Functest), Ana Cuna (Yardstick) Status:
Yardstick: -working on bug fixes, docs and troubleshooting
Issue: A large number of projects had code file licensing issues (either missing licenses or non-Apache licenses). Owner: Ray Paik Status:after re-run 3 projects have licensing files that need review and approval by OPNFV Board Legal team
Issue: NFV4KVM – plugin does not work with Fuel, The NFV4KVM does install a working kernel on the compute node. Owner:Don Dugger/Raghuveer Reddy Status: from Fuel team: This project is focusing on yardstick SLA test-cases aside of a true deployment (using bridges instead of OVS). They install a new kernel for which tere is no OVS kernel driver in the packages. All though they’re working on it. I’m not sure they will make it for our time plan – not even sure we need to wait for it since it is as a “technology project” verified by yardstick, but not yet deploying in a real environment. I would vote this to included in SR1 of Fuel@OPNFV B-release. =============================================================================================================== Pod Status:
Request Jira ticket be logged for any and all issues as email responses are difficult to keep track of. We need clearer communication.
All pods are currently working and in use. LF Pod 3 is up and running. Yardstick requests that this pod remain reserved for troubleshooting Yardstick for at least the next 2 weeks.
=============================================================================================================== Scenario Status: All scenarios which features they support and which have passed/failed/not run yet in functest is located https://wiki.opnfv.org/functextnexttaks#brahmaputra_summary Have asked for a similar view to be constructed for Yardstick
scenario’s which are approaching stable across installers are: nosdn ,onos ,odl(be) - once integrated, odll3(be), ova, doctor, sdnvpn. We are reasonably confident of having those release ready in a couple of weeks.
To be stabilised include SFC, KVM and OCL which could potentially come in under a stable release or in C. Dependant on the project teams efforts and targets.
=============================================================================================================== Release Test Status:
Functest (Morgan Richomme) .vPing/SSH apex fuel/odl_l3 troubleshooting in progress
- seems that workaround on non parallelisation of tests in Tempest lead to better results, wait and see iterations for confirmation
.last apex run 92% OK and joid run 98% .new doc to write for the results, wait for Chris merge before starting with scneario already ready (compass scenario)
Yardstick (Ana Cuna) .odl l2 with joid: fault in orange pod 2
- seems not to be yardstick problem: Cannot connect to the Docker daemon. Docker daemon is running on this host.
- continue troubleshooting
- functest last run was failed but previous were OK, we also got an unexpected failure at the end of last functest run…I need to connect to the machine because no clear message. We got 2 runs through before this latest issue
.next 24 hours: troubleshooting and fixing docs
=============================================================================================================== Installer Status:
Apex (Tim Rozet): .Apex Still in debug mode, working specifically on the ODL-L3 vPing debug today .Other bugs are still in flight and making progress .Waiting info from OpenContrail for fixes needed re: Liberty
JOID (Narinder Gupta) .in orange pod2 we have mutiple successful run now for install/functest/yardstick even few test cases are failing but we are good from installer prospective. Also intel pod5 and intel pod6 installs were successfull but functest are failing on test cases which needs to be debugged
Compass4nfv (Weidong Shao)
Fuel (Jonas Bjurel) Tickets and backlog jira dashboard .Everything is in, don't expect anything more to go in, Docs ready. All efforts on troubleshooting the ssh issue with Yardstick and ODL. .Fuel resources moving into yardstick.
=============================================================================================================== Project Updates: (as provided) Please find current Jira dashboards for each project here https://wiki.opnfv.org/releases/brahmaputra/release_plan#current_jira_dashboards
Availability：（Qiao Fu)
Bottlenecks: (Hongbo Tian)
Copper: (Bryan Sullivan)
Doctor (Ryota, Ildiko)
HA(Fu Qaio)
IPv6: (Bin Hu) - .ready for release, will help Yardstick analyze ping 6 results
NFV4KVM: (Don Dugger, Raghuveer Reddy)
ONOSFW: (Ashlee Young) .helping JOID get some charm fixes committed .will help analyze the test results once Yardstick has 4 good runs
OpenvSwitch (Mark Gray)
OPNFVdocs (Chris Price) .pushing functest results doc patch, working on user guide, working on config guide, working on overview docs. .also have an activity to improve pdf toolchain formatting .We have a workaround for table formatting, need to check. Also looking at blank pages.
OVNO (Stuart Mackie) We appear to have OpenContrail working on Liberty. Testing continues today Will let Apex know what changes were made from standard V2.21. Join can test with JOID as well?
Parser
Promise (Gerald Kunzmann)
Prediction (Hai Liu)
Qtip (Vikram)
SFC (Brady Johnson)
SDNVPN (Tim Irnich)
SFQM (Maryam Tahhan)
StorPerf (Mark Beierle)
=============================================================================================================== Action ItemsUnderlined Text Action: Create a dashboard view of Yardstick that shows all scenarios, all tests run per scenerio, all tests remaining to be run, and all tests that fail; Include feature support Owner: Ana Cuna Status: assigned 2/4/16
2/10/16
Countdown to Release: 16 days __**Upcoming deadlines:**__ TSC agreed upon a phased release with primary release date of 2/26/16. All scenarios that are stable and tested ready to release on that date will be released. Any currently in-plan scenarios that are not ready on 2/26/16 will be allowed to join a follow-on release on March 28th; if deemed necessary a 3rd release could follow in late-April =============================================================================================================== Minutes from daily standup 2/10/16
Top Accomplishments & Important Updates since last report
.First iteration of doctor on apex scenario OK from CI perspective
.We appear to have OpenContrail working on Liberty.
===============================================================================================================
Top Issues
Issue: Functest & Yardstick do not produce stable results. Troubleshooting scenarios. Need assistance from installer & feature project teams to fault isolation and fix. Owner: Morgan Richomme (Functest), Ana Cuna (Yardstick) Status: -regression on Tempest suite (investigation to know if it is in Functest or upstream) continues.
- ODL Beryllium resolves blocking vping and SSH issues. TSC voted to move from ODL Lithium to ODL Beryllium for this release in order to resolve blocking issues
- working through scenarios in logical order coordinated with Yardstick, deployment and feature projects as needed to allow for additional testing & troubleshooting
- JOID regression testing with Be is in progress, all other installers report it is supported
Yardstick: -working on bug fixes, docs and troubleshooting
Issue: A large number of projects had code file licensing issues (either missing licenses or non-Apache licenses). Owner: Ray Paik Status:after re-run 3 projects have licensing files that need review and approval by OPNFV Board Legal team
Issue: NFV4KVM – plugin does not work with Fuel, The NFV4KVM does install a working kernel on the compute node. Owner:Don Dugger/Raghuveer Reddy Status: from Fuel team: This project is focusing on yardstick SLA test-cases aside of a true deployment (using bridges instead of OVS). They install a new kernel for which tere is no OVS kernel driver in the packages. All though they’re working on it. I’m not sure they will make it for our time plan – not even sure we need to wait for it since it is as a “technology project” verified by yardstick, but not yet deploying in a real environment. I would vote this to included in SR1 of Fuel@OPNFV B-release. =============================================================================================================== Pod Status:
Request Jira ticket be logged for any and all issues as email responses are difficult to keep track of. We need clearer communication.
All pods are currently working and in use. LF Pod 3 is up and running. Yardstick requests that this pod remain reserved for troubleshooting Yardstick for at least the next 2 weeks.
=============================================================================================================== Scenario Status: All scenarios which features they support and which have passed/failed/not run yet in functest is located https://wiki.opnfv.org/functextnexttaks#brahmaputra_summary Have asked for a similar view to be constructed for Yardstick
scenario’s which are approaching stable across installers are: nosdn ,onos ,odl(be) - once integrated, odll3(be), ova, doctor, sdnvpn. We are reasonably confident of having those release ready in a couple of weeks.
To be stabilised include SFC, KVM and OCL which could potentially come in under a stable release or in C. Dependant on the project teams efforts and targets.
=============================================================================================================== Release Test Status:
Functest (Morgan Richomme) .2 new scenario candidate for "Release ready" status last night fuel/nosdn-ovs and joid/odl_l2 (on Orange POD), waiting for iteration to confirm .Still having vPing issues on Onos and old_l3 scenario .Making progress on Tempest. .Docs updated .First iteration of doctor on apex scenario OK from CI perspective (did not discuss with Ryota on the test content) POD to troubleshoot the onos and l3 scenbario's? -Ericsson POD2 could be used for troubleshooting. Jose has access to it. LF POD1/apex can also be used. .Over next 24 hours ee will continue on the 2 last items vPing and Tempest
Yardstick (Ana Cuna) .problems detected with yardstick installation last night - fix merged .new server for yardstick db is made available from LF, need access rights and user replication to continue, connection refused from yardstick data to previous server - no data was recorded last night. Aric has been contacted .request that no one uses the new server other than yardstick, for next two weeks, to avoid disturbances .for next 24 hours: check runs, fix problems and work on database
=============================================================================================================== Installer Status:
Apex (Tim Rozet): .a few more bugs to fix that are listed in JIRA .most are already in flight .top priority in next 24 hours is figuring out why Functest and yardstick are not succeeding .Intel Pod 7 available; we have access to it and are getting the env verified to hook back up to jenkins
JOID (Narinder Gupta) .we had successfull run on orange pod2 for all deploy, funtest, yardstick. .Functest reported only 4 temptest failure. Intel pod5 we are seeing more failure which needs to be debugged. Morgan might get access to intel pods soon to get it debugged further
Compass4nfv (Weidong Shao)
Fuel (Jonas Bjurel) Tickets and backlog jira dashboard Continue running & troubleshooting scenarios.
o NFVKVM: This project is focusing on yardstick SLA test-cases aside of a true deployment (using bridges instead of OVS). They install a new kernel for which tere is no OVS kernel driver in the packages. All though they’re working on it. I’m not sure they will make it for our time plan – not even sure we need to wait for it since it is as a “technology project” verified by yardstick, but not yet deploying in a real environment. I would vote this to included in SR1 of Fuel@OPNFV B-release.
o SFC, though we have been integrating SFC base features into fuel, the SFC team have gone with architecture that requires tacker as an orchestrater (although tacker is explicitly outside of the B-release). The SFC team has started on a Tacker Fuel plugin – and as the plug-in architechture provides proper fensing to the core of Fuel, we can wait and see if they come with something that works. Their deadline is Thu Feb 11 – but I don’t expect them to make it, hope I’m cut by surprise!
Reporting:I have included my reporting into this ether-pad: https://etherpad.opnfv.org/p/steps_to_brahmaputra as a link
Fuel@OPNFV’s input to the status and the plans: - This is our end-game status, that will be updated daily: https://wiki.opnfv.org/fuel_opnfv/b_release_status
- This is our end-game plan, updated twice a week: https://wiki.opnfv.org/fuel_opnfv_b-wp
=============================================================================================================== Project Updates: (as provided) Please find current Jira dashboards for each project here https://wiki.opnfv.org/releases/brahmaputra/release_plan#current_jira_dashboards
Availability：（Qiao Fu)
Bottlenecks: (Hongbo Tian)
Copper: (Bryan Sullivan)
Doctor (Ryota, Ildiko)
HA(Fu Qaio)
IPv6: (Bin Hu) -
NFV4KVM: (Don Dugger, Raghuveer Reddy)
ONOSFW: (Ashlee Young) Most of the team is still on holiday, but looking into vPing issues and making an update to docs
OpenvSwitch (Mark Gray)
OPNFVdocs (Chris Price) working on release notes and results publishing and structure today, will continue tomorrow
OVNO (Stuart Mackie) We appear to have OpenContrail working on Liberty. Testing continues today Will let Apex know what changes were made from standard V2.21. Join can test with JOID as well?
Parser
Promise (Gerald Kunzmann)
Prediction (Hai Liu)
Qtip (Vikram)
SFC (Brady Johnson)
SDNVPN (Tim Irnich)
SFQM (Maryam Tahhan)
StorPerf (Mark Beierle)
=============================================================================================================== Action ItemsUnderlined Text Action: Create a dashboard view of Yardstick that shows all scenarios, all tests run per scenerio, all tests remaining to be run, and all tests that fail; Include feature support Owner: Ana Cuna Status: assigned 2/4/16
2/8/16-2/9/16
Countdown to Release: 17 days __**Upcoming deadlines:**__ TSC agreed upon a phased release with primary release date of 2/26/16. All scenarios that are stable and tested ready to release on that date will be released. Any currently in-plan scenarios that are not ready on 2/26/16 will be allowed to join a follow-on release on March 28th; if deemed necessary a 3rd release could follow in late-April =============================================================================================================== Minutes from daily standup 2/9/16
Top Accomplishments & Important Updates since last report
.After investigation with deployment tools, the TSC approved a Change Request to release with ODL Beryllium instead of ODL Lithium.
.Functest
- Continuing to work through scenarios. Tempest functionality remains an issue in some scenarios.here Keep in mind that this is a static page, if you need more current information then visit the test dashboard
===============================================================================================================
Top Issues
Issue: We have passed target release date of Feb 2. Owner: Debra Status: TSC voted to target a phased release schedule with first release target for Feb 26. Additional release dates for scenarios that need more time are proposed for late March and, if needed, late April.
Issue: Functest & Yardstick do not produce stable results. Troubleshooting scenarios. Need assistance from installer & feature project teams to fault isolation and fix. Owner: Morgan Richomme (Functest), Ana Cuna (Yardstick) Status: -regression on Tempest suite (investigation to know if it is in Functest or upstream) continues.
- ODL Beryllium resolves blocking vping and SSH issues. TSC voted to move from ODL Lithium to ODL Beryllium for this release in order to resolve blocking issues
- working through scenarios in logical order coordinated with Yardstick, deployment and feature projects as needed to allow for additional testing & troubleshooting
- JOID regression testing with Be is in progress, all other installers report it is supported
Yardstick: -working on bug fixes, docs and troubleshooting
Issue: A large number of projects had code file licensing issues (either missing licenses or non-Apache licenses). Owner: Ray Paik Status:after re-run 3 projects have licensing files that need review and approval by OPNFV Board Legal team
Issue: Scenarios are frozen, but significant number not working yet, or have not been tried Owner: test teams Status: Closed. Test projects are stabilizing and now running through previously untested scenarios. Continuing to run & debug scenarios in coordinated manner.
Issue: NFV4KVM – plugin does not work with Fuel, The NFV4KVM does install a working kernel on the compute node. Owner:Don Dugger/Raghuveer Reddy Status: no new update. Fuel has bigger issues at the moment. We have discovered that OVS appears to require kernel modules that are not there. Working on creating a fix to the Fuel plugin creation scripts that will fix this problem, potentially will have a fix before code freeze, but certainly should be able to solve the problem for an SR1 release. =============================================================================================================== Pod Status:
Request Jira ticket be logged for any and all issues as email responses are difficult to keep track of. We need clearer communication.
All pods are currently working and in use.
=============================================================================================================== Scenario Status: All scenarios which features they support and which have passed/failed/not run yet in functest is located https://wiki.opnfv.org/functextnexttaks#brahmaputra_summary Have asked for a similar view to be constructed for Yardstick
scenario’s which are approaching stable across installers are: nosdn ,onos ,odl(be) - once integrated ,odll3(be) ,ova, doctor, sdnvpn. We are reasonably confident of having those release ready in a couple of weeks.
To be stabilised include SFC, KVM and OCL which could potentially come in under a stable release or in C. Dependant on the project teams efforts and targets.
=============================================================================================================== Release Test Status:
Functest (Morgan Richomme)
- we fixed a patch that triggers some regression on Functest
- we are now analyzing the errors linked to Tempest on the different installers
- please review test status on wiki https://wiki.opnfv.org/functextnexttaks#brahmaputra_summary
- we identified during the weekly meeting the list miles (some vPing on odl_l3 and onos scenario) + boosting Tempest results on apex, fuel and joid
- Apex: Tempest failed to run so we don't actually know if it fixed it or not Vping passed on L3 ODL and yardstick passed on ODL L2
- Kicking off a new Apex run now that Be is the target ODL version
example of log pushed to artifact (here tempest log on apex)
Yardstick:ONOS -
- continuing rebasing for fuel8
*=============================================================================================================== Installer Status: Apex (Tim Rozet): JOID (Narinder Gupta) Continue to work on OpenContrail issues. It deploys now but doesn't run with OpenStack Liberty. We migrated to Be yesterday and last night run was success from installer prospective Compass4nfv (Weidong Shao) Fuel (Jonas Bjurel) Tickets and backlog jira dashboard Overall status on Fuel and its OPNFV feature collaboration projects: Since the end of last working week, Fuel has converged dramatically and we’re getting consistent positive data from the deploys. We have performed massive runs over the week-end, and I can’t find one deployment with known working scenarios that has failed yet, we have performed 20 official OPNFV deploys - distributed over the known good scenarios and master/stable branch (The branches are currently almost identical) – and no single one failed. In addition we have ran 10 unofficial deploys with Fuel health check running (on my pretty hefty workstation), and no single one failed there either! Without going into details, I’m positively surprised about the consistency! A few issues: There are 2 issues though: - We seem to have an issue with LF-POD2 setup, causing func-test and yard-stick to fail, hitting the stability for stable. I will swap the ci runs such that stable/Brahmaputra runs on Ericsson-POD2 and master on LF-POD2 and after that we will start troubleshooting on LF-POD2. We need both, but statistics on stable/brahmaputra is more important than that on master right now! - We have issues with a few plugins: o BGP-VPN: No big deal, there is a patch for it and we will merge it to master tomorrow, and to stable on Tue. o NFVKVM: This project is focusing on yardstick SLA test-cases aside of a true deployment (using bridges instead of OVS), they install a new kernel for which tere is no OVS kernel driver in the packages. All though they’re working on it I’m not sure they will make it for our time plan – not even sure we need to wait for it since it is as a “technology project” verified by yardstick, but not yet deploying in a real environment. I would vote this to included in SR1 of Fuel@OPNFV B-release. o SFC, though we have been integrating SFC base features into fuel, the SFC team have gone with architecture that requires tacker as an orchestrater (although tacker is explicitly outside of the B-release). The SFC team has started on a Tacker Fuel plugin – and as the plug-in architechture provides proper fensing to the core of Fuel, we can wait and see if they come with something that works. Their deadline is Thu Feb 11 – but I don’t expect them to make it, hope I’m cut by surprise! Reporting: I have included my reporting into this ether-pad: https://etherpad.opnfv.org/p/steps_to_brahmaputra as a link BTW: Would it be possible to retrofit/rebase the Brahmaputra release WIKI to something where it is easy to find what is on our top prio today – at least I have difficulties to follow it right now. Fuel@OPNFV’s input to the status and the plans: - This is our end-game status, that will be updated daily: https://wiki.opnfv.org/fuel_opnfv/b_release_status - This is our end-game plan, updated twice a week: https://wiki.opnfv.org/fuel_opnfv_b-wp =============================================================================================================== Project Updates: (as provided) Please find current Jira dashboards for each project here https://wiki.opnfv.org/releases/brahmaputra/release_plan#current_jira_dashboards Availability：（Qiao Fu) Bottlenecks: (Hongbo Tian) Copper: (Bryan Sullivan) Doctor (Ryota, Ildiko) HA(Fu Qaio) IPv6: (Bin Hu) - NFV4KVM: (Don Dugger, Raghuveer Reddy) ONOSFW: (Ashlee Young) OpenvSwitch (Mark Gray) OPNFVdocs (Chris Price) OVNO (Stuart Mackie) Parser Promise (Gerald Kunzmann) Prediction (Hai Liu) Qtip (Vikram) SFC (Brady Johnson) SDNVPN (Tim Irnich) SFQM (Maryam Tahhan) StorPerf (Mark Beierle) =============================================================================================================== Action ItemsUnderlined Text Action: Create a dashboard view of Yardstick that shows all scenarios, all tests run per scenerio, all tests remaining to be run, and all tests that fail; Include feature support Owner: Ana Cuna Status: assigned 2/4/16 —————————————————————————————————————————————————————————————————————————————— 2/5/16 Countdown to Release: TBD days __**Upcoming deadlines:**__ under revision with proposed milestones dates until we work out details. TSC approved release delay but chose to wait one more week before setting date in hopes that by then some issues will be resolved allowing for better estimations of time; Etherpad currently proposed release date is 3/8; we also have a matrix of the scenarios and their pass/fail/stable status which will help drive decisions =============================================================================================================== Minutes from daily standup 2/5/16 Top Accomplishments & Important Updates since last report .Yardstick runs successfully with ODL Beryllium without work around; Investigating impact to other projects if we switch to Be from Li for all ODL scenarios, this will be discussed on the email lists .Functest -A full list of tests they have run per scenario, which tests have passed or failed, which features are supported by those scenarios, and which remain to be tested for the first time is available here Keep in mind that this is a static page, if you need more current information then visit the test dashboard -according to success criteria, we have already 1 validated scenario: compass/onos -another one just needs iterations to be validated: compass/nosdn =============================================================================================================== Top Issues Issue: have passed intended release date. Test cycles for each scenario take a significant amount of time. Even if all installers & scenarios are working properly it takes 7 days to run test through them all. Need to set a plan to identify new release date Owner: Debra Status: TSC voted to delay release rather than trying to phase release. Debra & key community members to identify remaining work and develop reasonably accurate schedule to propose to TSC. On Tuesday TSC was not comfortable setting a date while such significant issues remained in troubleshooting status for test projects. Major breakthroughs have been achieved in the past 2 days making better reasoned schedule estimates possible. Current estimate is Mar 8 release; Functest has created a wiki page that gives good overview of all tests run in each scenario and which features these scenarios support. Working on same visibility for Yardstick. These, along with test runs through all scenarios will help us identify what is ready to release now and what might need more work. Proposal is made for all hw resources to be applied to stable branch testing until this is complete Issue: Functest & Yardstick do not produce stable results. Troubleshooting scenarios. Need assistance from installer & feature project teams to fault isolation and fix. Owner: Morgan Richomme (Functest), Ana Cuna (Yardstick) Status: -regression on Tempest suite (investigation to know if it is in Functest or upstream) continues. -working through scenarios in logical order coordinated with Yardstick, deployment and feature projects as needed to allow for additional testing & troubleshooting Yardstick: -odl scenarios - identified that with ODL Beryllium, Yardstick passes. With ODL Lithium a workaround is required. Investigating impact on other projects if the supported version of ODL is switched ot Be for all ODL scenarios -working on bug fixes, docs and troubleshooting Issue: A large number of projects had code file licensing issues (either missing licenses or non-Apache licenses). Owner: Ray Paik Status:Scott discussed status during TSC meeting; Chris to post instructions online for license inclusion. Projects have made good progress on correcting issues. Scott rerunning scan to see if any projects need to address additional issues Issue: Scenarios are frozen, but significant number not working yet, or have not been tried Owner: test teams Status: Continuing to run & debug scenarios in coordinated manner, see above status Issue: NFV4KVM – plugin does not work with Fuel, The NFV4KVM does install a working kernel on the compute node. Owner:Don Dugger/Raghuveer Reddy Status: no new update. Fuel has bigger issues at the moment. We have discovered that OVS appears to require kernel modules that are not there. Working on creating a fix to the Fuel plugin creation scripts that will fix this problem, potentially will have a fix before code freeze, but certainly should be able to solve the problem for an SR1 release. =============================================================================================================== Pod Status: Request Jira ticket be logged for any and all issues as email responses are difficult to keep track of. We need clearer communication. Pharos lab status =============================================================================================================== Scenario Status: All scenarios which features they support and which have passed/failed/not run yet in functest is located https://wiki.opnfv.org/functextnexttaks#brahmaputra_summary Have asked for a similar view to be constructed for Yardstick =============================================================================================================== Release Test Status: Functest (Morgan Richomme) -no update since yesterday. please review test status on wiki https://wiki.opnfv.org/functextnexttaks#brahmaputra_summary - -http://artifacts.opnfv.org/logs/functest/opnfv-jump-1/brahmaputra/2016-02-03_19-05-12/brahmaputra/tempest/tempest.log example of log pushed to artifact (here tempest log on apex) -http://artifacts.opnfv.org/logs/functest/opnfv-jump-1/brahmaputra/2016-02-03_19-05-12/brahmaputra/rally/opnfv-neutron.html Yardstick:ONOS - -breakthrough with ODL scenarios using ODL Beryllium -continuing rebasing in progress for fuel8 -working on building easy to read status report *=============================================================================================================== Installer Status:
Apex (Tim Rozet):
JOID (Narinder Gupta)
Compass4nfv (Weidong Shao)
Fuel (Jonas Bjurel) Tickets and backlog jira dashboard
=============================================================================================================== Project Updates: (as provided) Please find current Jira dashboards for each project here https://wiki.opnfv.org/releases/brahmaputra/release_plan#current_jira_dashboards
Availability：（Qiao Fu)
Bottlenecks: (Hongbo Tian)
Copper: (Bryan Sullivan)
Doctor (Ryota, Ildiko)
HA(Fu Qaio)
IPv6: (Bin Hu) -
NFV4KVM: (Don Dugger, Raghuveer Reddy)
ONOSFW: (Ashlee Young)
OpenvSwitch (Mark Gray)
OPNFVdocs (Chris Price)
OVNO (Stuart Mackie)
Parser
Promise (Gerald Kunzmann) We have successfully verified Promise in both Joid and Fuel scenarios: https://wiki.opnfv.org/promise_brahmaputra_page Promise functest results are already pushed to the functest database http://testresults.opnfv.org/testapi/results?case=promise (you will still see some failures, but those are due to issues with the scenarios, not Promise code)
Prediction (Hai Liu)
Qtip (Vikram)
SFC (Brady Johnson)
SDNVPN (Tim Irnich)
SFQM (Maryam Tahhan)
StorPerf (Mark Beierle)
=============================================================================================================== Action ItemsUnderlined Text Action: Create a dashboard view of Yardstick that shows all scenarios, all tests run per scenerio, all tests remaining to be run, and all tests that fail; Include feature support Owner: Ana Cuna Status: assigned 2/4/16
2/3/16 and 2/4/16
Countdown to Release: TBD days __**Upcoming deadlines:**__ under revision with proposed milestones dates until we work out details. TSC approved release delay but chose to wait one more week before setting date in hopes that by then some issues will be resolved allowing for better estimations of time; Etherpad currently proposed release date is 3/8; we also have a matrix of the scenarios and their pass/fail/stable status which will help drive decisions =============================================================================================================== Minutes from daily standup 2/3/16 2/4/16
Top Accomplishments & Important Updates since last report
.Excellent progress by test teams today in overcoming blocking issues
.Functest
- becoming more stable on scenarios that it has been able to run.They successfully identified and fixed the issue they were having with vPing and SSH on ODL scenarios.
- A full list of tests they have run per scenario, which tests have passed or failed, which features are supported by those scenarios, and which remain to be tested for the first time is available here Keep in mind that this is a static page, if you need more current information then visit the test dashboard
- according to success criteria, we have already 1 validated scenario: compass/onos
- another one just needs iterations to be validated: compass/nosdn
.Yardstick
- have a work around in place for networking issues with ODL scenarios, these issues seem to be Lithium related, troubleshooting continues
===============================================================================================================
Top Issues
Issue: have release date. Test cycles for each scenario take a significant amount of time. Even if all installers & scenarios are working properly it takes 7 days to run test through them all. Owner: Debra Status: TSC voted to delay release rather than trying to phase release. Debra & key community members to identify remaining work and develop reasonably accurate schedule to propose to TSC. On Tuesday TSC was not comfortable setting a date while such significant issues remained in troubleshooting status for test projects. Major breakthroughs have been achieved in the past 2 days making better reasoned schedule estimates possible. Current estimate is Mar 8 release
Issue: Functest & Yardstick do not produce stable results. Troubleshooting scenarios. Need assistance from installer & feature project teams to fault isolation and fix. Owner: Morgan Richomme (Functest), Ana Cuna (Yardstick) Status: -regression on Tempest suite (investigation to knwo if it is in Functest or upstream) continues.
- working through scenarios in logical order coordinated with Yardstick, deployment and feature projects as needed to allow for additional testing & troubleshooting
Yardstick: -odl scenarios - have workaround in place for vping and ssh which allows for further testing & debug
- working on bug fixes, docs and troubleshooting
Issue: A large number of projects had code file licensing issues (either missing licenses or non-Apache licenses). Owner: Ray Paik Status:Scott discussed status during TSC meeting; Chris to post instructions online for license inclusion. Projects have made good progress on correcting issues. Scott rerunning scan to see if any projects need to address additional issues
Issue: Scenarios are frozen, but significant number not working yet, or have not been tried Owner: test teams Status: Continuing to run & debug scenarios in coordinated manner
Issue: NFV4KVM – plugin does not work with Fuel, The NFV4KVM does install a working kernel on the compute node. Owner:Don Dugger/Raghuveer Reddy Status: no new update. Fuel has bigger issues at the moment. We have discovered that OVS appears to require kernel modules that are not there. Working on creating a fix to the Fuel plugin creation scripts that will fix this problem, potentially will have a fix before code freeze, but certainly should be able to solve the problem for an SR1 release. =============================================================================================================== Pod Status:
Request Jira ticket be logged for any and all issues as email responses are difficult to keep track of. We need clearer communication.
=============================================================================================================== Scenario Status: All scenarios which features they support and which have passed/failed/not run yet in functest is located https://wiki.opnfv.org/functextnexttaks#brahmaputra_summary Have asked for a similar view to be constructed for Yardstick
=============================================================================================================== Release Test Status:
Functest (Morgan Richomme)
- according to success criteria, we have already 1 validated scenario: compass/onos
- another one just needs iterations to be validated: compass/nosdn
- some are closed but the criteria of 90% of success on Tempest is not reached
- support on troubleshooting the Tempest error / scenario / installer welcome –
- morning errors on CI were not due to tests but to result pushed in artifacts (need config on jumphosts), now artifacts are pushed
- joid has better results on non CI POD (Orange POD2) regarding Tempest (92%), so just considering CI POD could be unfair for installer if PODs have some network issues
example of log pushed to artifact (here tempest log on apex)
Yardstick:ONOS -
- works for ONOS (Compass) and nosdn (Compass and Fuel7) - rebasing in progress for fuel8
- on-demand tests (not CI PODs) - HA, KVM - works
- Parser test case OK (will be run on weekly)
- vTC verified daily and weekly on Ericsson POD1
- ipv6 test case runs on Compass Huawei SC POD
- working on bug fixes, docs and troubleshooting
=============================================================================================================== Installer Status:
Apex (Tim Rozet):
JOID (Narinder Gupta)
Compass4nfv (Weidong Shao)
Fuel (Jonas Bjurel) Tickets and backlog jira dashboard
=============================================================================================================== Project Updates: (as provided) Please find current Jira dashboards for each project here https://wiki.opnfv.org/releases/brahmaputra/release_plan#current_jira_dashboards
Availability：（Qiao Fu)
Bottlenecks: (Hongbo Tian)
Copper: (Bryan Sullivan)
Doctor (Ryota, Ildiko)
HA(Fu Qaio)
IPv6: (Bin Hu) -
NFV4KVM: (Don Dugger, Raghuveer Reddy)
ONOSFW: (Ashlee Young)
OpenvSwitch (Mark Gray)
OPNFVdocs (Chris Price)
OVNO (Stuart Mackie)
Parser
Prediction (Hai Liu)
Qtip (Vikram)
SFC (Brady Johnson)
SDNVPN (Tim Irnich)
SFQM (Maryam Tahhan)
StorPerf (Mark Beierle)
=============================================================================================================== Action ItemsUnderlined Text Action: Owner: Status:
1/28/16
Countdown to Release: TBD days __**Upcoming deadlines:**__ =============================================================================================================== Minutes from daily standup 1/28/16
Top Accomplishments & Important Updates since last report
.TSC voted to move code freeze to Friday 1/29 and to extend release date (to be defined). Debra and key community members to evaluate remaining work and recommend new release date with solid schedule to achieve that.
===============================================================================================================
Top Issues
Issue: We are at high risk of missing release date. Test cycles for each scenario take a significant amount of time. Even if all installers & scenarios are working properly it takes 7 days to run test through them all. Owner: Debra Status: TSC voted to delay release rather than trying to phase release. Debra & key community members to identify remaining work and develop reasonably accurate schedule to propose to TSC.
Issue: Functest & Yardstick do not produce stable results. Troubleshooting scenarios. Need assistance from installer & feature project teams to fault isolation and fix. Owner: Morgan Richomme (Functest), Ana Cuna (Yardstick) Status:
Yardstick:ONOS - successfully run with Compass. ODL scenario - debugging on-going, Yardstick will not run with ODL. Narrowing down subsystems involved. Working on SFC test cases. Working on influxdB+grafana
Issue: Getting RC0 out Owner: Debra Status: Installer code freeze extended due to Fuel trying to stabilize with upstream. Compass also troubleshooting some functest and other instability with the code base, need some extra time in getting Compass code stable. Apex, JOID ready to freeze
Issue: Fuel is not ready to code freeze. Upstream Fuel is not stable. Owner: Jonas Bjurel Status: autodeploy issue, hopefully solved now. Testing. Scenarios being rebased
Issue: A large number of projects had code file licensing issues (either missing licenses or non-Apache licenses). Owner: Ray Paik Status:Projects have made good progress on getting patches in to fix the files. Scott is making another license scan to see what if anything is outstanding. Thank you to all who have responded to this request in such a timely manner.
Issue: OpenContrail does not deploy Owner: Stuart Mackey Status: Apex still trying to get contrail to work, running into some openstack version issues where some contrail pieces rely on juno
Issue: Scenarios are frozen, but several not working yet, or have not been tried Owner: test teams Status: Next step is creating Jenkins jobs and testing each scenario. In process with test teams leading. https://wiki.opnfv.org/brahmaputra_testing_page#candidate_scenarios and https://wiki.opnfv.org/brahmaputra_testing_page
Issue: NFV4KVM – plugin does not work with Fuel, The NFV4KVM does install a working kernel on the compute node. Owner:Don Dugger/Raghuveer Reddy Status: 1/22 no new update. Fuel has bigger issues at the moment. We have discovered that OVS appears to require kernel modules that are not there. Working on creating a fix to the Fuel plugin creation scripts that will fix this problem, potentially will have a fix before code freeze, but certainly should be able to solve the problem for an SR1 release. =============================================================================================================== Pod Status:
All pods up and running
=============================================================================================================== Scenario Status: Scenarios are frozen. No feature project has reported that their scenarios are missing or unsupported. Candidate scenarios are having Jenkins jobs created for them. Test teams troubleshooting scenarios and test software. Reviewed scenarios in table and noted those that will not be implemented this release
Apex: 5 scenarios on Jenkins: 4 work from deploy point of view odl_l2 is about 90 pass/10 fail, odl_l3 worse pass rate Not successfully deployed yet: ocl- working on it
Compass: 4 scenarios on Jenkins: pure openstack, odl_l2, onos, and ocl. From deployment point of view, 3 working, 1 not. Deployed successfully: openstack, odl_l2, onos. Not successfully deployed yet: ocl (removed from this installer)
Fuel: 8 scenarios on Jenkins: pure openstack, odl_l2, odl_l3, onos, bgpvpn, ovs, kvm, kvm_ovs. From deployment point of view, 5 working, 1 not working, 2 not activated/attempted yet. Deployed successfully: openstack, odl_l2, odl_l3, onos, bgpvpn Not successfully deployed yet: ovs. Not activated/attempted: kvm, kvm_ovs.
Fuel has Fuel7 on stable/brahmaputra branch and Fuel8 uplift work is WIP on master branch.
Joid: 4 scenarios on Jenkins: pure openstack, odl_l2, onos, and ocl. From deployment point of view, 3 working, 1 not activated/attempted yet. Deployed successfully: openstack, odl_l2, onos. Not successfully deployed yet: none. Not activated/attempted: ocl (removed from this installer)
https://wiki.opnfv.org/brahmaputra_testing_page#candidate_scenarios and https://wiki.opnfv.org/brahmaputra_testing_page =============================================================================================================== Release Test Status:
Functest (Morgan Richomme) Current situation
Functest regression yesterday night due to sdnvpn integration, fixed now .vPing/SSH Ok on manual tests with compass, not working with all scenarios yet we defined 2 vPING: vPing SSH and vPing userdata/cloudinit https://wiki.opnfv.org/vping_brahmaputra_page vPing/SSH should be OK now on compass (wait for next run). Table to be updated for apex as well I think http://testresults.opnfv.org/testapi/dashboard?project=functest&case=vPing&installer=apex and you can change the installer. wait for the test dashboard of LF…
Yardstick (Ana Cunha) first successful run with joid yesterday (os scenario), writing for a patch for workaround for odl scenario - remains to be tested on the labs On-going work for grafana dashboard and documentation Started to document results for ha
=============================================================================================================== Installer Status:
Apex (Tim Rozet): Still trying to get contrail to work, running into some openstack version issues where some contrail pieces rely on juno
JOID (Narinder Gupta) JOID deployed successfully on all three labs yesterday including the virtual deployment. Unfortunately functest had some issues yesterday which seems to be resolved today.
Compass4nfv (Weidong Shao) there were some functest and other instability with the code base, and we will need some extra time in getting Compass code stable
Fuel (Jonas Bjurel) Tickets and backlog jira dashboard 1)
Struggling with an autodeploy issue, hopefully solved now. Testing. Scenarios being rebased
=============================================================================================================== Project Updates: (as provided)
Scenario view status: Reviewed scenarios in table and noted those that will not be implemented this release Apex: 5 scenarios on Jenkins: 4 work from deploy point of view odl_l2 is about 90 pass/10 fail, odl_l3 worse pass rate Not successfully deployed yet: ocl- working on it
Compass: 4 scenarios on Jenkins: pure openstack, odl_l2, onos, and ocl. From deployment point of view, 3 working, 1 not. Deployed successfully: openstack, odl_l2, onos. Not successfully deployed yet: ocl (removed from this installer)
Fuel: 8 scenarios on Jenkins: pure openstack, odl_l2, odl_l3, onos, bgpvpn, ovs, kvm, kvm_ovs. From deployment point of view, 5 working, 1 not working, 2 not activated/attempted yet. Deployed successfully: openstack, odl_l2, odl_l3, onos, bgpvpn Not successfully deployed yet: ovs. Not activated/attempted: kvm, kvm_ovs.
Fuel has Fuel7 on stable/brahmaputra branch and Fuel8 uplift work is WIP on master branch.
Joid: 4 scenarios on Jenkins: pure openstack, odl_l2, onos, and ocl. From deployment point of view, 3 working, 1 not activated/attempted yet. Deployed successfully: openstack, odl_l2, onos. Not successfully deployed yet: none. Not activated/attempted: ocl (removed from this installer)
ARMband:
Availability：（Qiao Fu) HA project has got all the rst file merged and ready for release. Will upload a full set of rst file for the two release doc later today.
Bottlenecks: (Hongbo Tian)
Copper: (Bryan Sullivan) at risk
Doctor (Ryota, Ildiko)
IPv6: (Bin Hu) -
LSOAPI (Kevin Leuhrs) at risk
NFV4KVM: (Don Dugger, Raghuveer Reddy)
ONOSFW: (Ashlee Young)
OpenvSwitch (Mark Gray)
OPNFVdocs (Chris Price) Issues: Docs working on central content. Working with projects on project content. Patches outstanding on many projects. Awaiting final scenario list to progress to completion. Working with projects on project content. Patches outstanding on many projects. Unable to code freeze until installers freeze. ONOSFW reviewing reviewing onosfw docs. Projects are working on docs, some feature projects are waiting for the integration to be complete to document that for the config guide
OVNO (Stuart Mackie) Apex still working on patch for plugin
Parser
Prediction (Hai Liu)
Qtip (Vikram)
SFC (Brady Johnson)
SDNVPN (Tim Irnich)
SFQM (Maryam Tahhan)
StorPerf (Mark Beierle)
=============================================================================================================== Action ItemsUnderlined Text Action: Owner: Status:
Action: Owner: Status:
Action: Owner: Status:
1/25/16
Countdown to Release: 6 days __**Upcoming deadlines:**__ =============================================================================================================== Minutes from daily standup 1/25/16
Top Accomplishments & Important Updates since last report
.Fuel made great strides over the weekend. ONOS scenarios working. Still unable to freeze.
.Yardstick successfully ran ONOS scenario with Compass.
.Functest now relatively stable from CI perspective. ONOS scenarios working.
===============================================================================================================
Top Issues
Issue: We are at high risk of missing release date. Test cycles for each scenario take a significant amount of time. Even if all installers & scenarios are working properly it takes 7 days to run test through them all. Owner: Debra Status: Additional options being proposed: 1. Release with what we have and move everything else to C Release 2. Delay release for up to 3 weeks 3. staggered/phased release of scenario support as a potential. Phase 1: Release what we can on Feb 2; Phase 2: Release what is ready 1-2 weeks later, Phase 3: final release 1-2 weeks after phase 2. These have been brought up to the TSC via email for consideration next TSC meeting.
Issue: Functest & Yardstick do not produce stable results. Troubleshooting scenarios. Need assistance from installer & feature project teams to fault isolation and fix. Owner: Morgan Richomme (Functest), Ana Cuna (Yardstick) Status: Functest now relatively stable from CI perspective (except just joid). Still work on the different vPing (and then vIMS) testcases (that need ssh access from the VMs). Onos OK.
Yardstick:ONOS - successfully run with Compass. ODL scenario - debugging on-going, Yardstick will not run with ODL. Narrowing down subsystems involved. Working on SFC test cases. Working on influxdB+grafana
Issue: Getting RC0 out Owner: Debra Status: Installer code freeze extended due to Fuel trying to stabilize with upstream. Compass also troubleshooting some functest and other instability with the code base, need some extra time in getting Compass code stable. Apex, JOID ready to freeze
Issue: Fuel is not ready to code freeze. Upstream Fuel is not stable. Owner: Jonas Bjurel Status: Requesting code freeze delay until 1/29 and suggesting release slip 3 weeks
Issue: A large number of projects had code file licensing issues (either missing licenses or non-Apache licenses). Owner: Ray Paik Status:Projects have made good progress on getting patches in to fix the files. Scott is making another license scan to see what if anything is outstanding. Thank you to all who have responded to this request in such a timely manner.
Issue: OpenContrail does not deploy Owner: Stuart Mackey Status: Apex still trying to get contrail to work, running into some openstack version issues where some contrail pieces rely on juno
Issue: Scenarios are frozen, but several not working yet, or have not been tried Owner: test teams Status: Next step is creating Jenkins jobs and testing each scenario. In process with test teams leading. https://wiki.opnfv.org/brahmaputra_testing_page#candidate_scenarios and https://wiki.opnfv.org/brahmaputra_testing_page
Issue: NFV4KVM – plugin does not work with Fuel, The NFV4KVM does install a working kernel on the compute node. Owner:Don Dugger/Raghuveer Reddy Status: 1/22 no new update. Fuel has bigger issues at the moment. We have discovered that OVS appears to require kernel modules that are not there. Working on creating a fix to the Fuel plugin creation scripts that will fix this problem, potentially will have a fix before code freeze, but certainly should be able to solve the problem for an SR1 release. =============================================================================================================== Pod Status:
All pods up and running
=============================================================================================================== Scenario Status: Scenarios are frozen. No feature project has reported that their scenarios are missing or unsupported. Candidate scenarios are having Jenkins jobs created for them. Test teams troubleshooting scenarios and test software.
https://wiki.opnfv.org/brahmaputra_testing_page#candidate_scenarios and https://wiki.opnfv.org/brahmaputra_testing_page =============================================================================================================== Release Test Status:
Functest (Morgan Richomme) Current situation * Functest Functest tests more stable, Some tests run in CI over the week end, Functest now relatively stable from CI perspective (except just joid I had to cancel). Still work on the different vPing (and then vIMS) testcases (that need ssh access from the VMs). Onos OK. ODL, maybe an update to be done, but we are integrating upstream suite, and it is not fully clear that the suite has been adapted to new version of ODL. https://wiki.opnfv.org/doku.php?id=vPing_brahmaputra_page https://wiki.opnfv.org/doku.php?id=vIMS_brahmaputra_page Test DB migrated this morning (9 UTC) http://testresults.opnfv.org/testapi/pods Still work ahead to tune some cases, and collect the error causes
Yardstick (Ana Cunha) ONOS first succesfull run on Compass, continue troubleshooting ODL scenarios
=============================================================================================================== Installer Status:
Apex (Tim Rozet): Still trying to get contrail to work, running into some openstack version issues where some contrail pieces rely on juno
JOID (Narinder Gupta) JOID we are ready for code freeze as last pending issue related to Orange pod has been fixed
Compass4nfv (Weidong Shao) there were some functest and other instability with the code base, and we will need some extra time in getting Compass code stable
Fuel (Jonas Bjurel) Tickets and backlog jira dashboard 1) Build
The building issues have been resolved to the point that it builds flawlessly in the OPNFV CI engine. Daily masters are produced every night since Friday. Work ongoing with getting it to build inside Ericsson corporate network. Stefan probably nailed it as it seems to be an issue with the 3.13.0-66 kernel.
2) Deployment. I have successfully deployed the daily master image from Friday CI build: http://artifacts.opnfv.org/fuel/opnfv-2016-01-22_16-05-16.iso It works more or less flawlessly, a few minor comments: - The ssh service is not started as early as it used to do, it is not started when Fuelmenu
pops up which will be an issue for auto deploy. # service sshd start fixes it
- Fuelmenu will sometimes tell you that the URLs for the bootimage cannot be found,
if you make sure you can reach url resources eg. Through open a consol from fuelmenu and type nslookup www.google.com – just don’t care, select “save & continue” and the “quit without saving” you should be good.
- Selecting centos boot image didn’t work for me
3) Early this week we need to prioritize getting autodeploy to work.
But to my mind, this seems to be stable enough for plugin work. Please start commit rebased plugins to Master.
- As I have indicated to TSC (Chris and Frank), my judgement is that we need another 3 week after Feb 2 to complete the things and have them in a good shape. The other option is to release with Fuel7/Kilo which is quite stable and a good veichle for developing stabilizing the scenarios and the test pipe-line.
- Issue with Jenkins server caused delays on Sunday but were resolved and builds are working again.
=============================================================================================================== Project Updates: (as provided)
ARMband:
Bottlenecks: (Hongbo Tian)
Copper: (Bryan Sullivan) at risk
Doctor (Ryota, Ildiko) Apex patch for AODH is merged. Doctor will make the release.
IPv6: (Bin Hu) -
LSOAPI (Kevin Leuhrs) at risk
NFV4KVM: (Don Dugger, Raghuveer Reddy)
ONOSFW: (Ashlee Young)
OpenvSwitch (Mark Gray)
OPNFVdocs (Chris Price) Issues: Docs working on central content. Working with projects on project content. Patches outstanding on many projects. Awaiting final scenario list to progress to completion. Working with projects on project content. Patches outstanding on many projects. Unable to code freeze until installers freeze. ONOSFW reviewing reviewing onosfw docs. Projects are working on docs, some feature projects are waiting for the integration to be complete to document that for the config guide
OVNO (Stuart Mackie) Apex
Parser
Prediction (Hai Liu)
Qtip (Vikram)
SFC (Brady Johnson) Apex patch is merged, SFC scenario is supported in the release
SDNVPN (Tim Irnich)
SFQM (Maryam Tahhan)
StorPerf (Mark Beierle)
=============================================================================================================== Action ItemsUnderlined Text Action: Owner: Status:
Action: Owner: Status:
Action: Owner: Status:
