====== Doctor Team Meetings ====== Meeting Info: * Weekly on Tuesday at [[http://www.timeanddate.com/worldclock/fixedtime.html?hour=14&min=0&sec=0|1400]] UTC (fixed in 6:00 PT) * Note: we are now in winter time (PST), i.e. one hour later in China/Japan. * IRC channel: ''#opnfv-doctor'' * GoToMeeting: [[https://global.gotomeeting.com/join/409154429|https://global.gotomeeting.com/join/409154429]] * Agenda for the next meeting: https://etherpad.opnfv.org/p/doctor_meetings Meeting minutes: http://meetbot.opnfv.org/meetings/opnfv-doctor/ and below ---- ==== March 1, 2016 ==== Agenda: * Congrats! Doctor is in the first cut of Brahmaputra! * we need fix on functest dashboard * Hackfest @ ONS: Doctor Session Planning * OpenStack Austin Summit prep - How to approach OpenStack with blueprints? (Ildiko) * Nova, Neutron, Cinder * Vitrage discussion. What is released in Austin, what would be expected next. * Doctor SB API * Integration & Testing * Maintenance? * Hacking session (integrating SFQM+Doctor) 2-3 hours? * AOB * Open to proposals: use JIRA, gerrit, etherpad, ML as you like Minutes: http://ircbot.wl.linuxfoundation.org/meetings/opnfv-doctor/2016/opnfv-doctor.2016-03-01-14.01.html ==== Feb 23, 2016 ==== Agenda: * OPNFV B release readiness * https://wiki.opnfv.org/releases/brahmaputra/daily_status * https://gerrit.opnfv.org/gerrit/10333 * https://build.opnfv.org/ci/view/functest/job/functest-apex-opnfv-jump-1-daily-brahmaputra/60/console * Doctor SB API * https://wiki.opnfv.org/_media/doctor/opnfv_doctor_sb_api.20160223.pptx * AOB * https://www.youtube.com/watch?v=w1XQATkrdmg * Cinder: https://review.openstack.org/#/c/258968/ Minutes: http://ircbot.wl.linuxfoundation.org/meetings/opnfv-doctor/2016/opnfv-doctor.2016-02-23-14.00.html ==== Feb 16, 2016 ==== Agenda: * OPNFV B release readiness * We still need our test scenario run successfully, but waiting apex jobs in functest run successfully first. * Release note * Vitrage discussion * C release features * AOB * OpenStack Summit CFS Vote Minutes: http://ircbot.wl.linuxfoundation.org/meetings/opnfv-doctor/2016/opnfv-doctor.2016-02-16-13.59.html ==== Feb 9, 2016 ==== Agenda: * OPNFV B release readiness + JIRA Tickets (project = doctor and Labels = R2 and Status != closed) * https://jira.opnfv.org/browse/DOCTOR-22 - Functest: doctor test scenario * https://jira.opnfv.org/browse/DOCTOR-32 - depends on DOCTOR-22 * Vitrage presentation * {{:doctor:vitrage_demo_for_doctor_v3.pdf|slides}} Minutes: http://ircbot.wl.linuxfoundation.org/meetings/opnfv-doctor/2016/opnfv-doctor.2016-02-09-14.00.html ==== Feb 2, 2016 ==== Agenda: * OpenStack BPs * Nova * Cinder * OpenStack summit * Vitrage joining next week meeting * OPNFV B release readiness + JIRA Tickets (Labels = R2 and Status != closed) * https://jira.opnfv.org/browse/DOCTOR-11 - Closed? Yes. * https://jira.opnfv.org/browse/DOCTOR-14 - will fix this week? Yes. * https://jira.opnfv.org/browse/DOCTOR-21 - will close this week after tests passed with apex team * https://jira.opnfv.org/browse/DOCTOR-22 - after DOCTOR-21 * https://jira.opnfv.org/browse/DOCTOR-32 - (Epic: Integration) * https://jira.opnfv.org/browse/DOCTOR-35 - (Epic: Req Docs) * https://jira.opnfv.org/browse/DOCTOR-42 - WIP --+--> will work with/as opnfvdocs * https://jira.opnfv.org/browse/DOCTOR-43 - WIP --/ * AOB * Hackfest agenda proposals Minutes: http://meetbot.opnfv.org/meetings/opnfv-doctor/2016/opnfv-doctor.2016-02-02-14.00.html ==== Jan 26, 2016 ==== Agenda: * OPNFV B release Readiness * Deployment tools code freeze * Status: Aodh support merged in Apex https://gerrit.opnfv.org/gerrit/#/c/7455/ , fix(es) : https://gerrit.opnfv.org/gerrit/#/c/7951/ * Docs (Configguides etc.) * OpenStack BPs * Nova * CInder * Q1 2016 Hackfest * Santa Clara, 14-15 March * https://etherpad.opnfv.org/p/Q1'2016_Hackfest * AOB Minutes: http://meetbot.opnfv.org/meetings/opnfv-doctor/2016/opnfv-doctor.2016-01-26-14.01.html ==== Jan 19, 2016 ==== Agenda: * OPNFV B release Readiness * Deployment tools code freeze * Status: Not ready, Aodh support is still missing * Related: * https://gerrit.opnfv.org/gerrit/#/c/5503/ * https://review.openstack.org/#/c/258520/ * Testing tools and test cases code freeze * Status: Done in Doctor, functest integration done (Jan 15), but no test * Scenarios operational (deploy/build/test defined and can be triggered from jenkins automatically) * Status: No, but can be started quickly once deployment tool is readay (test scenario can be triggered by doctor script change) * All documents ready for review. No new content, only corrections after this date. * Status: platform overview is ready, but config guide and use manual are still missing * OpenStack BPs * AOB Minutes: http://meetbot.opnfv.org/meetings/opnfv-doctor/2016/opnfv-doctor.2016-01-19-14.00.html ==== Jan 12, 2016 ==== Agenda: * OPNFV B release Readiness * By Jan/12 * Deployment tools code freeze * Status: Not ready for Doctor: * related: * https://gerrit.opnfv.org/gerrit/#/c/5503/ * https://review.openstack.org/#/c/258520/ * Testing tools and test cases code freeze * Status: Done in Doctor, but not ready in functest * Scenarios operational (deploy/build/test defined and can be triggered from jenkins automatically) * Status: No * By Jan/19 * All documents ready for review. No new content, only corrections after this date. * Status: platform overview is ready, but config guide and use manual are still missing * Maintenance * https://gerrit.opnfv.org/gerrit/#/c/5571/ * OpenStack BPs * https://etherpad.opnfv.org/p/doctor_bps * JIRA tickets * https://jira.opnfv.org/browse/DOCTOR-46?jql=project%20%3D%20DOCTOR%20and%20duedate%20%3C%20now()%20and%20status%20!%3D%20closed * AOB * branch has been cut, backport you patch when it merged in 'master' if you need -- should we have policy for backport? Minutes: http://meetbot.opnfv.org/meetings/opnfv-doctor/2016/opnfv-doctor.2016-01-12-14.00.html ==== Jan 5, 2016 ==== Agenda: * Milestone E Readiness * https://wiki.opnfv.org/doku.php?id=releases/brahmaputra/release_plan/doctor_milestone_e_report * http://lists.opnfv.org/pipermail/opnfv-tech-discuss/2015-December/007271.html * Status of Test Cases * https://gerrit.opnfv.org/gerrit/#/c/5439/ * this is a mix of the planned two test cases; need to update the test case description * Nova BPs * https://etherpad.opnfv.org/p/doctor_bps * https://review.openstack.org/192246/ * JIRA Issues * AOB Minutes: http://meetbot.opnfv.org/meetings/opnfv-doctor/2016/opnfv-doctor.2016-01-05-14.01.html ==== Dec 22, 2015 ==== Agenda: * Milestone E forcus * test case/script first by this Friday * then enable jenkins jobs and prepare manuals * JIRA Issues * AOB Minutes: http://meetbot.opnfv.org/meetings/opnfv-doctor/2015/opnfv-doctor.2015-12-22-14.00.html ==== Dec 15, 2015 ==== Agenda: * JIRA Issues * AOB * Next Meeting? * Dec 22 meeting * Dec 29 skip * Jan 5 meeting Minutes: http://meetbot.opnfv.org/meetings/opnfv-doctor/2015/opnfv-doctor.2015-12-15-14.02.html ==== Dec 8, 2015 ==== Agenda: * doctor docs * http://lists.opnfv.org/pipermail/opnfv-tech-discuss/2015-December/006948.html * /docs/platformoverview : overview of B release deliverable * /docs/configguide : manuals to configure components and project specific features (and check validation) * /docs/userguide : manuals to use specific features (and>> check validation) * https://jira.opnfv.org/browse/DOCTOR-3 * https://jira.opnfv.org/browse/DOCTOR-11 * https://jira.opnfv.org/browse/DOCTOR-17 * Cinder BP * Neutron RFE * JIRA Issues * https://jira.opnfv.org/browse/DOCTOR-46?jql=project%20%3D%20DOCTOR%20and%20duedate%20%3C%20now() Minutes: http://meetbot.opnfv.org/meetings/opnfv-doctor/2015/opnfv-doctor.2015-12-08-14.00.html ==== Dec 1, 2015 ==== Agenda: * doctor docs * new documentation toochain is ready * http://artifacts.opnfv.org/releng/docs/how-to-use-docs/documentation-example.html * let's use it * https://gerrit.opnfv.org/gerrit/#/c/3663/ * I'll stop doctor jenkins job for old docs creation * test cases * http://213.77.62.197/test_projects/doctor/cases * use case 1: {"name": "doctor-notification", "description": "immediate notification for fault management"} * use case 2: {"name": "doctor-mark-down", "description": "resource state correction for fault management"} * Add http://artifacts.opnfv.org/doctor/html/02-use_cases.html ? (low, but ask functest team about list regulation) * readiness for milestone D * https://wiki.opnfv.org/releases/brahmaputra/release_plan/doctor_milestone_d_report * AOB * fastpath: dpdk 2.2 mid of dec (new feature - link status), collectd plugin: not ready, after collecd staff done --> create post-installer for integration in OPNFV * nova: working on bp * alarming components packaging: suse - none, ubuntu - bug (easy workaround exists), * neutron: had closed discussion with neutron folks regarding mark-down RFE * cinder: checking tooz and service group Minutes: http://meetbot.opnfv.org/meetings/opnfv-doctor/2015/opnfv-doctor.2015-12-01-14.00.html ==== Nov 24, 2015 ==== Agenda: * Cinder BP (DOCTOR-19) * https://review.openstack.org/#/c/238832/ * Inspector BPs in OpenStack * Congress: https://review.openstack.org/#/c/246742/ * Vitrage: https://blueprints.launchpad.net/vitrage * NFV vs Ceilometer Alarm i/f (DOCTOR-46) * https://wiki.opnfv.org/_media/doctor/alarms_in_nfv_ceilometer.pptx * action item check * http://meetbot.opnfv.org/meetings/opnfv-doctor/2015/opnfv-doctor.2015-11-17-13.59.html Minutes: http://meetbot.opnfv.org/meetings/opnfv-doctor/2015/opnfv-doctor.2015-11-24-13.59.html ==== Nov 17, 2015 ==== Agenda: * OPNFV Summit Report * 3 Design Sessions https://wiki.opnfv.org/meetings/doctor#nov_9-10_2015 * Doctor Presentation * Doctor Demo (Presentation & Booth) * Functest * https://jira.opnfv.org/browse/DOCTOR-22 * Cinder mark-down * https://review.openstack.org/#/c/238832/ * Puppet Script for Congress Integration * Doctor project will use Congress in OPNFV C release or later * JIRA Issue check * https://jira.opnfv.org/browse/DOCTOR-43?jql=project%20%3D%20DOCTOR%20and%20updated%20%3E%20%22-1w%22 * https://jira.opnfv.org/browse/DOCTOR-46?jql=project%20%3D%20DOCTOR%20and%20duedate%20%3C%20now() * https://jira.opnfv.org/issues/?jql=project%20%3D%20DOCTOR%20and%20duedate%20%3E%20now()%20and%20duedate%20%3C%20endOfDay(4d) Minutes: http://meetbot.opnfv.org/meetings/opnfv-doctor/2015/opnfv-doctor.2015-11-17-13.59.html ==== Nov 9-10, 2015 ==== OPNFV design summit sessions Nov 9th, 15:15-14:00 at Room #3: * Controller: Nova, Neutron -- lead by Tomi * https://etherpad.opnfv.org/p/doctor_summit_controller Nov 10th, 11:00-12:00 at Room #2: * Inspector: requirements and candidates (Monasca, Congress, Vitrage) -- lead by Carlos * https://etherpad.opnfv.org/p/doctor_summit_inspector Nov 10th, 14:00-15:00 at Room #2: * Monitor Integration (SFQM) -- lead by Michael * https://etherpad.opnfv.org/p/doctor_summit_monitor * B release task break down -- lead by Ryota * https://jira.opnfv.org/browse/DOCTOR-32 * https://jira.opnfv.org/browse/DOCTOR-35 * https://jira.opnfv.org/browse/DOCTOR-21 * Ryota to create JIRA issue in genesis by this friday ==== Nov 3, 2015 ==== Agenda: * OpenStack Summit Report regarding doctor * Doctor F2F Meeting * Nova get valid state * Nova versioned notification API and notification when force-down * Ceilometer/Aodh * Congress * Vitrage * OPNFV design summit * https://wiki.opnfv.org/_media/2015_design_summit_schedule_rev2_sheet1.pdf * https://etherpad.opnfv.org/p/OPNFV_Design_Summit_2015 * Topics/Sessions: * Controller: Nova, Neutron -- lead by Tomi * Notifier: Alarming params * Inspector: requirements and candidates (Monasca, Congress, Vitrage) -- lead by Carlos * Monitor Integration (SFQM) -- lead by Maryam (TBC) * B release task break down -- lead by Ryota * OPNFV summit presentation * https://wiki.opnfv.org/_media/doctor/opnfv_doctor_20151111.rev1.pptx * please review and send comments to Ryota Minutes: http://meetbot.opnfv.org/meetings/opnfv-doctor/2015/opnfv-doctor.2015-11-03-13.59.html ==== Oct 28, 2015 ==== Meeting Info: * Oct 28 (Wednesday) F2F meeting in OpenStack Summit Tokyo * When: 9-11am * Where: Community Lounge, 1F, #3 International Convention Center Pamir Minutes: * Nova BPs * entities are "service" (representing nova-compute), "service" (VM, Instance) and physical machine * No notication when service-mark-down and vm-rest-state (reset state already exists in Nova) * Related BPs: * get valid VM state - may have discussion on friday * notificatin when force-down or disabled - no session... - should be in Mitaka * notification when server state has been reset - we have to draft and find asignee * add new parms in reset server state API * versioned nortification - 2:40pm - 3:20pm on Thursday <--- blocker for balaz's BP * make mark-service-down to allow mark affected VMs down as option of that API * 3 levels of computing failure: * compute node is down * nova-compute service is down * VM running on compute node is down * if nova-compute service goes down, VNF will likely continue running without any service disruption. Thus, VIM may not need to notify VNFM as the VNF instance is not being affected * Although the VNFM may not need to be notified, we need to notify the NFVO. * In this case in specific, there's a clear seperation of Consumer: VNFM and NFVO(/VIM administrator) * auto-reaction * vim should not execute auto reaction based on policy in server metadata specified by user, or it should? * use case should be discussed and described in doctor document * Neutron BP * Assignee: Carlos (NEC) * we need to clarify what we do in Nova, and discuss use cases and requirements in Doctor first * Cinder BP * Assignee: ZTE team * Alarming / Aodh etc. * Monasca, Congress, Vitrage * Doctor + Congress meeting, Wed 28 Oct 15:40-16:20, room?) * OPNFV Summit * Presentation * target audience are devs * 10mis for use cases, 10 mins for upstream activities including plans / project status * PoC Demo at docomo booth * Continuing preparing handouts * Design summit topics (design summit running Mon-Tue, 9-10 Nov) * agenda should be fixed in the next week (by Nov 6) * Congress * Nova BP * test cases/tool ==== Oct 20, 2015 ==== Agenda: * OpenStack Summit * Nova * https://wiki.openstack.org/wiki/Design_Summit/Mitaka/Etherpads#Nova * https://etherpad.openstack.org/p/mitaka-nova-spec-review-tracking * https://wiki.openstack.org/wiki/Nova/Mitaka_Release_Schedule (Proposal from Nova PTL) * Ceilometer / Aodh * https://wiki.openstack.org/wiki/Design_Summit/Mitaka/Etherpads#Ceilometer * Alarms in ETSI NFV & Ceilometer * Cinder force down * https://etherpad.opnfv.org/p/doctor_bp_cinder * JIRA Issue check / doctor status * https://jira.opnfv.org/browse/DOCTOR-43?jql=project%20%3D%20DOCTOR%20and%20updated%20%3E%20%22-1w%22 * OPNFV Summit Minutes: http://meetbot.opnfv.org/meetings/opnfv-doctor/2015/opnfv-doctor.2015-10-20-12.59.html ==== Oct 13, 2015 ==== Agenda: * fastpath metrics update * Extended Error statistics API in DPDK - patches submitted: * Please see http://dpdk.org/dev/patchwork/project/dpdk/list/?submitter=317 * [dpdk-dev,2/2] igb: fix VF statistic wraparound handling * [dpdk-dev,1/2] ixgbe: fix VF statistic wraparound handling macro * [dpdk-dev] ixgbe: fix 82599 / 82598 register differences * [dpdk-dev,v2,11/11] fm10k: add xstats() implementation * [dpdk-dev,v2,10/11] i40evf: add xstats() implementation * [dpdk-dev,v2,09/11] i40e: add xstats() implementation * [dpdk-dev,v2,08/11] ixgbevf: add xstats() functions to VF * [dpdk-dev,v2,07/11] ixgbe: update statistic strings to scheme * [dpdk-dev,v2,06/11] igbvf: add xstats() implementation * [dpdk-dev,v2,05/11] igb: add xstats() implementation * [dpdk-dev,v2,04/11] virtio: add xstats() implementation * [dpdk-dev,v2,03/11] ethdev: update xstats_get() strings and Q handling * [dpdk-dev,v2,02/11] doc: add extended statistics to prog_guide * [dpdk-dev,v2,01/11] doc: add extended statistics notes * DPDK Keep Alive - patches submitted: * http://dpdk.org/dev/patchwork/project/dpdk/list/?submitter=326 * [dpdk-dev,v2,3/3] docs: add keep alive sample app guide * [dpdk-dev,v2,2/3] l2fwd: keep alive sample application * [dpdk-dev,v2,1/3] rte: add keep alive functionality * Documenting all the options for DPDK KA would like to share with Doctor soon * Patches are currently awaiting review of the DPDK community * collectd-dpdk * Started implementing a dpdk plugin for collectd that will retrieve the DPDK extended stats. * Testing of initial implementation underway. * collectd-ceilometer * Have an implementation that pushes stats directly to ceilometer from collectd. * working on the deployment code for devstack. * Maintenance * https://jira.opnfv.org/browse/DOCTOR-11 * https://review.openstack.org/192246 * https://review.openstack.org/182350 * JIRA Issue check * https://jira.opnfv.org/browse/DOCTOR-43?jql=project%20%3D%20DOCTOR%20and%20updated%20%3E%20%22-1w%22 Minutes: http://meetbot.opnfv.org/meetings/opnfv-doctor/2015/opnfv-doctor.2015-10-13-12.59.html ==== Oct 6, 2015 ==== Agenda: * Typical use case for OPNFV B release * Zabbix limitation * Alarm gaps Gap analysis: NFV Alarms in IFA006 vs Ceilometer alarms * Maintenance * anyother JIRA items to discuss? * https://jira.opnfv.org/browse/DOCTOR-43?jql=project%20%3D%20DOCTOR%20and%20updated%20%3E%20%22-1w%22 * https://jira.opnfv.org/browse/DOCTOR-37 * JIRA scrum board * https://jira.opnfv.org/secure/RapidBoard.jspa?rapidView=71&view=planning.nodetail Minutes: http://meetbot.opnfv.org/meetings/opnfv-doctor/2015/opnfv-doctor.2015-10-06-13.00.html ==== Sep 29, 2015 ==== Agenda: * B release planning / status (quick announce) * Typical use case for OPNFV B release * Review Work Items for OpenStack Mitaka summit * Nova: https://jira.opnfv.org/browse/DOCTOR-29 https://jira.opnfv.org/browse/DOCTOR-11 * Monasca: https://jira.opnfv.org/browse/DOCTOR-27 * Neutron: https://jira.opnfv.org/browse/DOCTOR-18 * Cinder?: https://jira.opnfv.org/browse/DOCTOR-41 * anyother JIRA items to discuss? * https://jira.opnfv.org/browse/DOCTOR-43?jql=project%20%3D%20DOCTOR%20and%20updated%20%3E%20%22-1w%22 * https://jira.opnfv.org/browse/DOCTOR-37 Minutes: http://meetbot.opnfv.org/meetings/opnfv-doctor/2015/opnfv-doctor.2015-09-29-13.01.html ==== Sep 22, 2015 ==== Agenda: * Sep 22 * B release planning / status * Typical use case for OPNFV B release Minutes: http://meetbot.opnfv.org/meetings/opnfv-doctor/2015/opnfv-doctor.2015-09-22-13.01.html ==== Sep 15, 2015 ==== Agenda: * Vaccine project * B release planning / status Minutes: http://meetbot.opnfv.org/meetings/opnfv-doctor/2015/opnfv-doctor.2015-09-15-12.57.html ==== Sep 8, 2015 ==== Agenda: * DOCTOR-21: BGS(genesis) * New members Minutes: http://meetbot.opnfv.org/meetings/opnfv-doctor/2015/opnfv-doctor.2015-09-08-13.00.html ==== Sep 1, 2015 ==== Agenda: * Brief update about Nova BPs (DOCTOR-29) * SFQM use cases (DOCTOR-27) * Neutron mark down API (DOCTOR-18) * Requirement Doc fix: fignum (DOCTOR-13) * Ceilometer event-alarm (DOCTOR-24) * Requirement Doc fix: poison (DOCTOR-8) * DOCTOR-15 move fault table to Annex * assign JIRA items Minutes: http://meetbot.opnfv.org/meetings/opnfv-doctor/2015/opnfv-doctor.2015-09-01-13.01.html ==== Aug 25, 2015 ==== Agenda: * Ceilometer BP Status * Nova BP fixing evacuation * Pinpoint Intro Minutes: http://meetbot.opnfv.org/meetings/opnfv-doctor/2015/opnfv-doctor.2015-08-25-12.55.html ==== Aug 18, 2015 ==== Agenda: * Ceilometer BP Status * JIRA items Minutes: http://meetbot.opnfv.org/meetings/opnfv-doctor/2015/opnfv-doctor.2015-08-18-13.02.html ==== Aug 11, 2015 ==== Agenda: * SFQM update (Maryam) * Nova BP Status - merged! * https://review.openstack.org/#/q/topic:bp/mark-host-down,n,z * https://review.openstack.org/#/c/184086/ * Ceilometer BP Status * https://review.openstack.org/#q,topic:bp/event-alarm-evaluator,n,z * JIRA items * https://jira.opnfv.org/browse/DOCTOR-25?jql=project%20%3D%20DOCTOR * AOB Meeting minutes: http://meetbot.opnfv.org/meetings/opnfv-doctor/2015/opnfv-doctor.2015-08-11-12.56.html ==== Aug 4, 2015 ==== Agenda * Hackfest Sync (Demos if needed) * https://etherpad.opnfv.org/p/doctor_hackfest_20150730 * Release Planning * https://wiki.opnfv.org/releases/brahmaputra/release_plan * https://wiki.openstack.org/wiki/Liberty_Release_Schedule * Nova BP Status * https://review.openstack.org/#/q/topic:bp/mark-host-down,n,z * https://review.openstack.org/#/c/184086/ * https://etherpad.openstack.org/p/liberty-nova-non-priority-feature-freeze * Ceilometer BP Status https://review.openstack.org/#q,topic:bp/event-alarm-evaluator,n,z Meeting minutes: http://meetbot.opnfv.org/meetings/opnfv-doctor/2015/opnfv-doctor.2015-08-04-12.56.html ==== July 30, 2015 ==== Meeting at OPNFV Hackfest: https://etherpad.opnfv.org/p/doctor_hackfest_20150730 ==== July 28, 2015 ==== Agenda: * Nova BP status https://etherpad.opnfv.org/p/doctor_bps * Ceilometer BP status * Work session agenda at OPNFV hackfest https://etherpad.opnfv.org/p/doctor_hackfest_20150730 * Demo preparation status@ ODL Summit * AoB Meeting minutes: http://meetbot.opnfv.org/meetings/opnfv-doctor/2015/opnfv-doctor.2015-07-28-13.01.html ==== July 21, 2015 ==== Agenda: * Nova BP status https://etherpad.opnfv.org/p/doctor_bps * Ceilometer BP status * Work session agenda at OPNFV hackfest * Demo preparation status@ ODL Summit * List of tasks progress * AoB * next meeting? Meeting minutes: http://meetbot.opnfv.org/meetings/opnfv-doctor/2015/opnfv-doctor.2015-07-21-12.58.html ==== July 14, 2015 ==== Agenda: * Nova BP status https://etherpad.opnfv.org/p/doctor_bps * Ceilometer BP status * Work session agenda at OPNFV hackfest @ ODL Summit * Demo preparation status * OPNFV sessions @ OpenStack Summit in Tokyo * AoB Meeting minutes: http://meetbot.opnfv.org/meetings/opnfv-doctor/2015/opnfv-doctor.2015-07-14-12.56.html ==== July 7, 2015 ==== Agenda: * Release B Planning - https://etherpad.opnfv.org/p/doctor * Nova BP Status * Ceilometer BP Status * Demo preperation status * Doctor session at OPNFV hackfest * AoB * Doctor JJB - http://lists.opnfv.org/pipermail/opnfv-tech-discuss/2015-July/003566.html * List of tasks: https://etherpad.opnfv.org/p/doctor Meeting minutes: http://meetbot.opnfv.org/meetings/opnfv-doctor/2015/opnfv-doctor.2015-07-07-13.00.html ==== June 30, 2015 ==== Agenda: * Nova BP Status * Ceilometer BP Status * Demo status * Doctor session at OPNFV hackfest * AoB * Deliverables to OPNFV R2 Meeting minutes: http://meetbot.opnfv.org/meetings/opnfv-doctor/2015/opnfv-doctor.2015-06-30-12.58.html ==== June 23, 2015 ==== Agenda: * BP Status * Nova * Ceilometer * Deliverable status * AoB * Ceilometer *virtual* mid-cycle meetup * OPNFV Hackfest Minutes: * BP Status Nova * Roman (intel) is back and will continue implementation * Carlos is facing some issues with proposed patch (conflict in gerrit) * https://review.openstack.org/#/c/185849/ * BP Status Ceilometer * https://review.openstack.org/#/c/172893/ * spec got approved by 3 core reviewers * more developers from Intel working on it * in total 3 developers working on this spec * Alarm related activity: spin-off project from Ceilometer alarming * Deliverable status * tagged '2015.1.0' * working on open issues (review comments). pls join discussion in Gerrit. * proposal to discuss open comments in Doctor meeting * AoB * Ceilometer *virtual* mid-cycle * Ceilometer has a virtual mid-cycle as opposed to face-to-face * https://etherpad.openstack.org/p/ceilometer-liberty-midcycle * OPNFV Hackfest * at ODL summit (not OPNFV slot in ODL summit) https://etherpad.opnfv.org/p/OPNFV_at_ODL * #action Carlos to prepare demo script for demo * Ildiko and Gerald propose to have a session on Doctor, where demo is part of, but also show status of BPs/specs and way forward * req-doc discussion * fencing * Fencing gap had been part of earlier version of Doctor document, but was removed * agree mention fencing in "general features", but do not have gap on it * Maintenance state * DOCTOR-11 has wider scope. offline review needed. * agreement to have two maintenance states ("going-to-maintenance" and "in-maintenance") * User can stop maintenance * ...or user does not respond to maintenance request * error cases needed, e.g. resend maintenance request after timeout * how to handle cases where a user is sending NACK? * (need further discussion) IRC Log: 22:00:08 r-mibu > #startmeeting doctor 22:00:40 GeraldK > #info Gerald 22:00:46 tojuvone > #info Tomi Juvonen 22:02:03 r-mibu > #startmeeting doctor 22:02:10 r-mibu > #endmeeting 22:04:05 r-mibu > #link https://etherpad.opnfv.org/p/doctor_meetings 22:04:31 cgoncalves > #info Carlos Goncalves 22:04:37 ildikov > #info Ildiko Vancsa 22:04:55 bertys > #info Bertrand Souville 22:07:13 r-mibu > #topic BP Status Nova 22:08:42 GeraldK > #info Roman (intel) is back and will continue implementation 22:09:35 GeraldK > #info Carlos is facing some issues with proposed patch (conflict in gerrit) 22:11:06 bertys > #link https://review.openstack.org/#/c/185849/ 22:13:52 r-mibu > #topic BP Status Ceilometer 22:14:05 r-mibu > #link https://review.openstack.org/#/c/172893/ 22:14:23 GeraldK > #info spec got approved by 3 core reviewers 22:14:43 GeraldK > #info 2 more developers from Intel working on it 22:15:00 GeraldK > #info in total 3 developers working on this spec 22:16:14 ildikov > #link https://review.openstack.org/192684 22:17:04 ildikov > #link https://review.openstack.org/192688 22:18:57 r-mibu > #topic Deliverable status 22:19:26 GeraldK > #link http://artifacts.opnfv.org/ 22:19:27 r-mibu > #info tagged '2015.1.0' 22:22:00 GeraldK > #info working on open issues (review comments). pls join discussion in Gerrit. 22:26:37 GeraldK > #info proposal to discuss open comments in Doctor meeting 22:27:23 r-mibu > #topic AoB 22:27:24 GeraldK > #topic AOB 22:28:05 r-mibu > #info Ceilometer *virtual* mid-cycle 22:28:26 GeraldK > #info Ceilometer F2F midcylce event is canceled, there will be virtual discussion, e.g. IRC 22:28:35 r-mibu > #link https://etherpad.openstack.org/p/ceilometer-liberty-midcycle 22:29:16 ildikov > #link http://doodle.com/6vfksdu38wcwqqd3 22:29:17 GeraldK > #info dates are not yet fixed. doodle vote. 22:29:34 ildikov > #info Ceilometer has a virtual mid-cycle as opposed to face-to-face 22:32:49 r-mibu > #info OPNFV Hackfest 22:33:39 GeraldK > #action Carlos to prepare demo script for demo 22:37:19 GeraldK > #info Ildiko and Gerald propose to have a session on Doctor, where demo is part of, but also show status of BPs/specs and way forward 22:38:01 r-mibu > #topic req-doc discussion 22:38:13 r-mibu > #info fencing 22:39:01 r-mibu > #link http://artifacts.opnfv.org/doctor/html/03-architecture.html 22:41:14 cgoncalves > #link https://gerrit.opnfv.org/gerrit/#/c/882/ 22:41:47 GeraldK > #info Fencing gap had been part of earlier version of Doctor document, but was removed 22:41:59 r-mibu > #info fencing is one of external system responsibilities (when the host mark down) 22:42:19 GeraldK > #info gerrit patch is proposing to add this feature to Doctor 22:43:49 GeraldK > #info Ildiko: there is discussion on whether Nova or external tool is responsible for fencing. should this be part of Doctor project? 22:46:25 GeraldK > #info Ryota: okay to not have gap on fencing, but mention this feature in the architecture section 22:47:23 r-mibu > #info agreed 22:47:39 GeraldK > #agree mention fencing in "general features", but do not have gap on it 22:47:58 r-mibu > #info Maintenance state 22:48:06 r-mibu > Change: ("going to maintenance" and "in maintenance") 22:50:11 r-mibu > #info DOCTOR-11 22:51:21 GeraldK > #info DOCTOR-11 has wider scope. offline review needed. 22:51:55 GeraldK > #agree agreement to have two maintenance states ("going-to-maintenance" and "in-maintenance") 22:52:06 r-mibu > #info User can stop maintenance 22:53:09 GeraldK > #info ...or user does not respond to maintenance request 22:54:34 GeraldK > #info error cases needed, e.g. resend maintenance request after timeout 22:54:49 GeraldK > #info how to handle cases where a user is sending NACK? 22:55:09 r-mibu > #info or having error/force policy 22:56:15 GeraldK > #info give responsibility back to Administrator ==== June 16, 2015 ==== Agenda: * BP Status * Nova * Mark down * https://review.openstack.org/#/c/169836/ * https://review.openstack.org/#/q/topic:bp/mark-host-down,n,z * Posting new BP for having host status visible in VM data or in general to have VM states correct in case of hypervisor or host fault. * BP exists for service disabled notification: * https://blueprints.launchpad.net/nova/+spec/service-status-notification * Ceilometer * https://review.openstack.org/#/c/172893/ * renamed from 'nortification alarm' to 'event alarm' * Deliverable status * http://artifacts.opnfv.org/ * Stable branch / branch cut * AoB * Ceilometer mid-cycle meetup * http://lists.openstack.org/pipermail/openstack-dev/2015-June/066618.html * Etherpad: #link https://etherpad.openstack.org/p/ceilometer-liberty-midcycle * committer list update IRC Meeting Logs: http://meetbot.opnfv.org/meetings/opnfv-doctor/2015/opnfv-doctor.2015-06-16-13.00.html ==== June 9, 2015 ==== Agenda: * BP Status * Nova * Ceilometer * Deliverable status * AoB * committer list update IRC Meeting Logs: http://meetbot.opnfv.org/meetings/opnfv-meeting/2015/opnfv-meeting.2015-06-09-13.02.html ==== June 2, 2015 ==== Agenda: * BP Status * Nova * Ceilometer * Deliverable Status IRC Meeting Logs: http://meetbot.opnfv.org/meetings/opnfv-meeting/2015/opnfv-meeting.2015-06-02-13.02.html ==== May 26, 2015 ==== Agenda: * OpenStack Summit report (related to Doctor) * Doctor Breakout Session * Ceilometer: event alarm * Nova session(s) * Short information from informal meeting with ETSI NFV REL, OPNFV HA and OPNFV Doctor at NFV #10 in Sanya * BP Status * Nova * Ceilometer * Deliverable * Promotions IRC Meeting Logs: http://meetbot.opnfv.org/meetings/opnfv-meeting/2015/opnfv-meeting.2015-05-26-13.02.html ==== May 19, 2015 ==== Canceled ==== May 12, 2015 ==== Agenda: * Deliverable * Status of Deliverable. * Vote on document approval (i.e. declare it stable). * Status of BPs * OpenStack Summit * Preparation for Doctor session * Related summit sessiones * Copper session * May 21, 9:50-10:30, Design Summit Ceilometer "Event alarms" * Meeting with ETSI NFV REL at NFV #10 * Joint work w/ ETSI NFV REL "Active Monitoring" Participants: Gerald Kunzmann, Ryota Mibu, Carlos Goncalves, Bryan Sullivan, Adi Molkho, Dan Druta, Michael Godley, Maryam Tahhan, Tomi Juvonen, Tommy Lindgren, Gurpreet Singh Minutes: * use IRC instead of Etherpad? * Ryota will check technical issues, e.g. using the MeetBot * Deliverables * we have not finished yet * further comments received today by E//// ; we also have to docx files from Dan * few minor syntax errors when compiling the document (in patch set 6) * Carlos is working with Octopus team to auto-generate HTML/PDF version of the document, but still buggy (false-positive in jenkins-ci) * can we get concensus on the content? would be nice to have a stable version of the document. * Voting can also be done via gerrit or email approval (tech-discuss list) * common in open source community; we can publish a stable version and then implement bugfixes afterwards * no objections in the call on the current version of the deliverable * Status of BPs * Ceilometer * https://review.openstack.org/#/c/172893/ * Nova * Second BP added to launchpad (-> should be reflected in the document) * https://review.openstack.org/#/c/169836/ * AI to Tomi to update Document * "Service group driver" / evacuation * OpenStack Summit * Tomi will mainly join the Nova sessions * Preparation for Doctor session * Monday 5pm (after Promise session) * Ryota is preparing presentation slides based on deliverable and Prague slides * will provide slides by Thursday to collect comments by Doctor team; Bryan asks to upload the draft slides beforehand to discuss it as soon as possible * new content compared to prague (https://wiki.opnfv.org/_media/doctor/opnfv_doctor_prague_hackfest_20150224.n.pptx): * collaboration with other OPNFV project team and other SDOs and communities * Ceilometer session * Copper session * ETSI NFV at NFV#10 * a meeting with REL is scheduled for ETSI NFV meeting in Sanya * Dan, Tommy, Gurpreet, Obana-san (DOCOMO) will be there * Joint work w/ ETSI NFV REL "active monitoring" * proposal for bi-weekly meeting (still under discussion) ==== May 5, 2015 ==== Agenda: * Status of Deliverable * Status of BPs * Participation at OpenStack Summit Minutes: * Status of Deliverable * https://gerrit.opnfv.org/gerrit/#/c/304/ * Gerald addressed several comments received by community members (e.g. David, Dan) * To be published by Vancouver Summit (as HTML? which server?) * Status of BPs * Tomi: struggles in Nova review; several iterations made (https://review.openstack.org/#/c/169836/) * NEC: https://review.openstack.org/#/c/172893/ * Participation at OpenStack Summit * https://etherpad.opnfv.org/p/Vancouver_OpenStack * prepare agenda with Ryota next week * Design Summit: Tomi will participate the Nova sessions * DOCOMO: Ashiq, Gerald, Malla * NEC: Carlos, Ryota, Dirk * Intel: Michael Godley * Tommy, Maryam, Adi, Dan will not be there -> try to talk to your colleagues going there to support Doctor * Design Summit: https://wiki.openstack.org/wiki/Design_Summit/Planning * AOB * Tommy: in document we state "Inspector might be based on Monasca" * Carlos: originally proposed by NEC; integrated with OpenStack; NEC found some "bugs" and "gaps" (e.g. delay is significantly more than 1s); meet them in Vancouver; it is a candidate, but no other platform seems to be integrated in OpenStack * Gerald: meeting with Fujitsu on Monasca two weeks ago * Carlos: pluggable architecture, could support Nagios or Zabbix * Gerald: in Monasca there is currently no requirement to do reporting within 1s * Last meeting with REL * has Tommy plan to meet with ETSI NFV REL? if time allows * ETSI NFV IFA * IFA documents not yet open to public ==== April 28, 2015 ==== Joint meeting with ETSI NFV REL team. Agenda: - Identify Purpose of the call * Collaboration kick-off - NFV REL: * Project Overview * NFV upgrade * Active monitoring and failure detection - OPNFV Doctor: * Project Overview * Use cases - Collaboration methodology discussion - Wrap-up Minutes: - Purpose * Ryota: know each other; see how to work together; further technology discussion needed at later stage * Markus Schoeller (NEC): no IPR declarations today, today only exchange of public information * policies how to work together w.r.t IPR etc should be defined for later work * Gurpreet: high-level of Doctor project; fault-detection and management; what are use cases of Doctor? - NFV REL introduction (Markus Schoeller) - Project overview: see ETSI/NFVREL(14)000200) * dedicated reliability project * Ryota: target size / number of applications? * Tommy: which work items focus on VIM part? indirectly addressed in monitoring and failure detection. scalabilty per se has some impact on VIM * Tommy: this means "monitoring and failure detection" would be the main crossing point with Doctor? so far yes, but in next meeting new WIs may be created - NFV software upgrade mechanism (Stefan Arntzen - Huawei) * different to traditional upgrades: "old traffic" can still go to "old software version", whereas new traffic/connections can go to the new s/w version in parallel (this is enabled by virtualization); no hard switchover needed; old system/version is still running and it can be switched back in case of issues with the new version * assumption is that this can be done stateless (otherwise it would be more complex) - Active monitoring for NFV (Gurpreet) * Alistair Scott: interested in passive monitoring; where as attachment points for passive monitoring? REL has not looked in passive monitoring for NFV * Gurpreet: identify use cases where current implementation has gaps - OPNFV Doctor * Stefan: plan to use OpenStack components? * Ryota: we are not only focusing OpenStack, but open source in general * Tommy: but OpenStack is the primary s/w used in OPNFV * Gurpreet: work flow for upstream community? * Ryota: define requirements, gap analysis, provide blueprints, but no coding in Doctor project - Next action: * arrange meeting in the next NFV event * keep in touch ==== April 21, 2015 ==== Agenda: - Deliverable * Structure: uploaded to Gerrit and split into multiple files; need consensus from community * Propose requirement project deliverable template based on Doctor's (WIP: Carlos, Ryota, Ikdiko) * Review comments received so far - Blueprints * Nova BP * OpenStack: https://review.openstack.org/#/c/169836/ * OPNFV: https://gerrit.opnfv.org/gerrit/#/c/371 * Ceilometer BP * OpenStack: https://review.openstack.org/#/c/172893/ * OPNFV: https://gerrit.opnfv.org/gerrit/#/c/278/ * Inspector API * https://wiki.opnfv.org/_media/doctor/opnfv_doctor_sb_api.20150414.pptx Minutes: * Status of BPs * Nova BP * concept has been acccepted * single API to mark down nova-compute and change status of VMs * the scope has been narrowed, topic was modified to "mark-host-down" * Ceilometer BP * trying to have summit session regarding Ceilometer event topic * demo that ryota mentioned ingerrit is the same as the prague hackfest * Deliverable * We still have review comments which are not reflected to doc yet * RST files has been splited, the format would be template for other requirement projects * how we can publish ... * Inspector API * API point is OK * action(doctor): describe framework and inspector API * Logistics * from next week, we will start to use IRC (e.g. sharing links) * at #opnfv-meeting channel * Next meeting * joint meeting with NFV REL * action: Ryota to send out agenda ==== April 14, 2015 ==== Agenda: - Status of BPs - Doctor requirement deliverable * https://gerrit.opnfv.org/gerrit/#/c/304/ - Doctor Southbound API * https://wiki.opnfv.org/_media/doctor/opnfv_doctor_sb_api.20150414.pptx * reference * Monasca Event API: https://wiki.openstack.org/wiki/Monasca/Events * Congress Design Doc: https://docs.google.com/document/d/1f2xokl9Tc47aV67KEua0PBRz4jsdSDLXth7dYe-jz6Q/edit Minutes: - Status of BPs * Nova BP: https://review.openstack.org/#/c/169836/ * Nova BP will be split * Ceilometer BP: * posted to OpenStack * Ceilometer BP: https://review.openstack.org/#/c/172893/ - Doctor requirement deliverable * Carlos to complete conversion by this week * contact Octopus on HTML conversion tool. If necessary, help Octopus talk to TSC about the positioning of Requirements deliverables - API in between OpenStack and HW/NFVI monitiring module e.g. Zabbix (Ryota) * Change "southbound API" to API in between OStack and HW/NFVI monitoring module ==== April 7, 2015 ==== Agenda: - Input from Swfastpathmetrics team - Status of BPs Minutes: - Input from Swfastpathmetrics team * current integration plan is https://collectd.org/ and also ceilometer * Revisit the scope of the NIC to makes sure that we can collect VF stats. * TBD * Can the NIC report VF/PF stats capabilities? Investigate: Maryam * I’ve been looking into this for Intel® 82599 10 GbE Controller, and this might be possible through a level of indirection by checking what VFs are enabled. It’s not exactly what’s being asked, but if you know what knew a VF was enabled then you’d know what stats are also available. * BTW: Stats can then be retrieved then per VF for Niantic: * VF Good Packets Received Count * VF Good Packets Transmitted Count * VF Good Octets Received Count Low * VF Good Octets Received Count High * VF Good Octets Transmitted Count * VF Good Octets Transmitted Count * VF Multicast Packets Received Count * But then error stats are still shared. * Open Maryam is looking into is if we knew the Queues that were assigned to a VF could we use Queue Packets Received Drop Count (QPRDC) to retrieve the dropped packets for a VF? * Maryam in the process of writing a DPDK app that runs as a secondary process on the host and is capable of reading the stats, which can then be parsed by a script. * Still in progress * No Southbound interface for Doctor defined yet. * Action: Ryota to draft SB API of Doctor - Status of BPs * https://review.openstack.org/#/c/169836/ * Actions Ryota to upload Ceilometer BPs ==== March 31, 2015 ==== Agenda: - Status of requirement deliverable - Status and next steps of BPs (Tomi, Ryota) - Nova BP review - Input from Swfastpathmetrics team Minutes: - Status of requirement deliverable: Distributed to OPNFV community - Discussion about leveraging OpenStack Zaqar as multi-tenant messaging system for real-time event notifications - HTTP vs SNMP - Input from Swfastpathmetrics team: * https://collectd.org/ * The following set of statistics are those that are collected explicitly for the physical NIC (Intel 10G Niantic) * Physical Function (PF) Summed up/total ReceiveErrors/Drop Statistics Reported (note** these are per Port) * PF Packet drop: Sum of ( all drop registers we list)excluding error register * PF Packet errors: Sum of ( all error registers we list)excluding drop registers * PF PHY errors: Sum of (all PHY errors we list) * PF General errors: Sum of (all other rx/tx errors regs welist) * PF Missed RX: A missed packet is a packet that was correctly received by the NIC but because it was out of descriptors and internal memory, the packet had to be dropped by the NIC itself * Physical Function (PF) IndividualReceive Errors/Drop Statistics Reported * Illegal Byte Error Count: Counts the number of receive packetswith illegal bytes errors (such as there is an illegal symbol in the packet). * Error Byte Count: Counts the number of receive packetswith error bytes (such as there is an error symbol in the packet). * Rx Missed Packets Count * MAC Local Fault Count : Number of faults in the local MAC. * MAC Local Fault Count: Number of faults in the remote MAC. * Receive Length Error Count: Number of packets with receive length errors. A length error occurs if an incoming packet length field in the MACheader doesn't match the packet length. * Receive Undersize Count:. This register counts the number of received frames that are shorter than minimum size (64bytes from through , inclusively), andhad a valid CRC. * Receive Fragment Count: Number of receive fragment errors (frame shorter than 64 bytes from through ,inclusively) that have bad CRC (this is slightly different from the ReceiveUndersize Count register) * Receive Oversize Count: This register counts the number of received frames that are longer than maximum size as defined by MAXFRS.MFS (from through ,inclusively) and have valid CRC. * Receive Jabber Count: Number of receive jabber errors. This register counts the number of received packets that are greater than maximum size and have bad CRC (this is slightly different from the Receive OversizeCount register). The packets length is counted from through , inclusively. * Management Packets Dropped Count: Number of management packets dropped.This register counts the total number of packets received that pass the management filters and then are dropped because the management receive FIFO is full. Management packets include any packet directed to the manageability console (such as RMCP and ARP packets). * MAC Short Packet Discard Count: Number of MAC short packet discard packets received. * XSUM Error Count: Number of receive IPv4, TCP, UDP or SCTPXSUM errors. * FC CRC Error Count: Count the number of packets with goodEthernet CRC and bad FC CRC * FCoE Rx Packets Dropped Count: Number of Rx packets dropped due to lack of descriptors. * And the final thing we measure is packet latency through DPDK. ==== March 24, 2015 ==== Participants: Ryota, Tomi, Bertrand, Gerald Agenda: - Status and next steps of BPs (Tomi, Ryota) - Nova BP review - Input from Swfastpathmetrics team - Status of requirement deliverable - Document Review Minutes: * Recap of last weeks BP meeting (Thursday) * Ryota had presented; request was to use the template; more details needed, but general approach is okay * Bryan has raised issue that more discussion on notification on NB I/F interface is needed * important which protocol and data structure is used * Proposal of Doctor should be aligned with other projects * Ashiq: does anyone have experience in writing BPs -> Tomi: using the template it was straightforward * proposal to find someone with experience. Ryota already has some experience. * Ryota: problem is who could review our BPs. we need to socialize with community. Ryota has some channels he can use for this. * it is not clear if we need TSC approval for submitting the BPs; at least we should align at OPNFV level * Ashiq: it is important people join the meetings, e.g. the BP meeting, the individual BP meetings, socialize the BPs in the OPNFV community (by discussing it on the mailing list) * Tomi: proposal to send mail to tech-discuss asking for any objections within few days, then upload * Ryota: we need this window to also review other contributions * Ashiq: already upload the BPs and revise it in case needed * Status and next steps of BPs (Tomi, Ryota) * Tomi: BP following template is ready * Ryota: target is Thursday to have BPs ready following the template * Nova BP review * https://etherpad.opnfv.org/p/doctor_bps * References: sufficient to refer to Doctor project * Input from Swfastpathmetrics team * no one from this team; skip topic for next week * Status of requirement deliverable * draft distributed yesterday. * ACTION: perform Doctor internal review by end of this week. * Stable draft will be provided to two weeks OPNFV-wide review on Monday March 30th * Ashiq: architecture with 4 building blocks. Inspector filters some fault information. In Notifier is a policy to also filter which fault informations to send or not. * why do you enable the filtering in both Inspector and Notifier? Ryota: different kind of filters. in OpenStack all alarms from Controller will be emitted, thus we need to have policy to filter. * Ashiq: Inspector is filtering physical faults. notifier is filtering faults on virtual resource level. Correct? * Document Review * Others: * join HA meetings time-by-time to monitor their progress; check status of their BPs and offer help. * Ryota had joined the last HA meeting: main activity is requirement document; no discussion regarding BP ==== March 17, 2015 ==== Agenda: * Status of requirement deliverable (Gerald) * Review of document * Status of BPs (Tomi, Ryota) * https://wiki.opnfv.org/_media/opnfv-doctor-nova-blueprint.pptx * https://wiki.opnfv.org/_media/doctor/opnfv_ceilometer_bp_alignment.20150306.pptx Minutes: * Status of requirement deliverable * Updated interface specification andrelated information elements in Section 5 * New (merged) “Figure 12 – Implementation planin Ceilometer architecture” as discussed in last call. * 5.1.3: Tommy pointed that Controller does not have to notify updated capacity to other moules, this is out of our scope * 5.2.1: we should remove step 9 and step 10 from Figure 7 - Fault management work flow * Status of BPs * Nova: * see slideset from 10.3.2015 * Gap is "immediate notification" for faults over the NB I/F * Two alternative ways to implement it: * New Nova API -> might not be accepted easily * offline mail discussion ongoing * Use Pacemaker servicegroup driver -> might not address all requirements of Doctor * TL: Is there some possibility to ask Nova to update its state? TJ: currently doing some testing on this; also to measure the reaction times * TL: servicegroup is within one cluster. there is also a pacemaker remote, but might be slow * RM: blog post by Russell: http://blog.russellbryant.net/2015/03/10/the-different-facets-of-openstack-ha/ * Ceilometer * proposal to discuss "Event-Publisher for Alarm" and "Notification-driven alarm evaluator" for discussion in Thursday tech meetings (see email from Hu, Bin) * Ryota is doing code review in OpenStack * plan is to stop new "meter" due to performance issues. idea is to have a new type of notification. * TL: feedback from Ceilometer team? no yet, Ryota has not introduced Doctor project in this community. plan to do after OPNFV BP review. ==== March 5, 2015 ==== Agenda: * Status of requirement deliverable (Gerald) * https://wiki.opnfv.org/_media/doctor/opnfv_doctor_d1_doctor_fault_management_and_maintenance_draft_r9.docx * Status of BPs (Tomi, Ryota) * https://wiki.opnfv.org/_media/opnfv-doctor-nova-blueprint.pptx * https://wiki.opnfv.org/_media/doctor/opnfv_ceilometer_bp_alignment.20150306.pptx Minutes: * Status of Document * 3.1 there is unclaified maintenance usacese --> Tommy will send ETSI Doc Ph2 to us * 3.1.2 text is missing --> Gerald can add text * 3.5 * 4 Gap analysis: are there anyrelated BPs that should be added as references? --> BP links should be added * Carlos will help check if there are blueprints already filled in and add to the document * 5 Detailed implementation plan --> all, please check this chapter * Fig.9 what is maintenance sequence loke like? --> TBD (tentative FB and sequence are proposed) * Schedule * 1week for Doctor internal review (-3/17) * 2 week for OPNFV community review (-3/31) * 2 week for Doctor for team work (-4/14) * (OPNFV release 1 4/23) * Status of BPs: not handled ==== March 5, 2015 ==== Ad-hoc meeting for blueprint planning Agenda: * Discuss BPs * Report on meeting with HA team Minutes: * Presentation by Ryota on Ceilometer and where our BPs fit to the Ceilometer architecture (see slides) ==== March 3, 2015 ==== Agenda: * BPs alignment * Mike from Intel / swfastpathmetric will join this team * Hijacking the doctor meeting to discuss blue-prints next week for all projects * Doc status Minutes: * BPs alignment * https://etherpad.opnfv.org/p/doctor_bps * Bring BP topic also to TSC * BP should have a list of parameters/data missing * OpenStack BPs shall be in this format (using OpenStack terminology such that a developer can read/understand it) * Proposal to have a high-level description of the BPs in the Wiki * Ceilometer is the right place to implement such feature, although other alternatives may exist * TODO(Ryota): prepare slides, provide IRC available time * Hijacking the doctor meeting to discuss blue-prints next week for all projects: * we should keep 30min for discussing the requirement deliverable * Doc status: not handled ==== Feb 24, 2015 ==== Requirement project round table @ Prague Hackfest Participants: Ryota (NEC), Gerald (DOCOMO), Bertrand (DOCOMO), Ashiq (DOCOMO), Tomi (Nokia), Tommy (Ericsson), Carlos (NEC), Gianluca Verin (Athonet) Daniele Munaretto (Athonet), Sharon (ConteXtream), Christopher (Dorado Software), Russell (Red Hat), Frank Baudin (Qosmos), Chaoyi (Huawei), Al Morton (AT&T), Xiaolong (Orange), (Oracle), Randy Levensalor (CableLabs) ... Slides can be found here: https://wiki.opnfv.org/_media/doctor/opnfv_doctor_prague_hackfest_20150224.n.pptx Minutes: * Use case 1 "Fault management" * Main interest: northbound I/F * Reaction of VNFM is out of scope * VM (compute resouces) is the first focus, storage and network resources will follow at later stage * Fault monitoring: plugable architecture is needed to catch different (critical) faults in NFVI to enable use of different monitoring tools. Predictor (fault prediction) may also be one input. * see also https://wiki.opnfv.org/doctor/faults * 4 functional blocks: * controller (e.g. Nova), monitor (e.g. Nagios, Zabbix), notifier (e.g. re-use Ceilometer), inspector (fault aggregation etc) * VM state in resource map, e.g. "fault", "recovery", "maintenance" (more than just a heartbeat) * Question of whether other OpenStack components (e.g. Cinder, Glance, etc) can report events/faults * What is the timescale to receive such fault notification? this would be helpful for the motivation in the blueprints. Telco nodes: i.e. less than 1s, switch to ACT-SBY as soon as possible. * Preference is event based events, not polling. should be configurable. * Telco use case would have few hundreds of nodes, not thousands of nodes. * Demo 1 (using default Ceilometer) takes approximately 65 seconds to notify the fault (90 seconds total including spawning new VM), while demo 2 only takes <= 1 second (26 seconds total) * Pacemaker is running at application layer; different scope. ==== Feb 23, 2015 ==== Doctor/Fastpathmetrics/HA Cross Project Meeting @ Prague Hackfest Goal: * reduce conflicts between requirements projects Minutes: * Project Intro: * Doctor: * detecting fault from resource pool, HW, hypervisor, OS * table in Openstack mapping between VM and manager * Fault prediction (Predictor, Requirement project; https://wiki.opnfv.org/requirements_projects/data_collection_of_failure_prediction) * predict the will be fault in (near) future * overlap, but not conflict * FathPathMetrics (Development project; https://wiki.opnfv.org/collaborative_development_projects/opnfv_telco_kpi_monitoring) * SW fastpath implementation -> interfaces to DPDK * consume statistics and counters * information goes to VNF, it does not go to OpenStack (in that sense there is no overlap) * HA (Requirement project; https://wiki.opnfv.org/high_availability_for_opnfv) * quick response. time-critical. * very short response for fault detection and resolve * people from HA project really work on HA today, have a lot of knowledge on it * Identify Overlap: * NB I/F * Doctor is also requiring fast reaction. objective with HA is similar. * HA has more use cases and may send more information on the northbound I/F. VNFM should be informed about changes. * Doctor objective is to design a NB I/F. * Does HA already have flows available? * HA is focusing on application level. Reaction should be as fast as possible. Including the VNFM may slow down the progress. * In Doctor we will follow the path through VNFM. * In ETSI we have lifecycle mgmt, where VNFM is responsible for the lifecylce * There are certain information the VNFM doesn't know about. In Doctor we call it "consumer". * Proposal to do use case analysis for HA. Which use cases may require the VNFM to be involved? "Doctors" will have a look at HA use cases. * How is the entity to resolve race conditions? Some entity in the "north". * What about a shared fault collection/detection entity instead of collecting the same information 3 times? * Predictor could also notify immediate failures to Doctor. * Security issues are not addressed in Doctor. Currently assuming a single operator, where policies ensure who can talk to who. * In Doctor we do not look at application faults, only NFVI faults. * Huawei: we use Heat to do HA. if one VM died and Heat will find Scaling Group less than 2, it will start a new VM. This may take more than 60s, we need to find something faster for HA. Heat doesn't find error in the applications. * Failure detection time is an issue across all projects. * Which metrics of fastpath would Doctor be interested in? need to check in detail. Action Item to send metrics to Doctor. * Hypervisor may detect failure of VM and take action. * Other failures: VM is using heartbeat. it will e.g. reboot after not receiving a heartbeat for 7s. * Doctor: if VIM takes action on its own it may conflict the ACT-SBY configuration at the consumer side. this is why the consumer should be involved. * Which project would address ping-pong issue that may arise? * We need subscription mechanism including filter (which alarms to be notified about). Mapping VM-PM-VNFM can be recorded during the instantiation. * Relationship between Doctor and Copper: * policy defines e.g. when VIM can expose its interface * When to inform a fault, whom to inform etc is all a kind of policy. * Copper has both pro-active and reactive deployment of policies. In reactive case, there may be conflict when both Copper and Doctor receive the policies. * Wrapup: * Overlap in fault management * FastPath: monitor traffic metrics; Doctor will need some of the metrics in the VIM. plan to do regular meetings. * HA: large project with wider scope than Doctor, different use cases. direct flow (to be faster). task to check each others NB I/F in order not to block each other. ==== Feb 17, 2015 ==== Agenda: * Ashiq's proposed agenda for the Prague Hackfest next week * Doctor PoC Demo * Document Status Minutes: * Participants: Ryota Mibu, Khan Ashiq, Gerald Kunzmann, Carlos Goncalves, Susana, Thinh Nguyenphu, Tommy Lindgren, Bryan Sullivan, Bertrand Souville, Michael Godley, Manuel Rebellon, Uli Kleber * Hackfest * https://etherpad.opnfv.org/p/HackfestFeb23 * Ryota to prepare some slides on what's going to be presented in the demo by end of week * Carlos will help Ryota * Requirement projects are scheduled for Tuesday * Demo: * OpenStack Controller, Zabbix, 2-3 OpenStack compute servers to launch VMs, client to stress the system, Neutron, Nova, LB as a service, Heat, Ceilometer * Destroy one of the VM running a WebService * Key message to OpenStack? Which gap do you want to present? Why use Zabbix instead of Ceilometer (show first gap in our list)? Prepare for such questions. * Document status * Gerald to upload current merged draft -> Word version can be found here: https://wiki.opnfv.org/?image=doctor%3Aopnfv_doctor_d1_doctor_fault_management_and_maintenance_draft_r2_ak.docx&ns=doctor&tab_details=view&do=media&tab_files=files * All links can be found in https://etherpad.opnfv.org/p/doctor * Urgent work is required on technical sections * HA and fault prediction project and "Software FastPath Service Quality Metric" project * Proposal to meet Monday afternoon after BGS project meeting * Carlos will contact all potential participants ==== Feb 10, 2015 ==== Agenda: * TOC for deliverable: https://etherpad.opnfv.org/p/doctor - Gerald * Implementation Plan quick update - Ryota Minutes: * OPNFV should be careful with tools projects use and distribute as part of the platform due to their licensing * E.g. a GPL licensed tool cannot be distributed as part of a commercial OPNFV platform * OPNFV TSC should discuss this topic and bring awareness to existing and new projects * http://lists.opnfv.org/pipermail/opnfv-tech-discuss/2015-February/001051.html * Framework should be modular enough to be pluggable with multiple monitoring solutions * Editors for each first deliverable section were assigned * Gap analysis to be further extended * Section editors should have an initial draft ready by Feb 18 * Deliverable editors (Gerald and Ashiq) will have Feb 19-20 to compile everything together for the Prague Hackfest ==== Feb 6, 2015 ==== Extra meeting for Implementation Planning Agenda & Minutes: * Implementation Planning * Topic and agreement can be found in {{:doctor:doctor.20150206.impl.mtg.pptx|Slides}}. ==== Feb 2, 2015 ==== Agenda: * Timeline - Ryota * This proposal has close relation: data collection of failure prediction * https://wiki.opnfv.org/requirements_projects/data_collection_of_failure_prediction * Both collect host data? * Doctor to receive faults from predictor. * Is this feasible. Doctor can be configured to gather also similar data and make graphs. * Praque meetup time. * Implementation plan: review comments by Tomi in last week's minutes * Wiki Updates to follow BGS format - Ryota * Doctor team participation in the OpenStack Summit Vancouver - Carlos * Should we aim at proposing a talk session? * Deadline is February 9 * http://www.openstack.org/summit/vancouver-2015/call-for-speakers/ * Weekly meeting time - Ryota ? * There is conflict with ETSI IFA meeting. Minutes: * Participants: Carlos Goncalves, Don Clarke, Ryota Minu, Tomi Juvonen, Yifei Xue, Al Morton, Bertrand Souville, Gerald Kunzmann, Manuel Rebellon, Ojus K. Parikh, Ashiq Khan, Pasi, Paul French, Charlie Hale, * Ryota presents a refreshed Timeline * https://etherpad.opnfv.org/p/doctor * Initial draft of requirement document should be ready before the Hackfest 23-24 Feb in Prague * Ashiq asks about task allocation. See: https://etherpad.opnfv.org/p/doctor * Target architecture is OpenStack; Implementation plan is on how this will be realized in upstream projects, e.g. interfaces. * one proposal is using Zabbix. all is already there. * Predictor project: * still in proposal phase. we should keep eye on it. it has relation to Doctor * Implementation plan: * for evacuation we should stay implementation independent, not OpenDayLight or Neutron (they may use it in the actual testbed, but we should restrict Doctor to the interfaces definition) * it is not intended to use Ceilometer, but a similar service. * Agreement to use Zabbix for the GapAnalysis. * Doctor will have its own RestAPI as wrapper abstracting the in use monitoring solution underneath (e.g. Zabbix) * it is necessary to be able to isolate a faulty machine, such that new VMs are not started on this machine. * different ways/workflows for recovery; we should start by implementing a few sample workflows * e.g. switch to active hot standby VM, then instantiate a new hot standby instance (this is a Doctor requirement) * evacuation (if time allows) vs active hot standby (immediate action) *VNFM is deciding about the best action (this is out of scope of Doctor; Doctor only specifies NB I/F) * we need to get into more details for this plan. discussion should go via email to make progress before next meeting * Hackfest * Take to the hackfest what we have, i.e. if we "only" have one implementation plan so far let's use this. * https://etherpad.opnfv.org/p/HackfestFeb23 * Doctor is planned for Tuesday. Also other requirement projects will be discussed on Tuesday. * Ryota did cleanup of Doctor Wiki page * Doctor team participation in the OpenStack Summit Vancouver? * related topics. * most important blueprints should be ready by May and could be presented there * Proposal: Talk on a more general topic including Doctor requirements * Carlos will look into it * Meeting time -> via email ==== Jan 26, 2015 ==== Agenda: * Discuss maintenance use case - Tommy * Implementation outside Nova - Tomi Minutes: * Timeline milestone planning * Soft schedule for Fault Table, set 1 milestone end of Jan * Requirement Document should be finished by Mar 15 ? - No * Ashiq suggested doc should be finished by end of February * Set some milestone on Hackfest at Prague * First draft * TODO(Ryota): create wiki page * Discuss maintenance use case - Tommy * https://etherpad.opnfv.org/p/doctor_requirement * Cloud User (NFVO/VNFM) can express allowed operations to VIM by labeling VM some tag e.g. allow-live-migration * We don't need realtime notification for maintenance * Implementation outside Nova - Tomi * https://wiki.opnfv.org/_media/doctor-opnfv-proposal.pptx * FYI: http://blog.russellbryant.net/2014/10/15/openstack-instance-ha-proposal/ * Network resources? * Evacuation will move the network also regardless it being OpenDaylight or Neutron. * We are trying to tackle step-by-step, first focusing on Nova. * Ceilometer approch seems to be good rather than using metadata on Nova * What is the relation to Nova metadata? Ceilometer is therrible for FM. It uses polling, and suits for PM. It would be extra step causing delay. It makes a lot of network traffic. Database consumes a lot of memory. * Should we kick poweroff the host by Doctor? * One needs to fence host by powering off by shutdown trough OS (or Nova) or if reachable only with IPMI, then trough that. In some case host can be rebooted as recovery, but in most cases it is faulty and needs to be moved to disabled aggrigate or mark for maintenance. If one do not reach host at all, the evacuation trough Nova will anyhow isolate host as everything will be moved to other host (network, disk). ==== Jan 19, 2015 ==== Agenda: * Review of timeline of Doctor project * List of tasks Minutes: * Review of timeline of Doctor project * https://wiki.opnfv.org/_media/requirements_projects/timeline_doctor_r1.png * Milestone for First Spec ready might be put on OPNFV hackfest in Prague * Submit blueprints to Kilo has been Postponed * Action: Ryota to revise timeline table * Gap Analysis. * Third gap has been added to the etherpad (Fencing instances of an unreachable host in Nova) https://etherpad.opnfv.org/p/doctor_gap_analysis * Please all review / add new gaps to the etherpad * Implementation Plan * Discussion last week on good candidate monitoring projects for Doctor project * Need also to work on implementation plan / architecture. * Action: Tomi (Nokia) to come up with high level architecture proposal * List of tasks * Task 1 (Ryota) * Timeline milestone planning * Task 2 (Gerald) * Fault Table * Task 3 # punt * Gap Analysis * Task 4 (Tomi) * High level arch * Out-side of Nova arch first, Nova * One view for this here: http://blog.russellbryant.net/2014/10/15/openstack-instance-ha-proposal/ * Here is a picture for discussion to ipmlement outside of Nova: https://wiki.opnfv.org/_media/doctor-opnfv-proposal.pptx * Task 5 (Ryota) * Wiki update (meeting minuts and so on) * Task 6 # TBD * NB I/Fs * Messaging * Information elements * Impl. plan(Gathering faults) ==== Jan 12, 2015 ==== Agenda: Minutes: * Fault table: https://wiki.opnfv.org/doctor/faults * Action: check and revise / update / extend * Some faults are specific to a certain HW other are more general * We should try to come up with a high level description of common faults * Proposal not to go to such level of detail. * Keep one fault table and use the current fault table for study of the scope of Doctor * Are there other faults that cannot be detected by SNMP and Zabbix_agent? * We need a tool in Doctor that can retrieve such alarms. Should this tool be integrated with OpenStack or be independent? Should be kept open; * different tools may be used given clear interfaces are specified * Monasca: https://wiki.openstack.org/wiki/Monasca OpenStack project for monitoring of faults; still in its early days * ETSI meeting in Prague: proposal to meet there * https://etherpad.opnfv.org/p/HackfestFeb23 * Action: update your participation on that EtherPad page * Doctor "working page": https://etherpad.opnfv.org/p/doctor_gap_analysis * Action: edit this page for ongoing work on the gap analysis * Doctor wiki page updated * Use cases moved to extra page: https://wiki.opnfv.org/doctor/use_cases * Timeplan: * Action: Ryota to prepare a timeplan/timeline * Timeplan can be checked in each week's meeting * seems we are behind initial time plan (see https://wiki.opnfv.org/_media/doctor_141208.pdf slide 11) * Reminder: some documents should be available by March * Next meeting: Jan 19th ==== Dec 22, 2014 ==== Agenda: * work item updates * Fault table * GAP analysis template * Wiki pages Minutes: * Work item updates * Fault table * Status: waiting Palani's initial commit * Tomi also made initial list of faults. * TODO(Tomi): Open new wiki page to share the fault list * GAP analysis template * Ryota explained GAP analysis template drafted on https://etherpad.opnfv.org/p/doctor_requirement * TODO(Ryota): Open new etherpad page for Gap Analysis with the template * Wiki pages * Our plan for wiki/doc structure seems to be OK, cause there was no question and objection in the past week. * TODO(Ryota): Update wiki pages * Fault notification at the Northbound I/F * Critical faults * It was agreed that we should characterize faults as critical ornon-critical when reporting to VNFs. * We must report all critical faults northbound. We may report some of the non-critical faults, need further study. * Fault aggregation * Discussed whether toaggregate different alarms and faults before notifying via northbound interfaceto VNFs. * General agreement that there should be some level of aggregation, butneed to figure out what events need to be aggregated. * Some suggested that VNFs should be notified only if the faults are urgent. * Notifying data center operations folks about hardware faults is something that seems to be out of scope for this project. Tomi: I think they need the information and there should not be a duplicate mechanism to detect faults to be able to make HW maintenance operations. Surely they will not need the notification that we would send to VNFM, but the actual alarm information we are gathering to make those notifications. Anyhow I agree that this is not in our scope and tools like Zabbix that we could use here can easily be configured then for this also in case HW owner is interested. * Why should warnings be sent to VNFs (such as cpu temp rising but notcritical yet)? VNFs might want to take action such as setup/sync hot standbyand this could take some time. * Are there open souce projects already to detect hypervisor or host OS faults? * OpenStackNova devs said it should be kept simple, providers need to monitor processes ontheir own. * But there appears to be some open source tools(SNMP polling or SNMP agents on host). Need to pull things together. * Next call will be on January 12th. ==== Dec 15, 2014 ==== Agenda: Minutes: * wiki/doc structure * Agreed to have three sections * UseCase (High-level description) * Requirement (Detail description, GAP Analysis) * Implementation(includes monitor tools and alternatives) * Faults table * will create table that explain stories for each fault * columns would be physical fault, how to detect, effected virtual resource and actions to recover * in three categories Compute, Network and Storage, will start on Compute first * also try to keep separate table/categories for critical and warning * TODO(Palani): provide fault table example * TODO(Gerald): create first version of fault table after getting table example * framework * how we handle combination of faults and feature H/W faults that is still open question * suggestion to have fault management "framework" that should be configurable to define faults by developers or operators * Gap analysis * We should have list of items so that we can avoid duplicated work * TODO(Ryota): Post first item to show example how we describe that could be template for GAP analysis * Monitoring * We should check monitoring tools as well: Nagios, Ganglia, Zabbix * Check TODOs from the last meeting * seems almost all items have done or started (but we could not check 'fault management scenario based on ETSI NFV Architecture' although there is silde on wiki) * Next meetings * Dec 22, 2014 * Jan 12, 2015 # skip Jan 5th ==== Dec 8, 2014 ==== Agenda: * How we shape requirements * Day of the week and time of weekly meeting * Tools: etherpad, ML, IRC? * Project schedule, visualiztion of deliverables Minutes: * How we shape requirements * Use case study first * Gap Analysis should be included existing monitoring tools like Nagios etc. * How we format fault message and VNFD elements for alarms? * Fault detection should be designed within a common/standard manner * Those could be implement in existing monitoring tools separated from OpenStack * What is "common" monitoring tools, there are different tools and configurations * Focus on H/W faults * Do we really need that kind of notification mechanism? Can we use error from API polling, just get error detected by application or auto-healing by VIM? * Real vEPC needs to know fault that cannot be found by application like abnormal temperature. * VIM should not run auto-healing for some VNF. * There are two cases/sequences defined in ESTI NFV MANO that fault notification are send from VIM to VNFM and to Orchestrator. * Alarming mechanism is good to reduce the number of request from user who pooling virtual resource status. * We shall categorize requirements and create new table on wiki page. (layer?) * -> A general view of the participants is to have the 'HW monitoring module' outside of OpenStack * TODOs * Open etherpad page for collaborative working (Ryota) * Collect use cases for different fault management scenarios (Ryota) * Set IRC (Carlos) * Provide Gap Analysis (Dinesh, Everyone) * Provide fault management scenario based on ETSI NFV Architecture (Ashiq) * List fault items to be detected (Ashiq, Everyone) * Day of the week and time of weekly meeting * Monday, 6:00-7:00 PT (14:00-15:00 UTC) * TODO(Ryota): create weekly meeting entry in GoToMeeting * Tools: etherpad, ML, IRC? * We will use opnfv-tech-discuss ML with "[doctor]" tag in a subject. * We will use "opnfv-doctor" IRC channel on chat.freenode.net . * TODO(Carlos): update wiki * Project schedule, visualiztion of deliverables * All team members are asked to check project proposal page and slides that are approved by TSC and show our schedule and deliverables. * Northbound I/F first specification by Dec 2014. ==== Dec 1, 2014 ==== Agenda: * Project Proposal Refinement * https://wiki.opnfv.org/doctor/project_proposal * Project Categories * Deliverables and Schedule - SR Minutes: * Project proposal * There were two comments at project review in TSC meeting (Nov 26) * Large scope: TSC asked to narrow project scope, creating as requirement projects seems reasonable * Overlap with HA project, so collaborate in project activity * https://wiki.opnfv.org/requirements_projects/high_availability_for_opnfv * Ashiq and Qiao had talked before this meeting, and agreed that we would not eliminate duplication at proposal phase * Project proposal was fixed by some members * https://wiki.opnfv.org/doctor/project_proposal * The project categories was changed to requirement only * In new revision of project proposal, we removed detailed descriptions which don't suit requirement project * Links to original project proposal are replaced to point the new page, and the link to the old page that described further details can be found at the bottom of the new proposal page * We should not edit the proposal page after TSC approval to keep evidence what we planed at the beginning of the project * "Auto recovery" is missing, will continue discussion in mail with clarification by Tomi ==== Nov 17, 2014 ==== Agenda: * Scoping and Scheduling (what feature to be realized in what time frame) * Resources available and necessary for this project * Technical aspects and relevance to upstream projects * How to socialize with upstream projects [[doctor_minutes_20141117|Minutes]]