Wiki

This weekly meeting is for topics common across projects focused on testing … this is a good place to start if you want to get high-level status of the individual projects and learn about hot topics related to testing OPNFV

• Test projects should try to have a rep attend this meeting
• New test related projects should be discussed in this forum to gain feedback from the OPNFV testing community

• Agenda varies but typically is structured …
  • 15 min report out / sharing by projects (high level status / plans)
  • 20 min discussion on test project collaboration topic/s
  • 20 min – share new test projects or other test topics

• There is a Test and Performance Operational Meeting immediately after (topics include dashboards, tools, KPIs, traffic profiles, etc.) See https://wiki.opnfv.org/meetings

• When: Every Thursday 7:00-8:00 PDT PST (14:00-15:00 15:00-16:00 UTC)
  • Your local time and Every two weeks on Wednesday 8h30 UTC (Asian/European slot)

• Meeting notes (MeetBot) … http://meetbot.opnfv.org/meetings/opnfv-meeting/
• Etherpad: https://etherpad.opnfv.org/p/testperf

• IRC channel: #opnfv-meeting @ Freenode

• Join the meeting from your computer, tablet or smartphone
  • Thursday GTM: https://global.gotomeeting.com/join/305553637
    • Access Code: 305-553-637
    • Phone numbers: https://global.gotomeeting.com/305553637/numbersdisplay.html
      • United States : +1 (626) 521-0010
      • Canada : +1 (647) 497-9371
      • Finland : +358 (0) 942 41 5770
      • France : +33 (0) 182 880 159
      • Germany : +49 (0) 692 5736 7208
      • Sweden : +46 (0) 852 500 179
      • …
  • Wednesday GTM: https://global.gotomeeting.com/join/644444853
    • Access Code: 644-444-853
    • Phone numbers: https://global.gotomeeting.com/644444853/numbersdisplay.html
      • Germany : 0 800 723 5274
      • China : 4007 160008
      • Korea, République de: 0806180880
      • Spain: 800 900 593
      • Russia: 8 800 100 6216
      • Finland: 0 800 94473
      • France: 0 805 541 052
      • India: 000 800 100 8227
      • Japan: 0 120 242 200
      • Sweden: 0 200 330 924

Useful links …

Testing overview page with links to testing projects

Pharos project page

Upcoming Agenda Topics …

• ?

Meetings …

• Plugfest
• Resources for release C

• Benchmarking VNF/VNFIs - 30 min (Al)
• Storperf - 15 min (Mark)
• Pharos - 10 min (Trevor)

• Continue from last week with what Test Projects delivered in release B …

• VSPERF - 15 min (Maryam)
• Bottlenecks - 15 min (Hongbo / Yaoguang)
• Storperf - 15 min (Mark)

• Time allowing → Release C common topics across test projects including test cycles, test strategy, infrastructure, coverage, automation, etc.

• Test projects - what was delivered in release B …

• Qtip demo - 15 min
• Functest - 15 min
• Yardstick - 15 min
• VSPERF - 15 min
• Bottlenecks - 15 min
• Storperf - 15 min

May run over into Yardstick meeting (or continue next week)

• Plugfest test-cases (30 min)
• Testing plans for release C (25 min)

Meeting Cancelled

1. Plugfest testplan
2. Brahmaputra test readiness status
3. Test success criteria (now that scenarios clarified)
4. Plan for the visualization of the results (alternative to LF dashboard)
5. Organization Asian/European slot
6. Troubleshooting/debugging slot

1. Brahmaputra release (35 min)
   • https://wiki.opnfv.org/brahmaputra_testing_page
   • https://wiki.opnfv.org/releases/brahmaputra/release_plan#brahmaputra_release_participating_projects
   1. Test project status (5 min)
      • yardstick, functest, bottleneck, vsperf, qtip
      • Storperf
      • Companion projects
   2. Slots for testing / responsibility to trigger redeployments (5 min)
   3. Migration of test DB (5 min)
   4. Results display / dashboards (10 min)
   5. Troubleshooting organization/planning (10 min)
      • Success criteria / SLA
      • https://wiki.opnfv.org/vping_brahmaputra_page
      • https://wiki.opnfv.org/tempest_brahmaputra_page
2. Test meeting organization (5 min)
3. Test project participation in Dovetail / plugfest (10 min)

• Test dashboard demo (Marion, Andrew - LF) - 15 min

• Update on test scenarios - 15 min
  • https://wiki.opnfv.org/brahmaputra_testing_page

• Test artifacts for Rls B - 15 min
  • Test results
  • Test results sample / template
  • Other test documents?

• Test projects status updates - as time allows (few min per project)
  • https://wiki.opnfv.org/brahmaputra_testing_page
  • https://wiki.opnfv.org/releases/brahmaputra/release_plan#brahmaputra_release_participating_projects
    • pharos
    • yardstick
    • functest
    • bottleneck
    • vsperf
    • qtip

• Test dashboard - 10 min
  • Update on portal prototype status and roadmap (Marion, Andrew - LF)
  • Next steps and anything needed from other projects (Morgan)

• Test projects status for Rls B - 15 min (2 min per project)
  • https://wiki.opnfv.org/brahmaputra_testing_page
  • https://wiki.opnfv.org/releases/brahmaputra/release_plan#brahmaputra_release_participating_projects
    • pharos
    • yardstick
    • functest
    • bottleneck
    • vsperf
    • qtip

• Test Scenarios for rls B - 15 min
  • Review … https://wiki.opnfv.org/brahmaputra_testing_page
  • Clarify how test projects are managed from CI perspective
  • Questions on CI integration of feature projects / testing

• Documentation for Rls B - expectations and status - 15 min
  • Test result documentation - define and agree what to document
  • Pharos documents / templates

• Status of the test projects (Morgan) 20 min
  • Functest … CI labs - Fuel, Joid, Compass, Apex
  • Yardstick
  • Other test projects
• Retro planning attempt for B-Release from now to …. 20min
• Showcasing OPNFV test results using Kibana (Frank Brockners and Peter Bandzi) - 15 min

As part of test meeting, we’ll also continue the discussion on common configuration files – with the plan to evolve things to closure asap.

Agenda items for this topic include:

• finalize common inventory config file: https://gerrit.opnfv.org/gerrit/#/c/4079/
• review common network config proposal (see Jonas proposal) - https://gerrit.opnfv.org/gerrit/#/c/4095/
• review common install/scenario config proposal (Tim will create one today – taking into account input from Jonas, Narinder, Weidong).
• Updates on feature integration guides for installers (i.e. a HOWTO integrate your feature into a specific installer) – See the section on https://wiki.opnfv.org/genesis

• This meeting slot will be used for discussion on release plan for build/deploy/test
• CI needs to know which configuration to run and which tests to execute
• Need common configuration files for …
  • Build config
  • Deploy config
  • POD config
  • Test config
• Question 0: what are the combinations of installers/controllers/other sw component versions (compatibility matrix)?
• Question 1: how many deploy/test environments are needed?
• Question 2: what community labs will volunteer to provide needed resources?

• Release D milestone - status for test projects (10 min)
• Dashboard strategy and plan (45 min)

• Digital Rebar demo (this will be recorded) (30 min)
• Summit discussions re. lab resources (allocation and open issues) (10 min)
• Yardstick update (10 min)
• Asia meeting slot (5 min)

• Preparing/organizing test related activities at OPNFV Summit next week (30 min)
• OpenStack Summit last week - what was of interest to the testing group? (10 min)
• Update on Functest (15 min)

• Test projects - announcements (5 min)
• MAAS PoC demo (30 min) - will be recorded
• Test lab requirements https://wiki.opnfv.org/pharos_rls_b_spec (10 min)
• Project requirements for lab resources https://wiki.opnfv.org/projectreqonpharos (10 min)

• Test projects - notices / sync-up (5 min)
• Test definitions https://wiki.opnfv.org/testing (10 min)
• OPNFV tester journey https://wiki.opnfv.org/_media/devtest_journey.pdf (10 min)
• Test project requirements for Pharos labs (30 min)
  • Types of labs
  • Collecting project requirements https://wiki.opnfv.org/projectreqonpharos
  • Pharos lab ready criteria (test perspective)
  • etc.

• Rally: presentation of the new features (20 min)
• Pharos: Test project support (20 min)
• Yardstick Docker discussion (15 min)
• Individual test projects status (time allowing)

• Test projects status (10 min)
• Demo & discussion Test Dashboard (15 min)
• Yardstick demo - virtual Traffic Classifier (30 min)

• Test projects status (10 min)
• Discussions/decisions on the test deliverable for R2 (20 min)
• Discussions/decisions on the dependencies towards features and other test projects (20 min)

• Bottlenecks status (5 min) - Hongbo
• Transformer status (5 min) - Michael
• ODL related testing activities (45 min, Morgan)

• Test projects status (only projects not represented last week) - 2 min per project

  1. Bottlenecks - Find system bottlenecks (Huawei) - Hongbo
  2. CPERF - Controller Performance Testing (Red hat) – Daniel
  3. Policytest - Network policy testing (CISCO) - Keith
  4. Affirm - validation criteria for use of OPNFV trademarks (Ericsson) – Chris
  Projects Not yet approved by TSC ...
  5. STORPERF - Storage Performance Benchmarking for NFVI (EMC) - Edgar
  6. Transformer - Carrier Grade Requirements (Ericsson) - Michael

• Logistics for Test projects ”operational” meeting (5 min)

• Milestone C guidelines (15 min)
  • What are the expectations from Test Projects
  • Test needs from projects and how to integrate (test specs were supposed to have been defined end of Sep)

• Testing dashboard - update and first demo (15 min)

• Test definitions for rls B (10 min)
  • What are we testing for each installer
  • Definition of failure / severity

• Test projects status - all test project leads - 20 min
• Yardstick - Demo of Virtual Traffic Classifier VNF (Ana, Ericsson) - 30 min
• Qtip update (Vikram, Dell) - 15 min

Test projects

Projects approved by TSC

1. Functest - Base system functionality testing (Orange) - Morgan
2. Yardstick - Infrastructure Verification (Ericsson) – Ana
3. VSPERF -Characterize vSwitch Performance for Telco (Intel) - Maryam
4. Qtip - Platform Performance Benchmarking (Dell) - Vikram
5. Bottlenecks - Find system bottlenecks (Huawei) - Hongbo
6. CPerf - Controller Performance Testing (upstream) – Daniel Farrell
7. Policytest - Network policy testing (CISCO) - Keith
8. Affirm - validation criteria for use of OPNFV trademarks (Ericsson) – Chris

Projects Not yet approved by TSC …

9. STORPERF - Storage Performance Benchmarking for NFVI (EMC) - Edgar
10. Transformer - Carrier Grade Requirements (Ericsson) - Michael

• Yardstick - Discussion on planned deliverables for Mobility Traffic Profile (Ana, Ericsson) - 20 min
• VSPERF - update and demo (Maryam, Intel) - 30 min
• Test dashboard (wiki) (Morgan, Orange) - 10 min

• Yardstick - Demo of new functionality (Ana, Ericsson) - 30 min
• Bottlenecks - https://wiki.opnfv.org/project_proposals/bottlenecks (Hongbo Tian, Huawei) - 15 min
• Testing dashboard (Ray, Andrew - Linux Foundation) - 10 min

• Show the community how MAAS can deploy Arno release of OPNFV (Narinder Gupta, Canonical)
• Proposed Storage Benchmark project (STORPERF) https://wiki.opnfv.org/storperf (Edgar StPierre, EMC)
• "Policy Test" project proposal discussion https://wiki.opnfv.org/policytest/policytest_project_proposal (Keith Burns)

Meeting canceled

Meeting canceled

Proposed Agenda (60 min):

Qtip 45min

• Testing strategy, planning for Brahmaputra (Nauman Ahad / Wenjing Chu)

Pharos 5 min

• Status update

Functest 5 min

• Status Update

Yardstick 5 min

• Status update

Proposed Agenda (60 min):

Pharos 20 min

• Release B deliverables
• LF support plan
• Near term needs
• Lab status and availability
• MAAS follow up

Functest 5 min

• Status and Update - Morgan is out this week

Yardstick 30 min

• B-release deliverables
• PODs to use for test case verification
• Definition of Done for a test case
• SLAs for OPNFV infrastructure
• Weekly meeting for follow up on Jira tasks

Other topics 5min

• Pharos meeting at ODL Summit

Proposed Agenda (60 min):

Pharos

• Planning (15 min)
  • Plans presented to TSC
  • Logistics for Pharos dedicated meetings
• MAAS demo and discussion - Narinder Gupta - narinder.gupta@canonical.com (30 min)

Functest

• Status and Update (5 min)

Yardstick

• Status and Update (5 min)

Other topics

• Standing agenda (5 min)

Proposed Agenda (60 min):

Functest

• Demo Functest

Common topics

• Discussion on test coverage (if time allows), see proposal in wiki: https://wiki.opnfv.org/testing

Proposed Agenda (60 min):

• Discuss test strategy for Rls 2

• See proposal for Test deliverables and strategy for release 2 … https://wiki.opnfv.org/_media/test_strategy_v0.2.pptx

Proposed Agenda (60 min):

Pharos

• No agenda topics

Functest

• No agenda topics

Yardstick

1. Demo of the tool
2. Discussion on use of the tool and how to serve wider needs of testing projects

* Due to Arno retrospective this week meeting is changed to 14h30 UCT (7:30 am Pacific Daylight Time) and will be 30 min long

* Yardstick demo is postponed to the following week (Thursday 18th June)

Proposed Agenda (30 min):

Pharos

• No agenda topics planned

Functest

• Test strategy for Release 2

Yardstick

1. Demo postponed to next week

PLEASE NOTE MEETING HAS BEEN CANCELLED AS A NUMBER OF ACTIVE PARTICIPANTS CANNOT ATTEND (YARDSTICK DEMO WILL BE RESCHEDULED FOR NEXT WEEK)

Proposed Agenda (1hr):

OVS characterization test framework

1. Overview of test tools
2. Demo
3. Status and plans

Functest

• Introduction to Functest
  1. Arno release status
  2. Project organization
  3. Git description
  4. Integration with CI
  5. Results analysis
  6. New test case integration

Yardstick

1. Status
2. Demo June 4th
3. Backlog

Proposed Agenda (1hr):

Pharos (15 min)

• Revised Pharos spec for Arno
• Lab definitions and tracking

Functest (15 min)

• Organization format (meeting, IRC,..)
• Beyond R1
  • new test cases
  • new tools
  • analytics on bench results
  • synchro with ETSI Test

Yardstick (30 min)

• Questions / discussion from project proposal
• Project scope & synergies with other test projects
• Project members
• Organization format (meeting, IRC,..)

Meeting this week will be 30 min to accommodate Arno readiness call at 7:30 PDT. We will likely only have time for Yardstick topics in the call.

Pharos

• No agenda items this week

Functest

• No agenda items this week

Yardstick (30 min)

• Project scope & synergies with other test projects
• Project members
• Organization format (meeting, IRC,..)

Functest

• Release 1
• Validation new committers
• Review organization format (meeting, IRC,..)
• Beyond R1
  • new test cases
  • new tools
    • synergies with yardstick
    • analytics on bench results
  • synchro with ETSI Test

Pharos

• Release 1 documents

Meeting canceled - prioritizing release activities and meetings

Meeting canceled - prioritize release candidate meeting

Meeting canceled - prioritize release candidate meeting

Meeting time UTC 14:00-15:00 / 7:00am-8:00am PDT Your local time

• Pharos (20 min)
  • Test-bed status
  • Pharos specification and work for release one

• Functest (30 min)
  • Voting for functest leader
  • Working session re. status and activities for release one

• Wiki and tool links
  • Pharos project https://wiki.opnfv.org/pharos
  • Pharos specification https://wiki.opnfv.org/pharos/pharos_specification
  • Functest https://wiki.opnfv.org/opnfv_functional_testing
  • Documentation project https://wiki.opnfv.org/documentation_projects/opnfv_documentation
  • Jira https://jira.opnfv.org/secure/Dashboard.jspa
  • OPNFV Release page: https://wiki.opnfv.org/releases
  • Arno Release Description: https://wiki.opnfv.org/releases/arno
  • Arno Release Candidate plan: https://wiki.opnfv.org/releases/arno/releasecandidates

Meeting time UTC 14:00-15:00 / 7:00am-8:00am PDT Your local time

• Functest (35 min)
  • status and plan for rls one
  • CI integration
  • Documentation
  • Update on vIMS proposal – Metaswitch

• Test Tool initiative (25 min) - Ericsson

• Wiki and tool links
  • Pharos specification https://wiki.opnfv.org/pharos/pharos_specification
  • Functest https://wiki.opnfv.org/opnfv_functional_testing
  • Documentation https://wiki.opnfv.org/documentation_projects/opnfv_documentation
  • Jira https://jira.opnfv.org/secure/Dashboard.jspa

Meeting time UTC 14:00-15:00 / 7:00am-8:00am PDT Your local time

• Functest
  • Progress with test tools and CI integration
  • vIMS proposal from Metaswitch
• Rls 1 deliverables
  • Pharos specification https://wiki.opnfv.org/pharos/pharos_specification
  • Documentation https://wiki.opnfv.org/documentation_projects/opnfv_documentation
  • Functest https://wiki.opnfv.org/opnfv_functional_testing
• Tools
  • Jira https://jira.opnfv.org/secure/Dashboard.jspa
  • Test tools - Tempest / Rally

Meeting time UTC 15:00-16:00 / 7:00am-9:00am PST Your local time

• Wiki updates - meetings https://wiki.opnfv.org/meetings/test
• Rls 1 deliverables
  • Pharos specification https://wiki.opnfv.org/pharos/pharos_specification
  • Documentation https://wiki.opnfv.org/documentation_projects/opnfv_documentation
  • Functest https://wiki.opnfv.org/opnfv_functional_testing
  • vIMS
• Tools
  • Jira https://jira.opnfv.org/secure/Dashboard.jspa
  • Test tools - Tempest / Rally

2015-12-03