Wiki

User Tools

Site Tools

Trace: • vsperf
meetings:vsperf

Table of Contents

Characterize vSwitch Performance for Telco NFV Use Cases: Team Meetings

Info

vSwitch Performance Characterization Meeting Minutes

Mar 9th 2016

* Agenda:

  • Overview considerations when bench-marking of Virtual Switches
  • Project Dev Status
  • Traffic Gen Updates
  • Rel C Planning

* ACTIONS

  • [ACTION] Maryam to add CI overview to agenda, Wednesday 16th March
  • [ACTION] Bill to create an updated installation / usage version of the tool as the one I presented several weeks ago is now obsolete.
  • [ACTION] Simona to check with Maryam if we have a best practice guide when upgrading to new version of VSPERF
  • [ACTION] All please provide input to Rel C planning etherpad or confirm acceptance (see link below to etherpad)
  • [ACTION] Maryam to provide an update next week on help that is being provided by Ixia

* Benchmarking Overview

  • Al gave an overview of Benchmarking for Virtual Switches
  • Also presented the slides at Test projects - weekly coordination meeting, main aim to have it adopted by the wider community

* Project Dev Updates

* Traffic Gen updates

  • Xena Update
    • Trying to run VSPERF on Redhat plus fix Xena bugs, testing of this starting today
  • Moognen Update
    • Re-write of LUA wrapper done by Andrew more detail on github
    • User guide is currently obsolete, plan to stabilize the new version of Moongen before creating the new user guide, should have something for next week.
    • Changes can be viewed at Andrews github : https://github.com/atheurer/MoonGen/tree/dpdk2.x-opnfv
  • Spirent Update
    • Challenge to update to new VSPERF code', Simona to check with Maryam if we have a best practice guide for upgrading to new version of VSPERF

* Release C updates

  • Indicated our intent to be part of REL C as planned on March 8th
  • Now need to focus on assessing our impacts to other projects such as installers, CI, Infrastructure
  • Please provide input directly to etherpad link: https://etherpad.opnfv.org/p/VSPERF_C_Planning
  • Meeting with IXIA for packet capture was productive, Plan is to move some TCL to python

Mar 2nd 2016

* Agenda:

  • VPP Overview
  • Project Dev Updates
  • Traffic Gen Updates
  • Test Priority List for Release C

* ACTIONS

  • [ACTION] Maryam to add CI overview to agenda in two weeks, Wednesday 16th March
  • [ACTION] Bill to create an updated installation / usage version of the tool as the one presented several weeks ago is now obsolete.
  • [ACTION] Simona to send details via email of top Priorities for Rel C - Complete
  • [ACTION] All to review and agree final REL C proposal and or provide feedback for any changes/updates

* Project Dev Updates

* Traffic Gen updates

  • Xena Update
    • working on XML file configuration
  • Moognen Update
    • Well done to Andrew and Bill, fantastic progress has been made in last few weeks.
    • Andrew Theurer has made significant changes to the tool. This is in regards to the Lua wrapper around the DPDK engine, in particular the file opnfv-vsperf.lua which instruments RFC 2544 like testing.
    • The changes include but are not limited to: Multi (concurrent) device support. As many pairs of ports as the user wants -Multi queue support for transmit -Hardware tx method is default.
    • All bugs with hw tx should now be avoided (with multi queue support) -Major clean up and reorganization of code -Test parameters are now in file -no command line arguments -Tx rate is calibrated before testing to never be above requested rate
    • -Uni/bidirectional logic handled in connections[] array -Use log: facility that comes with MoonGen -Too low tx rate detection allows N attempts instead of aborting on first failure
    • The tool is now ported from DPDK 1.71 to DPDK 2.2
    • Changes can be viewed at Andrews github : https://github.com/atheurer/MoonGen/tree/dpdk2.x-opnfv
  • Spirent Update
    • Undecided whether to push two scripts or merge into one, however there potentially could be overlap. Maryam suggested to push scripts as they are and a decision can be made as part of the review.

* Release C updates

Feb 24th 2016

* Agenda:

  • Review Action Items from Feb 17th
  • Project Dev Updates
  • Traffic Gen Updates
  • IETF-95 Update
  • Rel B Status
  • Test Priority List for Release C

* ACTIONS

  • [ACTION] ALL can you please review patches and abandon those that are no longer valid
  • [ACTION] All please notify Maryam or Al if you plan on attending hackfest
  • [ACTION] : Maryam to update Jira with new tickets for Rel C
  • [ACTION] : All Draft tests on etherpad please review and revert back to Maryam and Al with your feedback
  • [ACTION] : All please ensure your Jira tickets that have been completed are closed in JIRA
  • [ACTION] : Bill to advise on mailing list for Moongen

* Project Dev Updates

* Traffic Gen updates

  • Xena Update
    • working on XML file configuration
  • Moognen Update
    • Success in getting 40gb supported, moongen now upgraded to DPDK 2.2
    • Started working on integrating into VSPERF infrastructure
  • Spirent Update
    • Need to fix issues with older version and then update scripts

* ITEF-95 Update

  • Al updated Jira ticket deadline has been extended, Maryam to add some updates to the draft
  • look at possibility of doing remote presentation

* Release B updates

  • Official External launch announcement March 1st
  • Tag added to stable Brahmaputra branch for VSPERF

* Release C updates

  • Potential collaboration between OVS and VSPERF on new feature (ability to classify packets and give priority) submitted by AL.
  • Could use VSPERF as test suite to measure the benefit

* Please send on any agenda items you would like to discuss next week

Feb 17th 2016

* Agenda:

  • Review Action Items from Feb 10th
  • IETF-95 Update
  • Project Dev Updates
  • Hackfest March
  • Traffic Gen Updates
  • Rel B Update
  • Test Priority List for Release C

* ACTIONS

  • [ACTION] All can you let Maryam know if you plan on attending hackfest
  • [ACTION] Maryam to add review feedback on test results to the backlog for Rel C

* Project Dev Updates

* Traffic Gen updates

  • Xena Update
    • working on some bugs, having issues with results generation, continue to find issues while testing
  • Moognen Update
    • Fix implemented for 0% and 1% loss, difference of variance is now closer
  • Spirent Update
    • newtestcase.rst, according to the comments in done. I’ll relook at this document next week to enhance it to cover other cases (other attributes of testcase) apart from the traffic generator.
    • Made initial commit on JIRA-VSPERF:237. Will continue to fine tune the scripts.
    • Planning to add scripts for multicast benchmarking (3918). Should create Jira ticket for the same soon.

* Rel B updates

  • Primary release date on target for Feb 25th
  • Only outstanding item, publish dashboard results
  • Received some good feedback, need to add definition for tests
  • Need to review unstable results, B2B frame test need to define what we are plotting
  • Rename scalability OVS/DPDK tests

* Hackfest

* Test priority list for release C

* Please send on any agenda items you would like to discuss next week

Feb 10th 2016

* Agenda

  • Review Action Items from Feb 3rd
  • IETF-95 Update
  • Project Dev Updates
  • Hackfest March 14-15
  • Traffic Gen Updates
  • Rel B Update
  • Test Priority List for Release C

* ACTIONS

  • [ACTION] Maryam to investigate MPLS support in OVS.
  • [ACTION] Al to open a JIRA ticket to Update the Internet draft, etc.
  • {ACTION} Al to combine the VM2VM-related tests, then JIRA ticket Priority 1 for Rel C [DONE]

* IETF-95 Update

  • Need to sync Internet Draft with the list of tests as of Brahmaputra Release

* Project Dev Updates

* Traffic Gen updates

  • Moongen Update
    • made 3 bug fixes to the tool, found missing packets causing tests to fail
  • Spirent update
    • Sridhar not on the call, no update
  • Xena Update:
    • Amit finishing up final integration to meet traffic gen spec
  • Ixia (RFC 5481)Update:
    • made progress on SW upgrade to support packet capture on the Pod in Shannon
    • some progress on PDV calculations for results report.

* Rel B updates

  • Primary release date pushed to Feb 26th
  • Any project that is ready will go out on Feb 26th, projects that are not ready will go out March 28th
  • Documentation: still open question about the new templates proposed by Chris, room for some flexibility perhaps we can continue to use what we have for this release, as far as we know.

* Hackfest

  • Will be held in Santa Clara March 14-15th
  • Both Maryam and Al attending, would like to know who else is attending so we can plan demos and breakout sessions
  • Registration is free, link https://www.regonline.com/Register/Checkin.aspx?EventID=1802426
  • It’s a good opportunity to get input from folks that don’t attend project meetings

* Test priority list for release C

  • Al added Jira tickets for new tests
  • When you combine the VM2VM-related tests, then Priority 1, need wiki update and JIRA ticket
  • Voting for Rel C tests are still welcome
  • If folks are happy with current top 4, please send a mail to Al confirming
  • link to Test Review Scratchpad etherpad: https://etherpad.opnfv.org/p/vSwitchTestsDrafts, welcome test contributions for match action and VM2VM.
  • New Draft Test at the Scratchpad (etherpad):
    • Overlay Network performance described in general for VXLAN, GRE, GENEVE
    • feedback that this approach works, saves time in review.
  • link to VM2VM Presentation https://wiki.opnfv.org/wiki/vswitch_readouts?&#vsperf_project_readouts

Feb 3rd 2016

* Agenda:

  • Traffic Gen update.
  • Dev update.
  • Test Priority discussion.
  • VSPERF results.
  • VM2VM discussion

* ACTIONS

  • [ACTION] Maryam to create wiki for tests.
  • [ACTION] Maryam to investigate MPLS support in OVS.

* Test Generator Update

  • Xena: Proving out implemented methods and working on/around Xena incompatibility with Python 3.
  • Spirent: Implemented changes to allow TestCenter to execute using the REST APIs, removing the need for a local client.

* General dev updates

  • Final Report or release B is being worked on.
  • Reporting % Frameloss from the traffic gen - support in progress - PLEASE NOTE (Spirent/Xena integrators) watch out for this patch.

* Test Priority discussion

* VSPERF results for Brahmaputra

* VM2VM discussion

Jan 27th 2016

* Agenda:

  • Moongen Overview
  • Project Dev Updates
  • Traffic Fen Updates
  • Test Priority List for Release C
  • Review Open Actions

* ACTIONS

  • [ACTION] Maryam to post test results report
  • [ACTION] Maryam to post list of integration tests/tests from OVS developers to ether pad
  • [ACTION] Sridhar add a "how to add a new test case" document
  • [ACTION] All Action all to provide feedback to Al on Top 6 tests for Rel C, deadline pushed out to Feb 6th https://etherpad.opnfv.org/p/vSwitchPerfNewTestList

* Moongen Overview

  • Notes taken during the presentation
  • Red Hat have integrated Moongen with an internal test framework. but mostly Bill's repo mirrors Moongen @ github, A detailed README exists on github
  • Additional setup information can be found @ https://raw.githubusercontent.com/bmichalo/MoonGen/opnfv/MoonGenSetUp.html
  • github repo can be found @ https://github.com/bmichalo/MoonGen/tree/opnfv, the relevant branch for vsperf from Bill's repo is the opnfv branch
  • Today Moongen uses DPDK 1.7.1, it would be a "nice to have" to upgrade to DPDK 2.2 but this is up to the Moongen maintainers.
  • Bill can't run Moongen with DPDK 2.1 or DPDK 2.2, but this isn't an issue for us, we should just use Moongen as a standalone traffic generator
  • Examples/opnfv-vsperf.lua is a script that Andrew and Bill wrote modelled on other scripts in the example dir and it runs an RFC2544 test
  • The script does a binary search, to find the throughput value for the specified traffic loss %
  • Bill added a validation run to the script to ensure that the throughput value found in the binary search is indeed the x% frameloss throughput value
  • If validation fails the script continues to run the binary search until the validation phase passes

* Project dev updates

* Traffic Gen updates

  • Spirent update
    • Sridhar should be pushing some patches to vsperf will use the spirent REST API.
    • Sridhar to follow up on Jira access with Aric
    • Sridhar has created a document for adding a new test case
    • Sridhar to migrate the ppt to rst and push to gerrit
  • Xena Update:
    • Xena code is written - it just needs to be tested
    • Christian is going to test it - working on installing vsperf

* Test priority list for release C

* OPNFV Summit

Jan 20th 2016

* Agenda:

  • Moongen Overview
  • Project Dev Updates
  • Traffic Gen Updates
  • Test Priority List for Release C

* ACTIONS

  • [ACTION] Bill Setup separate BlueJeans call for Moongen overview
  • [ACTION] Maryam to post traffic gen guide to Gerrit as a draft
  • [ACTION] Maryam to post list of integration tests from OVS developers to ether pad
  • [ACTION] Maryam and Sridhar add a "how to add a new test case" document
  • [ACTION] All to provide feedback to Al on Top 6 tests to start defining priorities for Release C

* Moongen Overview

  • Moved out to next week, due to presentation issue
  • Bill action to setup separate bluejeans session next week

* Project dev updates

* Traffic Gen updates

  • Spirent update
    • Brian has reviewed the documentation and provided feedback through email
    • Sridhar working on enabling b2b and other RFC2544 tests
    • Sridhar working on RFC3918 – implementation patches will be posted soon
    • Action: action add a "how to add a new test case" document – Sridhar and Maryam
  • Xena Update:
    • Action: Maryam to post traffic gen guide to gerrit as a draft
    • Action: Maryam To post list of integration tests/tests from OVS developers to ether pad
    • Xena developers working on burst and cont traffic methods… there are some missing return parameters that Xena needs to enable access to

* Test priority list for release C

  • Action all to provide feedback to Al on Top 6 tests to start defining priorities for Release C

* OPNFV Summit

Jan 13th 2016

* Agenda:

  • Readout From Ralph - is your SDN switch ready for the real world ?
  • CI
  • Test Prioritisation
  • Moongen Integration with VSPERF
  • Traffic Gen

*ACTIONS

  • [ACTION] Simona include Al’s bugs in next Sprint
  • [ACTION] All Review Test Prioritisation list and pick top 6 and email to Al

* Project dev updates

*Milestone E

* Test Prioritisation

* Test Generator Integration

  • Moongen integration with vsperf
    • Bill will go through installation tutorial of Moongen at next week’s meeting
      • User guide it 2/3rds complete, available for folks can try it out
  • Sridhar on Spirent Integration: Sridhar has written 2544 script, more coming
  • RFC 2544 b2b test implemented and under review
  • Traffic gen Xena: update from Rick good progress integrating RFC2544 test, next looking at Latency, jitter and frameloss testing, pick-up next Monday again

* OPNFV Summit

-

Jan 6th 2016

* Agenda:

  • Nightly Build
  • Milestone E
  • Test Generator Integration

* ACTIONS

  • [ACTION] Al & Brian have volunteered to review user Guides, and other docs (design docs and Release docs, ignore LTD Spec)
  • [ACTION] Pierre to help with upgrading firmware (RFC 5481)
  • [ACTION] Maryam to provide link to list of tests for next release
  • [ACTION] AL to prioritise tests by working with community and work test spec development

* Project dev updates

* Milestone E

  • Successfully reached and reported to Debra, link https://wiki.opnfv.org/releases/brahmaputra/release_plan/vsperf_milestone_e_report
    • Any Questions on the report please send them to the mailing list
  • Creation of Stable Release B Branch deadline tomorrow 7 Jan
  • New defined way of working on Stable branch - link https://wiki.opnfv.org/releases/stablebranch
  • Again, any questions please send to mailing list on Stable Branch work methods
  • Next Milestone, Jan 12, no changes allowed after this date
  • Final Milestone - Jan 19 - All Documentation must be available for Review, link http://artifacts.opnfv.org/vswitchperf/docs/all/index.html
  • Need volunteers to review user Guides, and other docs (design docs and Release docs, ignore LTD Spec)- Al and Brian have agreed to complete this task see AR above
  • Review means read through and consistency checks, pick up JIRA tickets
  • Jan 26, All Docs complete, All tests will be conducted on the various Pods on stable Releases, till Feb 1.

* Test Generator Integration

  • New Epic on Moongen with Bill, he will come up with Documentation for Using MoonGen
  • New work on Dummy Traffic Gen will help with MoonGen and Xena
  • Documentation on how to integrate a traffic gen with vsperf will help with MoonGen and Xena integration.
  • Amit on Xena: implemented part of vsperf traffic gen, need to add connect disconnect
  • Sridhar on Spirent Integration: RFC 2544 and RFC3849 scripts to support testing with Spirent
  • Test Prioritization and Definition – Maryam to provide link to existing link<MT> Link https://etherpad.opnfv.org/p/vSwitchPerfNewTestList
  • RFC 5481 scripts
    • risk in disturbing Pod 3, Firmware installed locally, Pierre has agreed to help with upgrade of the firmware
    • Details on Percentile calculation for RFC 5481 nailed down - one preferred method in RFC 2330 will be used

* OPNFV Summit

Minutes also available at https://wiki.opnfv.org/meetings/vsperf

Dec 2nd 2015

* Agenda:

  • Project dev updates
  • Traffic Gen updates
  • CI update
  • Inactive committers
  • OPNFV summit
  • PDV to be collected in RFC5481 format and IXIA

* ACTIONS

  • [ACTION] Add vloop_vnf to documentation as the reference image to use with vsperf.
  • [ACTION] Maryam to establish an initial roadmap including this capability as a feature.
  • [ACTION] Maryam to establish an initial test priority list to review with the community.
  • [ACTION] Maryam to include path finding and integration activity for a reference pipeline to the roadmap.
  • [ACTION] Gene to update vloop_vnf https://jira.opnfv.org/browse/VSPERF-133 and push new image to artifacts.opnfv.org
  • [ACTION] Gene to migrate our commit gate checks and establish a basic Jenkins job that runs nightly on POD3.

* Project dev updates

  • OPNFV test dashboard integration is well underway
  • vloop_vnf integration with vsperf complete – need some updates to the vloop_vnf image and need to push the updated image to artifacts.opnfv.org

* Traffic Gen updates

  • Xena: Amit mentioned that they’ve had couple of discussions about how to use existing xena code in vsperf. Amit has written small python scripts to perform individual tasks, Now the next step would be to use these small scripts in vsperf template. Amit is having a working session with Dan tomorrow that will hopefully result in having Xena take over the integration effort.
  • Moongen:
    • Looking for a volunteer to help Andrew with this.
    • [ACTION] Maryam and Billy to help facilitate the integration work.

* CI update

  • [ACTION] Gene to send on update
  • CI Nightly Build Steps:
    • Clone, install and build vsperf dependencies.
    • check if IxNet server is running and calls basic phy2phy_tput testcase
    • Run all test cases that are registered with the OPNFV test Dashboard and push the results to the dashboard.

* Inactive committers

  • I’ve kicked off the process to remove inactive committers, so we can promote contributors to committer status.

* OPNFV Summit

* PDV to be collected in RFC5481 format and IXIA

  • Al has looked at them.
  • tcl script and a python file. it looks like the tcl script needs to be dropped into the right place within vsperf and tying in the python script into vsperf.
  • Maryam will look at integrating these scripts in the next sprint

Nov 18th 2015

Agenda

  • Rel B requirements for installers
  • Project Updates - Xena, Spirent, Moongen
  • DUT Topology discussion.

DUT Topology discussion

  • Agreed at OPNFV conference to remain with current naming.

Rel B requirements

  • Compass is a possibility for installer for rel-C.
  • For Rel-B it is possible to have a simple code install on a "clean base" host - like a git clone or untar of the vsperf source code.

Dev updates

  • Can now use Spirent TestCentre traffic generator for RFC2544.
  • More tests/scripts to follow
  • [ACTION] Brian Castelli will add Jira tickets for these tasks
  • Still trying to get lic servers running on POD3

vnf_loop vnf

  • work on integrating this in progress

moongen

  • unlikely to be in a position to write vsperf integration code for Rel-B (unless help given)
  • RFC2544 with %loss is working in moongen itself.
  • no cost for licencsing or distribution of same (MIT lic.?)
  • [ACTION] Andrew Th. to verify licenseing requirements
  • moongen is distributed in src. May need to pkgd in order to deploy on distro.

Xena

  • hoping for eng resource to be allocated shortly.
  • [ACTION] billy put on agenda for 2wks from now for Ralph Daniels (Sprient) to present on slide deck (on vsperf ML) on OpenFlow vSwitch testing.
  • The slide deck for Ralph's presentation was sent to the mailing list on 2-Nov

AOB

  • We should try some push-back on Rel-B installer requirements and target Rel-C as originally committed.

Nov 11th 2015

  • No meeting due to OPNFV conference.

Nov 4th 2015

  • [ACTION] for ALL: Please review VSPERF feature wish list email and send on any features that we should include for next release.

Note no meeting November 11th

AGENDA for Next Meeting:

  • 20 mins DUT Topology discussion.
  • 20 mins OPNFV Summit update
  • 10 mins project updates.
  • Discussion around unified/common denominator API for Traffic Gens
    • Q: In VSPERF, what is the purpose of the common template(Traffic Item + API), is it to try to get the same results across the different traffic gens or something else?
      • A: The intention was to hide the implementation details and complexity in terms of parameters passed to the traffic gen behind a simple object that could describe the basic parameters (such as packet size, bi/uni directional traffic…) that we want to configure for a test. Today we pass a set of parameters that we want to modify to the generic traffic gen API. They get translated into the appropriate parameter by the underlying traffic gen implementation and all other parameters are set to their defaults.
    • We agreed that we will document the implicit settings of Trafficgens supported by VSPERF in a separate document under docs.
    • We need to standardize the set of input and output to and from traffic gens – Added this to the VSPERF design summit and will bring a proposal to the weekly meeting after the summit.
  • Traffic Gen integration update
    • Xena: Rick has been using the RFC2544 Linux executable. Rick can run the suite itself and run it in an automated env. Targeting the next (non-release B) VSPERF open feature development window for integration.
    • Spirent: Looking for another resource to continue the integration work.
    • Moongen: [ACTION] Andrew to send on update
  • CI Integration
    • [ACTION] Gene to provide the update
  • PDV to be collected in RFC5481 format and IXIA:
    • We received the initial scripts and will spend some time looking at them.
  • Deployment naming discussion
    • To be covered at the design summit.
  • VSPERF breakout session
  • VSPERF feature wish list – REMINDER LOCK DOWN 1st of Dec 2015 is there anything we need today for release B.

Oct 28th 2015

  • AGENDA for November 4th:
  • 20 mins DUT Topology discussion.
  • 20 mins Discussion around unified/common denominator API for Traffic Gens.
  • 10 mins project updates.
  • 10 mins Deployment naming discussion.
  • new tests discussion continuation – next session.
  • Traffic Gen integration update
    • Xena: Rick is working with the ported RFC2544 implementation.
    • Spirent: First set of patches upstreamed to vsperf. Lays the ground work for future tests. Throughput tests is what has been implemented thus far. Brian is also looking at a new solution that makes the implementation even simpler than what it is today.
    • Moongen: Bidi and latency testing have been enabled but not pushed yet.
  • CI Integration
    • [ACTION] Gene to provide the update
  • Documentation integration with RELENG
  • IETF-94 and vsperf bmwg draft
    • Presented an update of our draft at IETF 94 last night – it was well received – separate update email to follow shortly.
  • PDV to be collected in RFC5481 format and IXIA:
    • We received the initial scripts and will spend some time looking at them – could be a good topic for the Group breakout session at the OPNFV design summit.
  • New tests
    • Discussed some possibilities:
      • Scalability Tests adding More VMs in succession and building a profile
      • East-west across the same node [intra node]
      • Overlay Networking Tests: VXLAN/GRE performance testing, encap, decap, encap and decap?
      • (ACLs/Firewall config)Match action performance testing? The cost of the different actions supported by a vSwitch?
    • Some More possibilities to be discussed:
      • Classifying L2, L3 and L4 traffic Profile Tests.
      • Stream/bulk Data transfer "unidirectional stream" performance
      • request & response/transaction rate tests?
      • Performance testing with Mirroring enabled on the switch?
      • TCP Max connections per second, Max # of active sessions, Max transactions per second
      • IPv6??
      • Best of N and Worst of N Tests from Maciek
      • Deactivation tests?
      • vSwitch in QOS mode/load balancer
  • Deployment naming discussion
    • Should we stick with:
      • P2P, PVP, PVVP …
    • Or should we migrate to:
      • T1n – VSWITCH Onboard port Forwarding ( either P2P port forwarding or flow based )
      • T1u – VSWITCH Forwarding with user space L2 forwarding
      • T1v – VSWITCH Forwarding with one VM ( either switch based port forwarding or flow based )
      • T2v – VSwitch Forwarding with two VMs ( either switch based port forwarding or flow based with DNAT on VM1)
  • VSPERF feature wishlist (LOCK DOWN 1st of Dec 2015 is there anything we need from this today?)
    • VLOOP_VNF integration with VSPERF.
    • Single port traffic gen support (today we just use 2 ports)
    • Bypass/SR-IOV setup support.
    • VXLAN/overlay tests (when we have them defined).
    • VM 2 VM tests.
    • L2FWD integration with the VNF as an option instead of test-pmd or Linux Bridge.
    • test-pmd integration as an option for Vanilla OVS VNFs instead of Linux Bridge.
    • a vSwitch/VNF None option to allow us to bypass their configuration and leverage our traffic gen implementations (for Yardstick).
    • IMIX specification as an option to the traffic generator using https://tools.ietf.org/html/draft-morton-bmwg-imix-genome-02
    • pcap replay as an option for traffic gens.
    • Detailed benchmark reporting and graphing.
    • RFC5481 Latency measurements
    • RFC2889 tests implementation

Oct 14th 2015

  • Readouts
    • Morgan Richomme opnfv-test-dashboard
    • Cisco Readout on learnings when characterizing the performance of vSwitches next week. (14th October 2015)
  • Traffic Gen integration update
    • Xena: Sync up with Dan regarding integration work complete. [ACTION] Billy and Dan to send out list of open questions.
    • Spirent: Brian continuing work on integration.
    • Moongen (Andrew T): [ACTION] Andrew to send status.
  • CI Integration
    • [ACTION] Gene to send out more info
  • Documentation integration with RELENG
    • [ACTION] Ram to address Aric’s comments in the review. https://gerrit.opnfv.org/gerrit/#/c/2533/
    • Really good progress made on this front, Ram added all the relevant rst index files and restructuring of the documentation. A big thanks to Ram for looking after this.
  • IETF-94 and vsperf bmwg draft
    • Updated draft submitted to gerrit, it has also been reviewed and pushed. A big thanks to Al for looking after this.

TOPICs not discussed this week.

AGENDA for October 21st:

  • 20mins project updates.
  • 20mins new tests discussion.
  • 10mins Deployment naming discussion
  • 10mins CI discussion.

Oct 7th 2015

  • JIRA and TOIT/VSPERF development update
  • Features under development
    • PVP and PVVP enablement for Vanilla OVS – in Review
    • PVVP for vhost cuse and user – merged to master
  • Traffic Gen integration update
  • Xena: Initial python development looking at the integration with VSPERF has been kicked off. But there seem to be missing params that could affect the results of a test. [ACTION] Billy and Rick to setup a session to discuss issues & questions.
  • Spirent: Brian has been integrating Spirent with VSPERF and has traffic passing through the DUT. Brian gave a demo of the integration efforts so far. Thanks and great progress Brian!
  • Moongen: MoonGen update (Andrew T): Bidirectional traffic is working, latency measurement has been added, however, latency stats for bidirectional traffic is incorrect (seeing negative numbers). There is also now support to specify the frame size for the test. Changes haven’t been committed to github yet, but be will soon. Also, you can now specify a %frame loss as a pass/fail threshold (default is 0).
  • CI Integration
    • Gene is working on installing the Jenkins Client on the 2nd server attached to the VSPERF DUT in POD3.
  • Documentation integration with RELENG
    • [ACTION] Ram to send on what support/help is needed here to progress this issue/story.
  • IETF-94 and vsperf bmwg draft
    • Al is working through the LTD issues in JIRA to start looking at this.
    • New tests and spec developments should be added to the IETF Draft.
    • A demo would be very well received at the BMWG session.
  • PDV to be collected in RFC5481 format and IXIA:
    • Sync up with IXIA planned.
  • OVS-NFV collaboration
    • Need to investigate possible collaboration points in terms of testing the vswitch.
  • YARDSTICK and VSPERF joint design session
    • Billy installed Yardstick and OpenStack and has been studying the Yardstick framework (it is not using/an extension of Rally).
    • Area for collaboration seems to be to leverage vsperf’s traffic gen integrations for yardstick.
  • Readouts
    • Cisco Readout on learnings when characterizing the performance of vSwitches next week. (14th October 2015)
  • Packet Re-ordering
    • Spirent is compliant with RFC5481
    • Need to check if IXIA is compliant with RFC5481 also.
    • We should enable this as a configuration option for VSPERF. [ACTION] Maryam to add JIRA issue for this (won’t be addressed till post release B).

* New tests

  • 20 mins scheduled for October 21st vsperf weekly sync up.

Sept 23rd 2015

  • JIRA MILESTONE C 25th Sept 2015.
    • [ACTION] Gene to add any CI stories - In progress. Waiting on POD3 access to install jenkins.
  • [ACTION] Andrew, Brian and Rick to add any traffic gen integration stories they would like to see as part of release B.
    • Brian – DONE.
    • Andrew - Will do.
  • [ACTION] Maryam to update wiki with additional goals for Release B. - In progress
  • [ACTION] Billy supply vsperf status update to RelB planning.
  • TOIT/VSPERF development update
  • Features under development
    • PVP enablement for Vanilla OVS – in progress
    • Sysmetrics patch has been uploaded to gerrit.
    • Frame modification tests implementation – patch up for review for almost 2 weeks, Maryam will push under 1-week rule when she returns.
  • Traffic Gen integration update
    • Xena:
      • Some training complete. Talking to Xena on Monday. Update next week.
  • Spirent:
    • Brian C, RFC2544 running to completion. Needs some work to gather results.
    • [ACTION] Brian C, will investigate possibility of setting up Spirent Traffic Gen VM was on POD3.
  • Moongen:
    • bidirectional functionality enablement in progress
    • Via Bill M, Andrew still working on bidi functionality. Andrew will update next week.
  • CI Integration
    • Proposed triggers:
      • 1. Run pylint on the code.
      • 2. Run a short VSPERF sanity check.
      • Require jenkins to +1 before commit allowed.
    • [ACTION] Maryam/Billy Input from Maryam/Billy on 2. required.
    • [ACTION] Maryam/Billy Input from Maryam/Billy on 1. required.
  • Documentation integration with RELENG
    • Patch posted to gerrit from Aric and Fatih, need to review and add TODOs to JIRA.
    • Gene thinks further action on this is blocked on pylint and vsperf sanity check above.
    • [ACTION] Gene to find out from RELENG is .jjb files should be in vsperf or releng repo.
  • PDV to be collected in RFC5481 format and IXIA:
    • Do we need to define a loss threshold?
    • ACTION: Al M. Add to LTD Test Definition for a test such as 'Re-ordering at the throughput level. Simple, Pass/Fail test no-reordering or there was some re-ordering'.
    • Via Al M, Expecting script soon from Ixia to measure PDV.
    • Via Gurpreet. Spirent has a re-ordering detection/measurement built-in to other tests such as it's RFC2544 Throughput.
    • ACTION Gurpreet will check if this feature is compliant to a specific RFC.
    • ACTION Billy add re-ordering to next weeks agenda.
  • POD 3 update
    • [ACTION] Brian C to verify VPN access via second server which has been hooked up back to back with the server in POD 3 in preparation for Moongen installation.
  • VSPERF Release B participation
    • Profile test and RFC2544 reset test will need to be pushed to stretch goals. [ACTION] Wiki to be updated
  • OVS-NFV collaboration
    • Need to investigate possible collaboration points in terms of testing the vswitch.
    • Would be good to ensure that dpdk-enabled ovs deployed in rel-b is testable with vsperf.
  • YARDSTICK and VSPERF joint design session
    • [ACTION] Billy and Maryam to ramp on Yardsticks framework and analyse the opportunity for collaboration.
    • Meeting provisionally set for the 30th of Sept. – More details to follow.

Sept 15th 2015

  • [ACTION] Maryam to follow up with Trevor about setting up a server where Moongen can run connected back to back to the DUT – Done a board has been setup back to back with our server in POD3
  • [ACTION] Maryam to post wiki instructions on log in details to the new server.
  • [ACTION] Billy - TOIT High Level Design wiki addition to https://wiki.opnfv.org/vsperf/test_methodology - In progress
  • [ACTION] Al - Test Spec Overview wiki addition to https://wiki.opnfv.org/wiki/ltd_test_spec_overview - In progress, only new tests left to go in.
  • [ACTION] Maryam to invite Morgan to give an overview of the testing dashboard at the vsperf weekly.
  • JIRA MILESTONE C 25th Sept 2015.
  • [ACTION] Maryam and Billy to add any generic/feature stories. - In progress
  • [ACTION] Gene to add any CI stories
  • [ACTION] Al to add any spec update stories that we think we need to lock down for release B - In progress
  • [ACTION] Andrew, Brian and Rick to add any traffic gen integration stories they would like to see as part of release B.
    • Brian – DONE.
    • Maryam to send out JIRA instructions - Done sent out by Al, Thanks Al.
    • [ACTION] Folks make sure you have permissions to create and progress a ticket, if there are any issues, ping me and the linux helpdesk.
  • [ACTION] Maryam to update wiki with additional goals for Release B. - In progress
  • TOIT/VSPERF development update
  • Features under development
    • PVP enablement for Vanilla OVS – in progress
    • update sysmetrics implementation – in progress – patches should be up for review soon.
    • Frame modification tests implementation – patch up for review.
  • Traffic Gen integration update
    • Xena:
      • Amit is in the process of familiarizing himself with the Xena RFC2544 program and Rick and Amit are planning a follow up training session with Dan from Xena to review the new Linux-based Xena2544 program.
    • Spirent:
      • Created the Jira ticket for Spirent integration.
      • Ported setup to the Spirent VCT POD.
    • Moongen: bidirectional functionality enablement in progress

* CI Integration

  • CI Plan – Gene updated the wiki.
  • In terms of what kind of test we want to run as part of our commit gate, it looks like some verification in terms of pylint/ ReSTructured text validation as well as one of the simple performance tests. [ACTION] update JIRA with the tests we’d like to start running as part of our CI.
  • We need to figure out the triggers.
  • Need to run pylint on the codebase as is - [ACTION] Billy to submit JIRA ticket for this.
  • Documentation integration with RELENG
    • Patch posted to gerrit from Aric and Fatih, need to review and add TODOs to JIRA.
    • [RECOGNITION] Thanks Aric and Fatih for helping us kick start this.
  • PDV to be collected in RFC5481 format and IXIA:
    • Do we need to define a loss threshold? We do need to define a threshold.
    • Do we need to look at or take re-ordering into account? I don’t assume so because we aren’t calculating IPDV, we are calculating PDV? In the case of PDV this is correct. We should take this into account when
  • POD 3 update
    • Second server has been hooked up back to back with the server in POD 3 in preparation for Moongen installation.
    • [RECOGNITION] Thanks Michael and Trevor for completing this setup.
    • Will need some support to get moongen up and running on the second server.
    • Traffc gen VNF will also go here.
    • [ACTION] Maryam to update POD wiki with new server login info.
  • VSPERF Release B participation
    • Profile test and RFC2544 reset test will need to be pushed to stretch goals. [ACTION] Wiki to be updated
  • OVS-NFV collaboration
    • Need to investigate possible collaboration points in terms of testing the vswitch.
  • New tests
    • TODO – next meeting
  • YARDSTICK and VSPERF joint design session
    • [ACTION] Billy and Maryam to ramp on Yardsticks framework and analyse the opportunity for collaboration.
    • Meeting provisionally set for the 30th of Sept. – More details to follow.

FYI Folks I’m out of office next week, as such Billy will be leading the meeting next week.

Sept 9th 2015

  • [ACTION] Maryam to follow up with Aric about JIRA emails/updated to committers and contributors Status: In Progress, Pinged the helpdesk. Issue of progressing an issue has been resolved for Gene, Al to try to progress a story before we close.
  • [ACTION] Al to updated the LTD to discuss sampling and details of PDV – Done.
  • [ACTION] Maryam to follow up with Trevor about setting up a server where moongen can run connected back to back to the DUT – In Progress We can setup a board back to back with the DUT, will be kicked off in the next couple of days.
  • [ACTION] Billy - TOIT High Level Design wiki addition to https://wiki.opnfv.org/vsperf/test_methodology - In progress
  • [ACTION] Al - Test Spec Overview wiki addition to https://wiki.opnfv.org/vsperf/test_methodology - In progress
  • JIRA
    • [ACTION] Maryam and Billy to add any generic/feature stories.
    • [ACTION] Gene to add any CI stories
    • [ACTION] Al to add any spec update stories that we think we need to lock down for release B.
    • [ACTION] Andrew, Brian and Rick to add any traffic gen integration stories they would like to see as part of release B.
    • [ACTION] Maryam to update wiki with additional goals for Release B.
  • TOIT/VSPERF development update
    • Features under development
      • PVP enablement for Vanilla OVS – in progress
      • update sysmetrics implementation – in progress – patches should be up for review soon.
      • Frame modification tests implementation – in progress – patches should be up for review soon.
  • Traffic Gen integration update
    • Xena: Migration of RFC2544 exe to Linux executable has been completed.
    • Spirent:
      • Ramp on building and configuration complete.
      • Brian is going to migrate his work to the Spirent POD in Sunnyvale.
      • Brian also set the Goal of doing a demo next week (No Pressure :P).
    • Any traffic gen integration support should be directed to the tech-discuss mailing list with the VSPERF tag.
    • Gene suggested setting up a traffic gen working group for folks who are trying to integrate a traffic generator with VSPERF. If people are interested in participating please reply to the minutes [ACTION] Gene to send out some more info.
  • CI Integration
    • There was a missing link for the PVVP deployment scenario, L2FWD requires DNAT - → DONE – merged to master
    • CI Plan – Gene will publish soon.
  • Documentation integration with RELENG
    • On hold, as a new tool chain is being deployed for OPNFV projects.
    • We should consider adding pre-commit gates that run some ReSTructured text validation whenever the rst files are modified – [ACTION] Talk to Fatih about this
  • PDV to be collected in RFC5481 format and IXIA:
    • Good analysis put together by Billy to advise Vendors on this, additional comments provided by Al. http://lists.opnfv.org/pipermail/opnfv-tech-discuss/2015-September/004900.html
    • L2FWD module also collects stats, so we can monitor what’s going on in the VM.
    • Looking at the packet captures is a good first step towards building the right algorithms to capture PDV. The Algorithms should be built to do PDV calculations on the fly and present the results in histograms.
  • VSPERF Release B participation
    • [ACTION] Maryam to add a summary feature table to the wiki. – DONE – Added goals, time plan.
    • JIRA MUST be updated with all our highlevel stories for release B, by COB Friday.
  • New tests
    • TODO – next meeting
  • POD 3 update

* We should be able to hook up a server back to back with our DUT so that we can setup Moongen for the POD.

Sept 2nd 2015

  • [ACTION] Maryam to follow up with Aric about JIRA emails/updated to committers and contributors Status: Open, Pinged Aric, waiting for response. Will ping him again and CC Gene and Al
  • [ACTION] Al to updated the LTD to discuss sampling and details of PDV – In progress.
  • [ACTION] Maryam to follow up with Trevor about setting up a server where moongen can run connected back to back to the DUT - Open
  • [ACTION] Billy - TOIT High Level Design wiki addition to https://wiki.opnfv.org/vsperf/test_methodology - In progress
  • [ACTION] Al - Test Spec Overview wiki addition to https://wiki.opnfv.org/vsperf/test_methodology - In progress
  • [ACTION] Volunteer for Test Projects status update tomorrow – to give status update for vsperf. – Al Volunteered, Thank you Al.
  • TOIT/VSPERF development update
    • Features under development
      • Affinitization for Vanilla OVS – best without affinitization
      • PVP enablement for Vanilla OVS – in progress
      • update sysmetrics implementation – on hold
      • Frame modification tests implementation – in progress
  • Traffic Gen integration update
  • Xena: RFC test was successfully deployed under Linux. [ACTION] Dan to give a demo next week.
  • Moongen:
  • Andrew tested 40Gb adapter and it initializes successfully but sw rate control for the 40Gb NIC still needs to be implemented for the adapter. Moongen typically uses HW/SW rate control.
  • Andrew experimented with the latency measurements but this has not been integrated with the RFC2544 script yet.
  • bidirectional traffic generation is also a feature that needs to be looked at.
  • Spirent: environment has been setup and ramp up on the scripts is in progress.
  • Gene suggested setting up a traffic gen working group for folks who are trying to integrate a traffic generator with VSPERF. If people are interested in participating please reply to the minutes [ACTION] Gene to send out some more info.
  • CI Integration
    • There was a missing link for the PVVP deployment scenario, L2FWD requires DNAT - → JIRA ticket added for this, code to be submitted soon.
    • The CI plan to be published soon and some commits not long after.
  • New tests
    • We need to start addressing some of the testing gaps in our test specification and look at new deployment scenarios and any other test categories that we need to develop tests for, either from a standard or from a relevant use case.
    • I implore the VSPERF community to start adding tests to the test_spec and pushing to gerrit for review. [ACTION] VSPERF Community. Tests can be reviewed or discussed at the weekly call.
    • This is our current list of tests:
      • Throughput tests
        1. Test ID: LTD.Throughput.RFC2544.PacketLossRatio
        2. Test ID: LTD.Throughput.RFC2544.PacketLossRatioFrameModification
        3. Test ID: LTD.Throughput.RFC2544.Profile
        4. Test ID: LTD.Throughput.RFC2544.SystemRecoveryTime
        5. Test ID: LTD.Throughput.RFC2544.BackToBackFrames
        6. Test ID: LTD.Throughput.RFC2889.Soak
        7. Test ID: LTD.Throughput.RFC2889.SoakFrameModification
        8. Test ID: LTD.Throughput.RFC6201.ResetTime
        9. Test ID: LTD.Throughput.RFC2889.MaxForwardingRate
        10. Test ID: LTD.Throughput.RFC2889.ForwardPressure
        11. Test ID: LTD.Throughput.RFC2889.AddressCachingCapacity  [ACTION] Maryam add a JIRA ticket to define a new category for this test.
        12. Test ID: LTD.Throughput.RFC2889.AddressLearningRate  [ACTION] Maryam add a JIRA ticket to define a new category for this test.
        13. Test ID: LTD.Throughput.RFC2889.ErrorFramesFiltering  [ACTION] Maryam add a JIRA ticket to define a new category for this test.
        14. Test ID: LTD.Throughput.RFC2889.BroadcastFrameForwarding  [ACTION] Maryam add a JIRA ticket to define a new category for this test.
    • Packet Latency tests
      1. Test ID: LTD.PacketLatency.InitialPacketProcessingLatency
      2. Test ID: LTD.PacketDelayVariation.RFC3393.Soak
    • Scalability tests
      1. Test ID: LTD.Scalability.RFC2544.0PacketLoss
      2. Test ID: LTD.MemoryBandwidth.RFC2544.0PacketLoss.Scalability
    • Coupling between control path and datapath Tests
      1. Test ID: LTD.CPDPCouplingFlowAddition
    • CPU and memory consumption
      1. Test ID: LTD.CPU.RFC2544.0PacketLoss
  • Documentation integration with RELENG
    • Ram made some great progress here. The bash scripts and yml files have been setup and configured, and some testing is underway. More to follow next week.
  • PDV to be collected in RFC5481 format and IXIA:

August 26th 2015

  • [ACTION] Maryam to follow up with Aric about JIRA emails/updated to committers and contributors Status: Open, Pinged Aric, waiting for response. Will ping him again and CC Gene and Al
  • [ACTION] Al to updated the LTD to discuss sampling and details of PDV.
  • [ACTION] Maryam to setup a POD3 reservation wiki. Status: Closed.
  • TOIT/VSPERF development update
    • Features under development
      • PVP vhost-cuse merged to master
      • CPU and memory stress tests – merged to master.
      • Patch to affinitize DPDK_OVS threads – merged to master
      • Bug fix for pkt_size and duration selection from command line – merged to master
      • Migration of all documentation to docs/ and conversion to ReStructuredText – merged to master. Next steps: integration with RELENG, TV is looking at this
      • PVP enablement for Vanilla OVS – in progress
      • Affinitization for Vanilla OVS – in progress
      • update sysmetrics implementation – in progress
      • Frame modification tests implementation on hold till Billy returns..
    • Traffic Gen integration update
      • Xena: Rick is working on validating the workaround Dan provided for testing RFC2544 through th sshd server.
      • Moongen: author started adding support for 40Gb Intel NICs Xl710, and Andrew is working on adding the latency aspect to the RFC2544 script.
        • [ACTION] Maryam to follow up with Trevor about setting up a server where moongen can run connected back to back to the DUT
      • Spirent: New contributor onboard – Brian Castelli, welcome Brian.
    • CI Integration – In progress.
      • Setting up 2 separate instances back to back this week and running vsperf locally.
      • looking at integrating the Jenkins jjb with toit.
      • Gene setup a test methodology wiki https://wiki.opnfv.org/vsperf/test_methodology it’s a WIP.
      • Gene added the diagrams with Lucid Chart, an open source online service. The documents are public and can be exported in Visio format if additional editing is required.
      • TOIT High Level Design – [ACTION] Billy
      • Test Spec Overview – [ACTION] Al
  • PDV to be collected in RFC5481 format and IXIA:
    • Firmware on IXIA chassis attached to POD3 has been upgraded.
    • the POD3 TCL clients on the server have been upgraded and tested – previous tests pass.
    • PDV within a single flow is what we are looking to collect. We need an idea of the delay distribution to determine how best to apply the stats. We will probably have to play around with the bucket settings to see how the distributions are affected. This will be documented by Al in the LTD.
  • POD 3 update

* POD 3 Scheduling – Booking wiki Setup https://wiki.opnfv.org/wiki/pod3_booking_calendar

  • Only VSPERF contributors and committers should be requesting access to the POD to run vsperf tests.
  • The POD is not setup for performance testing right now, it’s a validation environment – will need to tweak BIOS and grub configuration as well as RAM on the system to set it up for performance.
  • YARDSTICK and VSPERF invite for a joint design session (Meeing Invitation attached)
    • When: August 31st, UTC 14h00
    • Goal: Establishing communication between YARDSTICK and VSPERF frameworks
  • [pharos][functest][yardstick][qtip] Weekly Test Project Sync and Inform
    • VSPERF Demo 27th August 2015
  • VSPERF Release B participation

August 19th 2015

  • Recognitions/Shout outs
    • Al Morton, Pierre Lynch, Bogdan Tenea , and Trevor Cooper for driving and supporting the RFC5481 PDV collection with IXIA.
  • Actions
    • [ACTION] Al update LTD and IETF draft with the correct terminology in terms of throughput/maximum forwarding rate for benchmarking methodology. Status: Closed,
    • [ACTION] Gene to share abstracts for OPNFV Summit/talk names – Status: Closed, abstract in the minutes below.
    • [ACTION] Maryam PVP Demo to be setup on POD 3 for people to be able to run – Status: Closed, Please see POD3 update below
    • [ACTION] Maryam to follow up with Aric about JIRA emails/updated to committers and contributors Status: Open
  • TOIT/VSPERF development update
    • Features under development
      • PVP vhost-user merged to master.
      • RFC 2544 scalability test Merged to master.
      • PVP vhost-cuse will be pushed to gerrit this week.
      • Frame modification tests implementation on hold till Billy returns..
      • CPU and memory stress tests – tests to be pushed to gerrit this week.
      • Migration of all documentation to docs/ and conversion to ReStructuredText - pushed to gerrit for review. Next steps: integration with RELENG
    • CI Integration – In progress, Gene is looking at this.
  • PDV to be collected in RFC5481 format and IXIA:
    • Details of the methodology to orchestrate the time stamping has been locked down: start and stop the packet capture from the TCL scripts, then we should be able to extract the pcap file and do some post processing on it. We will need some help from IXIA on interpreting the time stamps but after that the calculation of PDV should be straight forward.
  • YARDSTICK and VSPERF invite for a joint design session (Meeing Invitation attached)
    • When: August 31st, UTC 14h00
    • Goal: Establishing communication between YARDSTICK and VSPERF frameworks
  • OPNFV Summit Call for Participation Gene’s Abstracts
    • Test Automation with OPNFV: Test Automation with OPNFV ( Gene Snider, Huawei Inc. ) - Leveraging the assorted test infrastructures developed by the OPNFV can be quite challenging. This talk will discuss various components of the OPNFV test infrastructure and their deployment in house by OPNFV members. Topics to be covered will include an overview of Gerrit and Jenkins, the use of Jenkins Job Builder to manage automation and how to create and use a virtual machine infrastructure for OPNFV testing. Initially this is intended to include the OCTOPUS and VSPERF projects, the functest project will be covered to the extent that it is integrated with the releng process.
    • Creating VNFs with vmbuilder: Creating VNFs with vmbuilder ( Gene Snider, Huawei Inc. ) - The process of creating a minimum bootable runtime image to provide virtual network functions is not well documented or well understood. In this presentation Gene Snider will present a step by step tutorial on creating a minimal virtual machine suitable for hosting VNF solutions. This presentation will cover scriptable solutions for building and packaging custom VNF solutions.
  • Readout/discussion this week: VSPERF Release B participation: slides and plan can be found here https://wiki.opnfv.org/wiki/vswitch_release_plan
    • Brahmaputra target release date for February 2nd, 2016
    • Please review the backlog to see if there are stories that are missing that need to be added in.

August 12th 2015

  • Actions
    • [ACTION] Gene to update https://wiki.opnfv.org/wiki/vswitch_automated_test_framework –> WIP, Gene is looking at this
    • [ACTION] Gene to share abstracts for OPNFV Summit/talk names.
    • PDV to be collected in RFC5481 format - investigate if it is possible to get the delay times for individual packets so we can calculate ourselves or it this is supported by traffic generators - Status: Open – See below
      • [ACTION] Maryam to get the model of the IXIA chassis connected to POD3 – Closed. Chassis is Optixia XM2, Version: IxNetwork 7.30.917.27 GA, IxOS 6.70.1050.16 GA .
      • [ACTION] Andrew Theurer to check availability for a Moongen Demo/readout – closed Demo/Readout August 12th 2015
    • [ACTION] Maryam PVP Demo to be setup on POD 3 for people to be able to run – In progress
  • TOIT/VSPERF development update
    • Features under development
      • PVP vhost-user reworked and pushed to gerrit.
      • PVP vhost-cuse will be pushed to gerrit this week.
      • Vanilla OVS merged to master.
      • Simple perf test – merged to master
      • Frame modification tests implementation on hold till Billy returns..
      • CPU and memory stress tests investigation kicked off.
      • RFC 2544 scalability test is being reworked to the new definition in the LTD.
    • Integration of Xena, Moongen & Spirent traffic generators
      • Xena – Work around for RFC2544.exe for Linux using Xena Web presented by Dan Amzulescu dsa@xenanetworks.com, if anyone is looking for assistance with this please contact Dan directly for instructions.
      • Moongen: Please see Moongen RFC2544 script below.
      • Spirent – no update [ACTION] Maryam to check in with Gurpreet
    • CI Integration – In progress, Gene is looking at this.
  • PDV to be collected in RFC5481 format and IXIA:
    • Al has been investigating this and it is possible with some IXIA equipment which have a HW capture buffer. We can capture a certain number of packets for a particular flow/stream (through filtering) with IXIA and should be able to calculate the PDV for packets by processing the timestamps on the packets.
    • Version of IXOS installed on the IXIA chassis connected to POD 3 doesn’t have the capabilities we need to do this – investigating the option of upgrading IXOS.
  • OPNFV Summit Call for Participation Abstract
    • Virtual switches have been long recognized as an essential component of the virtual network in the data centre, and are growing in popularity in the telecoms sector. Exploding traffic is creating extraordinary demand on the network and with that, exceptional performance requirements for the interfaces into Virtual Network Functions in the NFVI. One such interface is the virtual switch, thus it is crucial to be able to characterize its performance in order to determine its capabilities and suitability for deployment in Telco NFV environments. This talk looks at the work undertaken by the OPNFV VSPERF project to define a test suite for characterizing the performance of the virtual switch and any considerations that must be taken into account when doing so. http://events.linuxfoundation.org/cfp/proposals/6770/6242
    • 2 Proposals from Gene for OPNFV – one relating to releng and vsperf test automation, Gene to share more details.
  • Readout this week: VSPERF Release B participation: slides and plan can be found here https://wiki.opnfv.org/wiki/vswitch_release_plan - Only did a quick overview, discussion Item for next week.
  • Open discussion:
    • IMIX traffic (IPv4, TCP and IPv6) and Mobility MIX in OPNFV – OPNFV Testing problem we need to resolve with the community.

August 5th 2015

  • Actions
    • TV to update Packet loss ratio tests acceptable loss % based on the test duration. Status: Closed.
    • [ACTION] Gene to update https://wiki.opnfv.org/wiki/vswitch_automated_test_framework –> WIP, Gene is looking at this
    • Andrew (Xena) offered a readout on the Xena traffic generator –[ACTION] Billy to follow up - Open.
    • PDV to be collected in RFC5481 format - investigate if it is possible to get the delay times for individual packets so we can calculate ourselves or it this is supported by traffic generators - Status: Open
      • Al has been investigating this and it is possible with some IXIA equipment which have a HW capture buffer.
      • [ACTION] Maryam to get the model of the IXIA chassis connected to POD3.
      • Andrew Theurer joined in the discussion and mentioned that it is possible Moongen could also be extended/modified for this purpose. Andrew gave a good overview of the Red Hat contributions to Moongen and future work, please see Moongen update below.
      • [ACTION] Andrew Theurer to check availability for a Moongen Demo/readout.
  • TOIT/VSPERF development update
    • Features under development
      • PVP pushed to gerrit, being reworked.
      • Vanilla OVS pushed to gerrit.
      • Simple perf test – merged to master
      • Frame modification tests implementation ongoing.
      • vhost-cuse support for PVP deployment implementation ongoing.
      • CPU and memory stress tests investigation kicked off.
    • Integration of Xena, Moongen & Spirent traffic generators
      • Xena are dedicating some resources to port their pre-canned RFC2544 tests from an exe process to a Linux process. (few months away)
        • No definite timeframe, there’s a potential workaround that we can use – need to do an analysis on the cost of doing this work now..
      • Moongen – update below.
    • Gurpreet update on integrating the Spirent traffic generator with VSPERF - Open.
    • CI Integration – In progress, Gene is looking at this.
  • Moongen
    • Moongen is a fully scriptable high-speed packet generator built on DPDK and LuaJIT. It can saturate a 10 Gbit/s connection with 64 byte packets on a single CPU core while executing user-provided Lua scripts for each packet. Multi-core support allows for even higher rates. It also features precise and accurate timestamping and rate control. (https://github.com/emmericp/MoonGen)
    • Andrew Theurer mentioned that Red Hat are looking at modifying/extending it in an attempt to emulate/conform to the VSPERF LTD. This will lower the barrier of entry to VSPERF and is useful for prototyping/beta testing.
    • Andrew has been looking at, modifying and contributing to Moongen (modifying existing tests, and added an implementation for a 10Gb adapter, trying to enable a 40Gb adapter at the moment).
    • Red Hat are interested in implementing the VSPERF Spec in Moongen and integrating it with VSPERF.
  • OPNFV Summit Call for Participation Abstract
    • Virtual switches have been long recognized as an essential component of the virtual network in the data centre, and are growing in popularity in the telecoms sector. Exploding traffic is creating extraordinary demand on the network and with that, exceptional performance requirements for the interfaces into Virtual Network Functions in the NFVI. One such interface is the virtual switch, thus it is crucial to be able to characterize its performance in order to determine its capabilities and suitability for deployment in Telco NFV environments. This talk looks at the work undertaken by the OPNFV VSPERF project to define a test suite for characterizing the performance of the virtual switch and any considerations that must be taken into account when doing so.
    • http://events.linuxfoundation.org/cfp/proposals/6770/6242
  • IETF Draft LTD Summary
    • [Action] update LTD and IETF draft with the correct terminology in terms of throughput/maximum forwarding rate for benchmarking methodology – Al volunteered.
    • Put this on hold and try to implement the tests and getting results – to build a compelling story.
    • Should aim continually update the spec in the IETF meeting cycles.
  • Gerrit patches Status (https://gerrit.opnfv.org/gerrit/#/q/project:vswitchperf ):
  • vnfs: Enable PVP using vhost-user – to be reworked
  • test_spec: Add LTD.CPDPCouplingFlowAddition
  • test_spec: Clarify LTD.CPU.RFC2544.0PacketLoss
  • test_spec: Clarify LTD.Throughput.RFC2889.BroadcastFrameForwarding – Merged to master
  • src/dpdk: Enable building of vhost-user in src/dpdk. – merged to master
  • Support vswitch selection from the command line. – merged to master
  • Add a simple performance test that sends a continuous stream – merged to master
  • test_spec: Clarify LTD.PacketLatency.InitialPacketProcessingLatency – Merged to master
  • Vanilla OVS support implementation – Merged to master
  • Readout this week: VSPERF VM loopback demo on POD3. [ACTION] Maryam PVP Demo to be setup on POD 3 for people to be able to run.
  • Open discussion:
    • IMIX traffic (IPv4, TCP and IPv6) and Mobility MIX in OPNFV – OPNFV Testing problem we need to resolve with the community.

July 29th 2015

  • Actions
    • TV to update Packet loss ratio tests acceptable loss % based on the test duration. Status: open (commit permission issues)
    • Gene to update https://wiki.opnfv.org/wiki/vswitch_automated_test_framework –> WIP, Gene is looking at this
    • PDV to be collected in RFC5481 format - Maryam to investigate if this it is possible to get the delay times for individual packets so we can calculate ourselves or it this is supported by traffic generators - Status: Open
    • Andrew (Xena) offered a readout on the Xena traffic generator – Billy to follow up - Open.
    • Rick to follow up with Dan on Moongen.
  • TOIT/VSPERF development update
    • Features under development
      • PVP pushed to gerrit, please see patch review list below.
      • Vanilla OVS landing end this week or start of next.
    • Integration of Xena & Spirent traffic generators
      • Xena are dedicating some resources to port their pre-canned RFC2544 tests from an exe process to a Linux process.
      • No update on Spirent - Open
    • Gurpreet update on integrating the Spirent traffic generator with VSPERF - Open.
    • CI Integration – In progress, Gene is looking at this.
  • IETF Draft LTD Summary
    • Maryam & Al presented the draft to IETF on Thur 23rd (https://wiki.opnfv.org/_media/wiki/benchmarking_vswitches_in_opnfv_1.6_aspect_4_3.pdf), presentation was very well received and the BMWG is intrested in seeing how our work progresses. We got some invaluable feedback from Scott Bradner regarding the terminology we are using when we refer to throughput and maximum forwarding rate for our soak tests. For Throughput you need to stop transmitting for a period of time to the DUT, then measure what the device is pushing. Whereas in the soak tests what we are measuring is the DUT max forwarding rate.
  • Patches open for review (https://gerrit.opnfv.org/gerrit/#/q/project:vswitchperf ): Please review folks.
    • test_spec: Clarify LTD.Throughput.RFC2889.BroadcastFrameForwarding
    • vnfs: Enable PVP using vhost-user
    • test_spec: Clarify LTD.PacketLatency.InitialPacketProcessingLatency
    • src/dpdk: Enable building of vhost-user in src/dpdk.
    • test_spec: Add LTD.CPDPCouplingFlowAddition
    • Support vswitch selection from the command line.
    • Add a simple performance test that sends a continuous stream
  • Open discussion:
    • IMIX traffic (IPv4, TCP and IPv6) and Mobility MIX in OPNFV - Investigation underway
  • Readout next week: VSPERF VM loopback demo on POD3.

July 22nd 2015

  • TOIT/VSPERF development update
    • Features under development
      • Bidirectional traffic merged.
      • Frame Modification (push VLAN tag) merging this week.
      • PVP & Vanilla OVS landing end this week or start of next.
    • POD3
      • At least one community member has logged on used POD3 successfully.
    • Integration of Xena & Spirent traffic generators
      • Andrew (Xena) gave a brief demo of Xena in respect of the latency test discussions below. Also said Linux support (using mono) was about 6wks away - Open
      • No update on Spirent - Open
      • RH also considering moongen (dpdk based app, University in Munich) as an off-DUT software trafficgen.
  • Test definitions under review
  • Slowpath Latency & Fastpath Latency: good input from all contributers. Billy to submit clarified versions - Open.
  • PDV to be collected in RFC5481 format – Maryam to investigate if this it is possible to get the delay times for individual packets so we can calculate ourselves or it this is supported by traffic generators – Status: Open
  • Open discussion
    • IMIX traffic (IPv4, TCP and IPv6) and Mobility MIX in OPNFV – Investigation underway
    • Andrew (Xena) offered a readout on the Xena traffic generator - Open.

July 15th 2015

  • Actions From Last week
    • Billy to draft a new test for profiling packet loss. Status: closed, merged to master.
    • Add Test report section to every test case in the LTD and specify the fields that should be reported. We should also expand the LTD to report platform specific info that we should gather. Status: closed, merged to master.
  • Future action – Cover SDN flow controller tests. Add a Jira ticket for this, Maryam. Status: Closed, VSPERF-54.
  • TV to update Packet loss ratio tests acceptable loss % based on the test duration. Status: open
  • Gene to update https://wiki.opnfv.org/wiki/vswitch_automated_test_framework –> open till Gene comes back. Status: open.
  • TOIT/VSPERF development update
    • Features under development:
      • Bidirectional functionality ready to be pushed to gerrit.
      • xunit parsing patch merged to master.
      • Frame modification (vlan tag pushing) under review in gerrit. https://gerrit.opnfv.org/gerrit/#/c/996/
      • pvp deployments (PVP is in progress)
      • Enabling OVS kernel Datapath (in progress)
    • POD3 has been setup to allow folks to login and run VSPERF https://wiki.opnfv.org/get_started/pod_3_-_characterize_vswitch_performance
    • Gurpreet update on integrating the Spirent traffic generator with VSPERF – Open.
    • RH/Xena traffic gen being integrated with vsperf.
    • Gene sw traffic gen integration
      • evaluating netexpect – experiments so far have had disappointing results (nowhere near 10Gb)
      • CI – integration with Jenkins (exploration/discussion underway).
  • vloop VNF – uploaded to the large object archive (Project Milestone) Kudos to Gene!
  • DPDK 2.0 Build
    • TV is to push any additional fixes – Status: Closed.
  • Test definitions under review:
  • PDV to be collected in RFC5481 format – Maryam to investigate if this it is possible to get the delay times for individual packets so we can calculate ourselves or it this is supported by traffic generators – Status: Open
    • We should add a JIRA ticket for meshed tests – Status: Closed, VSPERF-55.
  • Open discussion:
    • IMIX traffic (IPv4, TCP and IPv6) and Mobility MIX in OPNFV – Investigation underway

July 8th 2015

  • Actions From Last week
  • Future action – Cover SDN flow controller tests. Add a Jira ticket for this, Maryam. Status: open, need to check if this is covered by Yardstick
  • TV to update Packet loss ratio tests acceptable loss % based on the test duration. Status: open
  • Gene to update https://wiki.opnfv.org/wiki/vswitch_automated_test_framework –> open till Gene comes back. Status: open.
  • TOIT/VSPERF development update
    • Features under development:
      • Back to back test merged to git.
      • Bidirectional functionality ready to be pushed to gerrit.
      • xunit parsing patch under review in gerrit.
      • Frame modification (vlan tag pushing) tests ready to be pushed to gerrit.
      • pvp and pvvp deployments (PVP is in progress)
      • Enabling OVS kernel Datapath (still on hold)
    • POD3 is being setup to allow folks to login and run VSPERF (Traffic Gen issues being debugged and worked through).
    • Gurpreet update on integrating the Spirent traffic generator with VSPERF – Open.
    • Gene sw traffic gen integration – is this something that is still on the horizon.
      • evaluating netexpect.
      • CI – integration with Jenkins (exploration/discussion underway).
  • DPDK 2.0 Build
    • TV is to push any additional fixes – Status: Open.
  • Test definitions under review:
  • PDV to be collected in RFC5481 format – Maryam to investigate if this it is possible to get the delay times for individual packets so we can calculate ourselves or it this is supported by traffic generators – Status: Open
    • We should add a JIRA ticket for meshed tests – Status: Open.
  • Open discussion:
    • IMIX traffic (IPv4, TCP and IPv6) and Mobility MIX in OPNFV – Investigation underway
    • The LTD should specifically mention the tuples that should be used when matching a flow (variations include: (inport, outport), Network 5 tuple, Network 13 tuple) – Status: Closed – A flow classification entry has been added to the LTD.

July 1st 2015

  • Future action – cover SDN flow controller tests. Add a Jira ticket for this, Maryam. Status: open.
  • TV to update Packet loss ratio tests acceptable loss % based on the test duration. Status: open
  • Gene to update https://wiki.opnfv.org/wiki/vswitch_automated_test_framework –> open till Gene comes back. Status: open.
  • TOIT development update
    • Features under development:
      • Back to back test ready to be pushed to gerrit.
      • Bidirectional functionality ready to be pushed to gerrit.
      • xunit parsing patch to be pushed to gerrit.
      • Frame modification tests – Have a working solution, needs some more testing before integrating Openflow rules into VSPERF.
      • Enabling OVS kernel Datapath (still on hold), pvp and pvvp deployments (still ironing out some issues.)
    • POD3 is being setup to allow folks to login and run VSPERF (CentOS 7 has been installed, we are just configuring the system in use for VSPERF).
    • Gurpreet update on integrating the Spirent traffic generator with VSPERF – Open.
  • IETF Draft LTD Summary
    • Al has pushed the latest version of the draft to gerrit – some final touches to be made.
    • Isolation and reporting should be included in the presentation.
    • Should be finalized Friday 03 July 2015.
  • DPDK 2.0 Build
    • Maryam will publish the basic build fixing patches → Status: Done, patches merged onto master
    • TV is to push any additional fixes – Status: Open.
  • LTD New additions :
    • Test results section.
    • Test Summary section.
    • New Latency profile test.
  • Test definitions under review:
    • PDV to be collected in RFC5481 format – Maryam to investigate if this it is possible to get the delay times for individual packets so we can calculate ourselves or it this is supported by traffic generators – Status: Open
    • RFC2889 Forwarding test – should be renamed to max forwarding tests and reference RFC2889.
    • We should add a JIRA ticket for meshed tests.
  • Open discussion:
    • IMIX traffic (IPv4, TCP and IPv6) and Mobility MIX in OPNFV – Investigation underway
    • The LTD should specifically mention the tuples that should be used when matching a flow (variations include: (inport, outport), Network 5 tuple, Network 13 tuple) – Status: Open

June 24th 2015

  • Actions From Last week
    • Gurpreet to publish slides on “Overture vSwitch Performance” to the project wiki.
    • Gene to submit JIRA bug for relative paths issue with VSPERF.  Maryam to take this as Gene is on holiday. Status: closed.
    • Maryam to update ltd to indicate RFC2889 address caching and learning tests are applicable to MAC learning switches only. Status: closed.
  • Future action – cover SDN flow controller tests. Add a Jira ticket for this, Maryam. Status: open.
  • Billy to draft a new test for profiling packet loss. Status: open.
  • TV to update Packet loss ratio tests acceptable loss % based on the test duration. Status: open
  • Al to publish Isolating Functions in the System Under Test on the wiki. Status: open
  • Add Test report section to every test case in the LTD and specify the fields that should be reported. We should also expand the LTD to report platform specific info that we should gather.
  • Gene to update https://wiki.opnfv.org/wiki/vswitch_automated_test_framework –> open till Gene comes back. Status: open.
  • Update the meeting link or remove from the wiki, Status:Done.
  • TOIT development update
    • Features under development: Final testing of Back to back test is in progress, xunit parsing patch to be posted soon, frame modification tests (we’re still exploring how to implement this test), enabling OVS kernel Datapath (On hold), pvp and pvvp deployments (ironing out some issues.)
    • POD3 is being setup to allow folks to login and run VSPERF (encountered some remote control issues and we are working on resolving this).
    • Gurpreet update on POD11?
    • Gurpreet update on integrating the Spirent traffic generator with VSPERF – Gurpreet has ramped up on the code and gone throught the High level design document and it’s just implementation that remains.
  • IETF Draft LTD Summary
    • Al has pushed the latest version of the draft to gerrit – needs to be rebased against the latest master.
    • Submission date July 6th
  • DPDK 2.0 Build
    • Maryam will publish the basic build fixing patches this week.
    • TV is to push any additional
  • Test definitions under review:
    • LTD.Throughput.RFC2544.SoakFrameModification
      • 0% packet loss to be taken out
      • Metrics collected to be modified to match LTD.Throughput.RFC2544.Soak.
      • PDV to be collected in RFC5481 format – Maryam to investigate if this is possible.
    • LTD.Throughput.RFC2544.BacktoBack – no modifications
  • Open discussion:
    • IMIX traffic (IPv4, TCP and IPv6) – we should expand the LTD to include IMIX traffic to tests as a traffic type.
    • The LTD should specifically mention the tuples that should be used when matching a flow (variations include: (inport, outport), Network 5 tuple, Network 13 tuple)

June 17th 2015

  • Actions From Last week
    • Gene to submit JIRA bug for relative paths issue with VSPERF.  AR Maryam to take this as Gene is on holiday.
    • Gene to update https://wiki.opnfv.org/wiki/vswitch_automated_test_framework –> open till Gene comes back.
    • Maryam to investigate RFC2889 and whether it’s simple related to MAC learning. If it is, tests should indicate that they are only applicable to vswitches that support MAC learning. → Update ltd to indicate they are applicable to MAC learning switches.
      • Investigation CLOSED, the 2 tests related to address caching and address learning are related to MAC learning frames.
      • OPEN: Maryam to update test cases with the relevant comments.
    • Future action – cover SDN flow controller tests. Add a Jira ticket for this, Maryam
      • Maryam to look into how we can achieve/implement the RFC 2889 tests
        • Tests can be conducted with IXIA or Spirent traffic generator.
      • Billy to draft a new test for profiling packet loss. Status: open.
  • Packet loss ratio tests
    • Tests that run for a short period of time should be updated to test 0% packet loss.
    • Tests that run for a long period of time (soak tests) should have the 10^-7% packet loss ratio.
      • Action TV to update these tests
  • TOIT development update
    • Features under development: Back to back tests, frame modification tests, enabling OVS kernel Datapath, pvp and pvvp deployments.
    • POD3 is being setup to allow folks to login and run VSPERF.
    • Gurpreet mentioned we might be able to also get setup on POD11.
    • Gurpreet is looking at integrating the Spirent traffic generator with VSPERF.
    • Ideally we want to find a POD where we are connected to all the VSPERF supported traffic generators.
  • IETF Draft LTD Summary
    • Really good progress being made here, Al added in a 3x4 matrix within which we can track the coverage of our tests and identify gaps.
    • The draft has been merged to master.
  • DPDK 2.0 Build
    • TV is making good progress on this front.
    • validated the suggested fixes from Maryam.
    • currently has connectivity issues with gerrit and can’t push fixes.  ACTION TV to contact opnfv-helpdesk@rt.linuxfoundation.org

* Readout

  • Isolating Functions in the System Under Test: Al led a discussion on how we can try to get consistent test results without too much variation.
  • ACTION: Slides to be published on the wiki.
  • Action Maryam work with Al on making additions to the diagrams.
  • Other ACTIONS:
    • Gurpreet to publish slides on “Overture vSwitch Performance” to the project wiki.
    • Homework for next week - Billy to nominate tests for discussion.

June 10th 2015

  • VSPERF (TOIT) Update
    • Code has been +2’d and merged to master.
    • We are looking at getting POD3 in the Portland Lab setup for folks to be able to go in and run VSPERF and provide feedback on the framework.
    • We need to investigate if we can setup a test bed in another OPNFV lab that has Spirent equipment. Gene mentioned that the Huawei Lab could be a suitable candidate.
    • Gene is looking at integrating a SW packet generator with VSPERF. Gurpreet mentioned that Spirent could do the same.
  • IETF Summary LTD Draft
    • Good progress being made here, Al is going to push a revised version of the Draft.
    • We need to add in to our spec and this draft how we plan on isolating our measurements so that the results of tests can be repeatable.
      • This will be a discussion item for next week, please see Homework.
    • We need to put some slides together for the Next IETF meeting and socialize our work in order to get it adopted as a working group draft.
  • Homework:
    • 'Considerations on Benchmarking VNFs and their Infrastructure'
  • Actions
    • Gene to submit JIRA bug for relative paths issue with VSPERF.
    • Gene to update https://wiki.opnfv.org/wiki/vswitch_automated_test_framework
    • Maryam to investigate RFC2889 and whether it’s simple related to MAC learning. If it is, tests should indicate that they are only applicable to vswitches that support MAC learning.
    • Maryam to look into how we can achieve/implement the RFC 2889 tests.
    • Billy to draft a new test for profiling packet loss.
  • Packet loss ratio tests
    • Tests that run for a short period of time should be updated to test 0% packet loss.
    • Tests that run for a long period of time (soak tests) should have the 10^-7% packet loss ratio.
    • A new test to be drafted to give a packet loss profile for a vSwitch.

June 3rd 2015

  • TOIT Update
    • Code has been pushed for Gerrit review last Friday. [Billy O'Mahony]
    • Huawei currently playing with same. [Gene Snider]
      • Gene is keen to integrate Spirent, Huawei traffic generators as well as some open source VM-based Traffic Generators.
  • Readout on Overture vSwitch Performance Test
    • Gurpreet gave a readout on some performance tests using a low-end COTS system. [Gurpreet]
    • [AR] Gurpreet will post these slides on the VSPERF wiki
  • AOB:
    • Billy asked for volunteers to go on as 'Reviewed-by' for test case reviews. Gene, TV, Gurpreet and Al all agreed.
  • Homework:
    • Next week the following two tests will be reviewed:
      • LTD.Throughput.RFC2889.AddressCachingCapacity
      • LTD.Throughput.RFC2889.AddressLearningRate

May 27th 2015

  • TOIT Update
    • Code is expected to be pushed to OPNFV Gerrit at the end of this week. [Billy O'Mahony]
  • IETF Summary LTD Draft
    • On track to meet draft submission deadline [Al Morton]
    • Some figures require editing to reduce column count as per IETF documentation norms.
  • AOB
    • Gene Snider asked if TOIT would support other traffic generators. Billy said this was a core requirement of TOIT.
    • Gurpreet asked for some time to present some information of VNF testing at the next meeting.
    • [AR] Gurpreet to contact Billy to agree presentation duration.
  • Reviewed 2 LTD test cases:
    • LTD.Throughput.RFC6201.ResetTime:
      • Definition of 'reset' should include both a ungraceful forced reset by the OS (e.g. kill -9) and a graceful restart as per the virtual switches documentation. A test implementation should support both reset methods as an argument to the test.
      • Make explicit that 'average' refers to the average over n trials of the test under identical conditions.
    • LTD.Throughput.RFC2544.SystemRecoveryTime:
      • Clarify that the pre and post overload traffic should be 75% of the RFC2544 maximum throughput.
      • Increase the latency averaging interval
    • [AR] Billy will make and submit these changes to Gerrit.

May 20th 2015

  • TOIT Update
    • Framework integration with low level components is almost complete. We are starting the port to python 3.3.3.
  • IETF Summary LTD Draft
    • Progress made on adding in deployment scenarios to the first draft.
  • Jenkins Build:
    • [Action] TV Rao to push draft patch to gerrit for review so we can try to replicate the encountered issues.
  • Reviewed 2 LTD test cases:
    • LTD.Throughput.RFC2544.Soak: some clarification is required on where 10^-7% packet loss figure came from [action] investigate this. Latency should also be collected as part of this test. Time intervals should be defined for the long/soak tests so that we can determine the time segment where anomalies occur.
    • LTD.Throughput.RFC2544.PacketLossRatioFrameModification: decision made to change the default modification to adding/modifying a VLAN tag.
    • Overall there is consensus for a new section to be added to test cases that specifies what is to be documented in the test report in terms of metrics and other parameters.

Maryam is out of office till June 8th, 2015 Billy O Mahony is providing cover on VSPERF

May 13th 2015

  • LTD Summary for IETF
    • Maryam and Billy are setup with XML mind
    • Al started a great draft for summarizing the LTD to highlight to the IETF how we’ve been using the BMWG RFCs.
    • [ACTION] Maryam to push the draft to gerrit for review (https://gerrit.opnfv.org/gerrit/#/c/555/ )
    • [ACTION] Al to send on XML to RFC text online generation tool – DONE.
  • vloop_vnf
    • Has been pushed to master.
  • Discussion Backlog
    • Billy to nominate a test case to discuss at the weekly call next week.
  • Open Gerrit Reviews
    • TestSpec: Add LTD.Latency.Slowpath
    • testspec: cleanup + update tests with new metrics
    • TestSpec: Add RFC2889 Broadcast Frame Latency test
    • TestSpec: Modify soak tests to measure PDV
    • TestSpec: Modify soak tests to measure PDV
    • TestSpec: Modify LTD access permissions

I encourage VSPERF committers and contributors to review these and provide their feedback.

Thanks Sothy for the very informative readout on the Performance of OVDK + port and flow mirroring.

May 6th 2015

  • TOIT Update
    • 1st development stream (Implementing the stubbed out skeleton framework) Almost complete
    • 2nd development stream (porting the existing low level components like vswitchs and vnfs… to the new directory structure while maintaining functionality/the ability to run tests) – only one component left.
    • 3rd development stream (path-finding porting the existing framework from python 2.7 to 3.x while maintaining the ability to run tests etc…) – Done.
  • Discussion Backlog
    • I would like to propose we use part of this slot to discuss some of the test cases and their parameters that are a little unclear in the LTD. For example for Frame modification tests we recommend:
      • Modifying the Ethernet source or destination MAC address.
      • Modifying/adding a VLAN tag.
      • Modifying/adding a MPLS tag.
      • Modifying the source or destination ip address.
      • Modifying the TOS/DSCP field.
      • Modifying the source or destination ports for UDP/TCP/SCTP (Recommended).
      • Modifying the TTL.

Which is the most appropriate/applicable for our case?

I Suggest we review LTD.Throughput.RFC2544.SoakFrameModification as the first item for discussion.

  • I think the next item for discussion I would recommend is the tuple parameters to use to match on for each of the tests. i.e. what do we consider a flow.
  • Updating our pre-commit Jenkins Build
    • TV Rao volunteered to do this.
    • Maryam to send TV the details.

April 29th 2015

Attendees: Eric Dubé, Maryam Tahhan, Al Morton, Billy O’Mahony, TV Rao, gsingh, Dave Neary

Issues with GotoMeeting resulted in a meeting over IRC. So only a couple of topics covered this Week.

  • IETF - draft submission of LTD
    • 3 Volunteers: Al Morton, Billy O’Mahony and Maryam Tahhan
    • [Action: Al Morton] to have a look at the MD open and the ease of conversion to XML, we can create a simple/base doc (next week or so) and work from there.
  • TOIT update: at the moment we have 3 development streams:
    • 1st stream is porting the existing tests we had into the agreed on directory structure - this is going well, we have all the low level components (vswitches/trafficgens/vnfs) bar one moved and old toit still functioning.
    • 2nd stream we have is implementing a stubbed out skeleton framework into which the low level components can be plugged in. We’re making good progress on this front also.
    • 3rd stream which is more of a path finding stream, is taking the exisiting code and porting it from python 2.x to python 3 the methodologies and lessons learned from here can be used to port the skeleton code to python 3 also…
    • As it stands we're on track to making our first code drop in 4 weeks’ time.
  • Opens for next meeting:
    • vloop_vnf
      • Does it bypass the network stack in the guest?
      • Are we concerned that the network stack is a bottle neck?
      • Does code exist for this? If so can we push to gerrit for review?
      • Is it a proprietary VNF, or is it something we plan on developing?
    • Creating a backlog for general discussion items for the weekly meeting.

Action Items

  • Gene Snider to share notes through email or maybe do a wiki entry on the tools that were demo’ed last week.
  • Gurpreet to share a timeslot that suits to read out on DPDK enabled vSwitch performance

April 22nd 2015

Open Questions

  • LTD Summary for IETF:
    • Are we at a point where we can start this?
    • How do we get started?
    • What’s the best way to do this? My suggestion would be etherpad, or a draft patch in gerrit that we can work on, Any recommendations for a way of working on this?
  • vloop_vnf
    • Does it bypass the network stack in the guest?
    • Are we concerned that the network stack is a bottle neck?
    • Does code exist for this? If so can we push to gerrit for review?
    • Is it a proprietary VNF, or is it something we plan on developing?

Action Items

  • Gene Snider to share notes through email or maybe do a wiki entry on the tools that were demo’ed last week.
  • Gurpreet to share a timeslot that suits to read out on DPDK enabled vSwitch performance

Project Update

  • TOIT
    • The refined TOIT HLD can be found at https://gerrit.opnfv.org/gerrit/#/c/225/, Please feel free to download the patch and review.
    • Development has started and we plan on submitting our first patches to VSPERF in 5 weeks’ time. (will update on progress every week)
  • VNFs
    • Can be shared with Aric (Gardner) to push to artefact.opnfv.org so that our Jenkins jobs can access them.

General discussion

  • The generated benchmark reports need to document the minute details of the system setup including the vNIC version as performance can vary greatly between different versions.
  • Spirent’s virtual traffic generator supports RFC testing – great enabling tool, but important to note we can’t get the same performance measurement accuracy as the HW generator.
  • Need to keep the p→v use case that bypasses the vswitch (pNIC to vNIC) on our minds also.

April 8th 2015

Project Update

  • TOIT:
    • We had a quick walkthrough the TOIT High Level Design (HLD) document.
    • This HLD is available for review at: https://gerrit.opnfv.org/gerrit/#/c/225/ and covers the high level architecture for TOIT.
    • Work is continuing on the HLD to define the interfaces and the various classes, this will be published shortly for review.
  • Merged Tests to test spec:
    • Throughput.RFC2889.ForwardingRate
    • Throughput.RFC2889.ForwardPressure
    • Throughput.RFC2889.AddressCachingCapacity
    • Throughput.RFC2889.AddressLearningRate
    • Throughput.RFC2889.ErrorFramesFiltering
    • Throughput.RFC2889.BroadcastFrameForwarding
  • Opens:
    • Need some more clarification on [JIRA] (VSPERF-39) Add simple L2 forwarding module
    • Item for Discussion: How we can advertise our work to other benchmarking communities (ETSI_NFV, IETF Benchmarking Methodology Group)?
  • A new regular slot to feature in the weekly meeting going forward, this slot will be focused around what else is going on in the benchmarking community (ETSI_NFV, IETF Benchmarking Methodology Group). Al Morton kindly agreed to lead the discussions on this.
  • Suggested agenda Items for next week:
    • Eugene Snider to lead a discussion around a plug n play packet generator.
    • Clarification on [JIRA] (VSPERF-39) Add simple L2 forwarding module
    • Item for Discussion: How we can advertise our work to other benchmarking communities (ETSI_NFV, IETF Benchmarking Methodology Group)?
    • Patch submission process walkthrough.

March 25th 2015

Project Update

  • Commit gate issue:
    • Patches pushed to Gerrit were being -1'd by Jenkins CI as a result of a failure in the DPDK build issue on the CentOS 6 machine the verification job was running on. This is because the version of DPDK being built was too old and not ported to the kernel version on the platform running the test.
    • Solution:
      • Patch pushed to master to use a newer version of DPDK (1.8) and also to use a newer tag on the OVS master branch.
      • Requested that the verification job runs on a CentOS 7 or Ubuntu 14.04
  • New Contributors:
    • Tv Rao tv.rao@freescale.com
    • Kin-Yip Liu Kin-Yip.Liu@caviumnetworks.comkliu@cavium.com
  • TOIT
    • High Level Design doc will be pushed for review early next week.
  • Merged Tests:
    • Throughput.RFC2544.SystemRecoveryTime
    • Throughput.RFC2544.BackToBackFrames
    • Throughput.RFC2544.ResetTime
    • Throughput.RFC2544.Soak
    • Throughput.RFC2544.SoakFrameModification

March 11th 2015

Project Update

  • New Project Topic and email tag: VSPERF. (based on teh JIRA project)
  • 2 tests pushed to master:
    • Throughput.RFC2544.PacketLossRate
    • Throughput.RFC2544.PacketLossRateFrameModification
  • Three tests have been rebased and open for review:
    • Throughput.RFC2544.SystemRecoveryTime
    • Throughput.RFC2544.BackToBackFrames
    • Throughput.RFC2544.Soak
  • TOIT
    • Undergoing redesign to fit the prescribed directory structure
  • Vote on promoting Al Morton from contributor to committer has passed.
  • Branching Models:
    • general consensus is to continue development on master and worry about a release when the time comes.
  • CI validation
    • we need to figure out what pre and post commit tests we would like to have for the project.
  • JIRA:
    • still no notifications being received on ticket creation [Action: Maryam to follow up with LF]

Feb/11/2015

Mailing List

TOIT

  • Code is being cleaned up and prep'ed for release
  • Slides to be sent out by Stephen
  • Feedback given that tests should have a configurable duration.

Admin/Infrastructure point of contact for the project: Maryam Tahhan

Patch reviews

  • One +1 for the intro. Need another committer to +1 to progress that test spec section to the next phase
  • Review cycle to be reduced to 2 days rather than 3 days
  • Features to be tested patch rework started today by Maryam Tahhan
  • Owner column added to patch review wiki (https://wiki.opnfv.org/wiki/vswitchperf_test_spec_review) so that other contributors can also re-work patches.
  • Patch reviews and rework should be conducted in the order specified in the table (top to bottom).

Project Leader

  • Will need to select one when the selection process is hardened.

Feb/04/2015

Minutes:

  • Stephen Finucane gave an overview/demo of TOIT
    • TOIT is an OVS test integration framework written in Python that Intel would like to share with the community for re-use for this project. [Action: Stephen F. to send out slides].
    • The framework is generic and licensed under a permissible apache license.
    • Right now it supports OVS and IXIA: IxOS and IxNetworks but the framework can be easily extended to include other traffic generators and switches.
    • Community feedback was very positive, an initial commit for the phy-phy test cases is to be made to gerrit. [Action: Stephen F.]
    • Out of cycle sessions will be setup to walkthrough the code.

Project updates:

  • Initial code reviews for test spec are open. The test spec has been broken down into sections and committed to gerrit for review.
    • Question: why not a single commit? Answer: This is to avoid continuous reviewing sections of a doc that we approve.
    • In an effort to co-ordinate reviews, so sections of the test spec get pushed to gerrit, open reviews will be documented @https://wiki.opnfv.org/wiki/vswitchperf_test_spec_review
    • The suggested review cycle is 3 Working days.
  • Governance model roles for committed and contributor have been updated to the wording from the TSC-Charter (https://wiki.opnfv.org/wiki/vswitchperf_governance_model)
  • OPNFV lab HW spec for the project can be found here: https://wiki.opnfv.org/wiki/vswitch_work_environment
    • Question, what happens if we want to get different NICs…? [Action: Maryam to follow up with Trevor Cooper]
  • Tools:
    • JIRA has been setup.
      • Project name is VSPERF.
      • Please make sure you can access the project and can see the associated issues.
      • Gerrit now enforces a JIRA tag for commits.
  • Opens:
    • Do we need a separate mailing list?
    • It’s difficult to follow the project updates via the current/generic opnfv-tech-discuss mailing list. [Action: Maryam to follow up with Linux Foundation]
  • General Wiki updates:
    • Coding Guidelines : OPNFV Coding Guidelines
    • OPNFV Developer Resource Page : Getting started with your account, Gerrit and Git
    • HW Spec (specified above)
    • Open Gerrit reviews (specified above)
    • Governance model wording (specified above)
  • Open Gerrit review for repo structure (https://gerrit.opnfv.org/gerrit/#/c/9/ )
    • The repo structure review has received a single +1 from one committer. [Action: A committer to review and if applicable provide a second +1 so the patch can committed to our repo]

Jan/21/2015

21 Attendees

Summary:

  • Repo structure proposal review update:
    • Aihua has made the first commit that will allow us to review the repo structure using Gerrit. The URL for review is https://gerrit.opnfv.org/gerrit/#/c/9 .
    • Timeline for this review: All comments must be submitted by: Tuesday 27th January 2015. [Action: vswitchperf committers]
  • Governance model review:
    • Lots of good questions here and discussion about the model.
    • The following updates need to be added to the model [Action: Maryam]
      • Role of reviewer to be defined
      • Integrate features/bugs with JIRA upfront
        • This would mean implicit addition of JIRA tags to commits to ensure they are associated with a planned item, or a fix for a known issue…
        • Will allow us to have agreement on changes upfront
        • Make sure it’s Gerrit enforced.
    • Follow up with Linux Foundation to make sure committers/reviewers, contributors and maintainers have the appropriate +1, +2 options for code review. [Action: Maryam/Mike]
  • Test-bed infrastructure
    • When will we have access to the lab?
      • Access not available yet, work is ongoing to get the lab fully operational.
      • Part of the lab will be dedicated to the vSwitch characterization project.
      • Project specific requirements are necessary to provide an accurate date for accessibility to the Lab. [Action: Maryam] Put together a proposal for community review at the next weekly sync.
  • Git Tags:
    • What tags will we use for various projects?
      • This will be dependent on the switches we use. The Suggestion would be to use the tag for latest stable release of a particular switch and take the appropriate tags (that the switch is compatible with) for any other projects the switch is dependent on. For example the current stable release for OVS is 2.3.1 and it’s compatible with DPDK 1.7.1… [Action: Maryam] Propose tags for community review at the next weekly sync.
  • Tools update
  • Test spec update:
    • The test spec has been updated with Aihua, Srini and Al’s comments. We propose uploading the test spec to Gerrit in sections for review, for e.g. each test will be a separate review in Gerrit. General sections like the intro and methodology will be submitted as one patch.

Jan/14/2015

Attendees: Approx. 25 attendees

Notes:

  • Opens
    • [Action]: Mike to put in place IRC channel for next week’s meeting
    • Where do we store artifacts like VM images? Consensus that they really do not belong in GIT….
    • [Action]: Mike to raise this at Performance and Test group meeting on how to resolve as it won’t be a problem specific to this project.
    • Raised today (Jan 15th); P&G group have taken an action to make a proposal on how to handle this project-wide
  • Comments on Repo Structure proposal
    • Need directory for test specification (+ 1 by Aihua)
    • Plan not to include the source code in the repo but just to clone the Tags (+1 by Aihua)
    • Other comments: included in Aihua’s Google Docs review
    • [Action]: Aihua to present status of review at next week’s project meeting
  • Project Repo and Development tools
    • Git Review in conjunction with Gerrit is set up as part of repo creation by Linux Foundation
    • This is being used in Huawei (GitReview). Operationally similar to GitPush. No objections to using this

Project Development Tools – what candidates?

Code review: Gerrit… no objections

Bug tracking:

  • Could use GitHub bug tracker (assuming we are building on GitHub – need to confirm with Linux Foundation) Alternative could be to use Bugzilla.
  • Bryan Sullivan has experience of GitHub Bug tracker – seems to be pretty effective +1
  • Bugzilla is acceptable fallback if GitHub bug tracker not available (Bryan again).
  • Need to ensure we can enforce a mechanism that makes it clearly visible what bugs are being fixed (Gene Snider)
  • Ideally we should not re-invent the wheel, let’s use as much default from Linux Foundation as possible, unless consensus exists that it’s not acceptable
  • Test Specification draft
    • Maryam presented the structure of the test spec
      • Bryan: is there anything there from QoS perspective? No….Need to get feedback from community here.
      • Al Morton (AT&T): Happy to contribute to some parts of this. Mike to add Al as a contributor [done]
      • Al to send out links to some IETF benchmarking/performance RFCs and drafts [done]
  • Governance structure
    • We need to agree a Governance and Review structure!
    • We are doing some trailblazing here from OPNFV perspective which may serve the benefit of the whole community
    • [Action]: Maryam to provide a detailed proposal to the mailing list on a Governance Model
meetings/vsperf.txt · Last modified: 2016/03/11 13:31 by Simona Coppola

Page Tools

Except where otherwise noted, content on this wiki is licensed under the following license: Creative Commons Attribution 4.0 International License
Donate Powered by PHP Valid HTML5 Valid CSS Driven by DokuWiki