Characterize vSwitch Performance for Telco NFV Use Cases: Team Meetings
Info
Weekly Meeting Every Wednesday 08:00 PT (
1600) UTC
-
-
Mar 9th 2016
* Agenda:
* ACTIONS
[ACTION] Maryam to add CI overview to agenda, Wednesday 16th March
[ACTION] Bill to create an updated installation / usage version of the tool as the one I presented several weeks ago is now obsolete.
[ACTION] Simona to check with Maryam if we have a best practice guide when upgrading to new version of VSPERF
[ACTION] All please provide input to Rel C planning etherpad or confirm acceptance (see link below to etherpad)
[ACTION] Maryam to provide an update next week on help that is being provided by Ixia
* Benchmarking Overview
Al gave an overview of Benchmarking for Virtual Switches
Also presented the slides at Test projects - weekly coordination meeting, main aim to have it adopted by the wider community
* Project Dev Updates
* Traffic Gen updates
Xena Update
Moognen Update
Re-write of LUA wrapper done by Andrew more detail on github
User guide is currently obsolete, plan to stabilize the new version of Moongen before creating the new user guide, should have something for next week.
-
Spirent Update
* Release C updates
Indicated our intent to be part of REL C as planned on March 8th
Now need to focus on assessing our impacts to other projects such as installers, CI, Infrastructure
-
Meeting with IXIA for packet capture was productive, Plan is to move some TCL to python
Mar 2nd 2016
* Agenda:
* ACTIONS
[ACTION] Maryam to add CI overview to agenda in two weeks, Wednesday 16th March
[ACTION] Bill to create an updated installation / usage version of the tool as the one presented several weeks ago is now obsolete.
[ACTION] Simona to send details via email of top Priorities for Rel C - Complete
[ACTION] All to review and agree final REL C proposal and or provide feedback for any changes/updates
* Project Dev Updates
* Traffic Gen updates
Xena Update
Moognen Update
Well done to Andrew and Bill, fantastic progress has been made in last few weeks.
Andrew Theurer has made significant changes to the tool. This is in regards to the Lua wrapper around the DPDK engine, in particular the file opnfv-vsperf.lua which instruments
RFC 2544 like testing.
The changes include but are not limited to: Multi (concurrent) device support. As many pairs of ports as the user wants -Multi queue support for transmit -Hardware tx method is default.
All bugs with hw tx should now be avoided (with multi queue support) -Major clean up and reorganization of code -Test parameters are now in file -no command line arguments -Tx rate is calibrated before testing to never be above requested rate
-Uni/bidirectional logic handled in connections[] array -Use log: facility that comes with MoonGen -Too low tx rate detection allows N attempts instead of aborting on first failure
The tool is now ported from DPDK 1.71 to DPDK 2.2
-
Spirent Update
* Release C updates
Feb 24th 2016
* Agenda:
* ACTIONS
[ACTION] ALL can you please review patches and abandon those that are no longer valid
[ACTION] All please notify Maryam or Al if you plan on attending hackfest
[ACTION] : Maryam to update Jira with new tickets for Rel C
[ACTION] : All Draft tests on etherpad please review and revert back to Maryam and Al with your feedback
[ACTION] : All please ensure your Jira tickets that have been completed are closed in JIRA
[ACTION] : Bill to advise on mailing list for Moongen
* Project Dev Updates
Documents now pushed to Master Branch
-
Draft test in etherpad, Maryam suggested that we should move them to gerrit and add to Jira
* Traffic Gen updates
Xena Update
Moognen Update
Success in getting 40gb supported, moongen now upgraded to DPDK 2.2
Started working on integrating into VSPERF infrastructure
Spirent Update
* ITEF-95 Update
Al updated Jira ticket deadline has been extended, Maryam to add some updates to the draft
look at possibility of doing remote presentation
* Release B updates
* Release C updates
* Please send on any agenda items you would like to discuss next week
Feb 17th 2016
* Agenda:
* ACTIONS
* Project Dev Updates
Martin has written patch that will automate test results to the Dashboard
Documents now pushed to Master Branch
-
-
Update from Sridhar patch for documenting addition of new test case is complete it now needs to be pushed to gerrit.
* Traffic Gen updates
Xena Update
Moognen Update
Spirent Update
newtestcase.rst, according to the comments in done. I’ll relook at this document next week to enhance it to cover other cases (other attributes of testcase) apart from the traffic generator.
Made initial commit on JIRA-VSPERF:237. Will continue to fine tune the scripts.
Planning to add scripts for multicast benchmarking (3918). Should create Jira ticket for the same soon.
* Rel B updates
Primary release date on target for Feb 25th
Only outstanding item, publish dashboard results
Received some good feedback, need to add definition for tests
Need to review unstable results, B2B frame test need to define what we are plotting
Rename scalability OVS/DPDK tests
* Hackfest
Held in SantaClara March 14-15th
who is attending ?
Al and Billy(Intel)will be attending
if we don't have a core list of attendees than might not be worth setting up offline discussions/demos
-
* Test priority list for release C
* Please send on any agenda items you would like to discuss next week
Feb 10th 2016
* Agenda
* ACTIONS
[ACTION] Maryam to investigate MPLS support in OVS.
[ACTION] Al to open a JIRA ticket to Update the Internet draft, etc.
{ACTION} Al to combine the VM2VM-related tests, then JIRA ticket Priority 1 for Rel C [DONE]
* IETF-95 Update
* Project Dev Updates
* Traffic Gen updates
Moongen Update
Spirent update
Xena Update:
Amit finishing up final integration to meet traffic gen
spec
-
* Rel B updates
Primary release date pushed to Feb 26th
Any project that is ready will go out on Feb 26th, projects that are not ready will go out March 28th
Documentation: still open question about the new templates proposed by Chris, room for some flexibility perhaps we can continue to use what we have for this release, as far as we know.
* Hackfest
Will be held in Santa Clara March 14-15th
Both Maryam and Al attending, would like to know who else is attending so we can plan demos and breakout sessions
-
It’s a good opportunity to get input from folks that don’t attend project meetings
* Test priority list for release C
Al added Jira tickets for new tests
When you combine the VM2VM-related tests, then Priority 1, need wiki update and JIRA ticket
Voting for Rel C tests are still welcome
If folks are happy with current top 4, please send a mail to Al confirming
-
New Draft Test at the Scratchpad (etherpad):
Overlay Network performance described in general for VXLAN, GRE, GENEVE
feedback that this approach works, saves time in review.
-
Feb 3rd 2016
* Agenda:
* ACTIONS
* Test Generator Update
Xena: Proving out implemented methods and working on/around Xena incompatibility with Python 3.
Spirent: Implemented changes to allow TestCenter to execute using the REST APIs, removing the need for a local client.
* General dev updates
* Test Priority discussion
* VSPERF results for Brahmaputra
* VM2VM discussion
Jan 27th 2016
* Agenda:
* ACTIONS
[ACTION] Maryam to post test results report
[ACTION] Maryam to post list of integration tests/tests from OVS developers to ether pad
[ACTION] Sridhar add a "how to add a new test case" document
-
* Moongen Overview
Notes taken during the presentation
Red Hat have integrated Moongen with an internal test framework. but mostly Bill's repo mirrors Moongen @ github, A detailed README exists on github
-
-
Today Moongen uses DPDK 1.7.1, it would be a "nice to have" to upgrade to DPDK 2.2 but this is up to the Moongen maintainers.
Bill can't run Moongen with DPDK 2.1 or DPDK 2.2, but this isn't an issue for us, we should just use Moongen as a standalone traffic generator
Examples/opnfv-vsperf.lua is a script that Andrew and Bill wrote modelled on other scripts in the example dir and it runs an RFC2544 test
The script does a binary search, to find the throughput value for the specified traffic loss %
Bill added a validation run to the script to ensure that the throughput value found in the binary search is indeed the x% frameloss throughput value
If validation fails the script continues to run the binary search until the validation phase passes
* Project dev updates
* Traffic Gen updates
Spirent update
Sridhar should be pushing some patches to vsperf will use the spirent REST
API.
Sridhar to follow up on Jira access with Aric
Sridhar has created a document for adding a new test case
Sridhar to migrate the ppt to rst and push to gerrit
* Test priority list for release C
* OPNFV Summit
Jan 20th 2016
* Agenda:
* ACTIONS
[ACTION] Bill Setup separate BlueJeans call for Moongen overview
[ACTION] Maryam to post traffic gen guide to Gerrit as a draft
[ACTION] Maryam to post list of integration tests from OVS developers to ether pad
[ACTION] Maryam and Sridhar add a "how to add a new test case" document
[ACTION] All to provide feedback to Al on Top 6 tests to start defining priorities for Release C
* Moongen Overview
Moved out to next week, due to presentation issue
Bill action to setup separate bluejeans session next week
* Project dev updates
* Traffic Gen updates
Spirent update
Brian has reviewed the documentation and provided feedback through email
Sridhar working on enabling b2b and other RFC2544 tests
Sridhar working on RFC3918 – implementation patches will be posted soon
Action: action add a "how to add a new test case" document – Sridhar and Maryam
Xena Update:
Action: Maryam to post traffic gen guide to gerrit as a draft
Action: Maryam To post list of integration tests/tests from OVS developers to ether pad
Xena developers working on burst and cont traffic methods… there are some missing return parameters that Xena needs to enable access to
* Test priority list for release C
* OPNFV Summit
Jan 13th 2016
* Agenda:
*ACTIONS
* Project dev updates
*Milestone E
-
Jan 26, All Docs complete, All tests will be conducted on the various Pods on stable Releases, till Feb 1.
* Test Prioritisation
* Test Generator Integration
Moongen integration with vsperf
Sridhar on Spirent Integration: Sridhar has written 2544 script, more coming
RFC 2544 b2b test implemented and under review
Traffic gen Xena: update from Rick good progress integrating RFC2544 test, next looking at Latency, jitter and frameloss testing, pick-up next Monday again
* OPNFV Summit
-
Jan 6th 2016
* Agenda:
* ACTIONS
[ACTION] Al & Brian have volunteered to review user Guides, and other docs (design docs and Release docs, ignore LTD Spec)
[ACTION] Pierre to help with upgrading firmware (
RFC 5481)
[ACTION] Maryam to provide link to list of tests for next release
[ACTION] AL to prioritise tests by working with community and work test
spec development
* Project dev updates
* Milestone E
-
Creation of Stable Release B Branch deadline tomorrow 7 Jan
-
Again, any questions please send to mailing list on Stable Branch work methods
Next Milestone, Jan 12, no changes allowed after this date
-
Need volunteers to review user Guides, and other docs (design docs and Release docs, ignore LTD Spec)- Al and Brian have agreed to complete this task see AR above
Review means read through and consistency checks, pick up JIRA tickets
Jan 26, All Docs complete, All tests will be conducted on the various Pods on stable Releases, till Feb 1.
* Test Generator Integration
New Epic on Moongen with Bill, he will come up with Documentation for Using MoonGen
New work on Dummy Traffic Gen will help with MoonGen and Xena
Documentation on how to integrate a traffic gen with vsperf will help with MoonGen and Xena integration.
Amit on Xena: implemented part of vsperf traffic gen, need to add connect disconnect
Sridhar on Spirent Integration:
RFC 2544 and RFC3849 scripts to support testing with Spirent
-
* OPNFV Summit
Minutes also available at https://wiki.opnfv.org/meetings/vsperf
Dec 2nd 2015
* Agenda:
* ACTIONS
[ACTION] Add vloop_vnf to documentation as the reference image to use with vsperf.
[ACTION] Maryam to establish an initial roadmap including this capability as a feature.
[ACTION] Maryam to establish an initial test priority list to review with the community.
[ACTION] Maryam to include path finding and integration activity for a reference pipeline to the roadmap.
-
[ACTION] Gene to migrate our commit gate checks and establish a basic Jenkins job that runs nightly on POD3.
* Project dev updates
* Traffic Gen updates
Xena: Amit mentioned that they’ve had couple of discussions about how to use existing xena code in vsperf. Amit has written small python scripts to perform individual tasks, Now the next step would be to use these small scripts in vsperf template. Amit is having a working session with Dan tomorrow that will hopefully result in having Xena take over the integration effort.
Moongen:
* CI update
* Inactive committers
* OPNFV Summit
* PDV to be collected in RFC5481 format and IXIA
Al has looked at them.
tcl script and a python file. it looks like the tcl script needs to be dropped into the right place within vsperf and tying in the python script into vsperf.
Maryam will look at integrating these scripts in the next sprint
Nov 18th 2015
Agenda
Rel B requirements for installers
Project Updates - Xena, Spirent, Moongen
DUT Topology discussion.
DUT Topology discussion
Rel B requirements
Dev updates
Can now use Spirent TestCentre traffic generator for RFC2544.
More tests/scripts to follow
[ACTION] Brian Castelli will add Jira tickets for these tasks
Still trying to get lic servers running on POD3
vnf_loop vnf
moongen
unlikely to be in a position to write vsperf integration code for Rel-B (unless help given)
RFC2544 with %loss is working in moongen itself.
no cost for licencsing or distribution of same (MIT lic.?)
[ACTION] Andrew Th. to verify licenseing requirements
moongen is distributed in src. May need to pkgd in order to deploy on distro.
Xena
hoping for eng resource to be allocated shortly.
[ACTION] billy put on agenda for 2wks from now for Ralph Daniels (Sprient) to present on slide deck (on vsperf ML) on OpenFlow vSwitch testing.
The slide deck for Ralph's presentation was sent to the mailing list on 2-Nov
AOB
Nov 11th 2015
Nov 4th 2015
Note no meeting November 11th
AGENDA for Next Meeting:
Discussion around unified/common denominator API for Traffic Gens
Q: In VSPERF, what is the purpose of the common template(Traffic Item +
API), is it to try to get the same results across the different traffic gens or something else?
A: The intention was to hide the implementation details and complexity in terms of parameters passed to the traffic gen behind a simple object that could describe the basic parameters (such as packet size, bi/uni directional traffic…) that we want to configure for a test. Today we pass a set of parameters that we want to modify to the generic traffic gen
API. They get translated into the appropriate parameter by the underlying traffic gen implementation and all other parameters are set to their defaults.
We agreed that we will document the implicit settings of Trafficgens supported by VSPERF in a separate document under docs.
We need to standardize the set of input and output to and from traffic gens – Added this to the VSPERF design summit and will bring a proposal to the weekly meeting after the summit.
Oct 28th 2015
-
-
[ACTION] Ram to send on what support/help is needed here to progress this issue/story. - DONE
[ACTION] Billy and Dan to send out list of open questions regarding traffic generation. - OPEN, Added to the Meeting agenda for the next meeting.
AGENDA for November 4th:
20 mins DUT Topology discussion.
20 mins Discussion around unified/common denominator
API for Traffic Gens.
10 mins project updates.
10 mins Deployment naming discussion.
new tests discussion continuation – next session.
Oct 14th 2015
-
-
-
[ACTION] Maryam to schedule 20 minutes for a discussion about new tests for the weekly meeting on the 21st of October.
[ACTION] Ram to send on what support/help is needed here to progress this issue/story.
TOPICs not discussed this week.
AGENDA for October 21st:
Oct 7th 2015
[ACTION] Maryam to invite Morgan to give an overview of the testing dashboard at the vsperf weekly. - DONE
-
-
[ACTION] Billy - mention on wiki need to get vpn creds from linxufoundation - DONE
-
[ACTION] Andrew, and Rick to add any traffic gen integration stories they would like to see as part of release B. - DONE
[ACTION] Maryam to schedule 20 minutes for a discussion about new tests for the weekly meeting on the 21st of October.
[ACTION] Maryam to check %loss in packet loss ratio tests.
JIRA and TOIT/VSPERF development update
Features under development
Traffic Gen integration update
Xena: Initial python development looking at the integration with VSPERF has been kicked off. But there seem to be missing params that could affect the results of a test. [ACTION] Billy and Rick to setup a session to discuss issues & questions.
Spirent: Brian has been integrating Spirent with VSPERF and has traffic passing through the DUT. Brian gave a demo of the integration efforts so far. Thanks and great progress Brian!
Moongen: MoonGen update (Andrew T): Bidirectional traffic is working, latency measurement has been added, however, latency stats for bidirectional traffic is incorrect (seeing negative numbers). There is also now support to specify the frame size for the test. Changes haven’t been committed to github yet, but be will soon. Also, you can now specify a %frame loss as a pass/fail threshold (default is 0).
Packet Re-ordering
Spirent is compliant with RFC5481
Need to check if IXIA is compliant with RFC5481 also.
We should enable this as a configuration option for VSPERF. [ACTION] Maryam to add JIRA issue for this (won’t be addressed till post release B).
* New tests
Sept 23rd 2015
General Actions
-
-
[ACTION] Maryam Place Packet Re-ordering on next weeks agenda.
[ACTION] Billy - mention on wiki need to get vpn creds from linxufoundation
-
[ACTION] Maryam to invite Morgan to give an overview of the testing dashboard at the vsperf weekly.
Spirent:
Brian C, RFC2544 running to completion. Needs some work to gather results.
[ACTION] Brian C, will investigate possibility of setting up Spirent Traffic Gen VM was on POD3.
Moongen:
bidirectional functionality enablement in progress
Via Bill M, Andrew still working on bidi functionality. Andrew will update next week.
Sept 15th 2015
[ACTION] Maryam to follow up with Trevor about setting up a server where Moongen can run connected back to back to the DUT – Done a board has been setup back to back with our server in POD3
[ACTION] Maryam to post wiki instructions on log in details to the new server.
-
-
[ACTION] Maryam to invite Morgan to give an overview of the testing dashboard at the vsperf weekly.
JIRA MILESTONE C 25th Sept 2015.
[ACTION] Maryam and Billy to add any generic/feature stories. - In progress
[ACTION] Gene to add any CI stories
[ACTION] Al to add any
spec update stories that we think we need to lock down for release B - In progress
[ACTION] Andrew, Brian and Rick to add any traffic gen integration stories they would like to see as part of release B.
Brian – DONE.
Maryam to send out JIRA instructions - Done sent out by Al, Thanks Al.
[ACTION] Folks make sure you have permissions to create and progress a ticket, if there are any issues, ping me and the linux helpdesk.
[ACTION] Maryam to update wiki with additional goals for Release B. - In progress
* CI Integration
CI Plan – Gene updated the wiki.
In terms of what kind of test we want to run as part of our commit gate, it looks like some verification in terms of pylint/ ReSTructured text validation as well as one of the simple performance tests. [ACTION] update JIRA with the tests we’d like to start running as part of our CI.
We need to figure out the triggers.
Need to run pylint on the codebase as is - [ACTION] Billy to submit JIRA ticket for this.
POD 3 update
Second server has been hooked up back to back with the server in POD 3 in preparation for Moongen installation.
[RECOGNITION] Thanks Michael and Trevor for completing this setup.
Will need some support to get moongen up and running on the second server.
Traffc gen VNF will also go here.
[ACTION] Maryam to update POD wiki with new server login info.
FYI Folks I’m out of office next week, as such Billy will be leading the meeting next week.
Sept 9th 2015
[ACTION] Maryam to follow up with Aric about JIRA emails/updated to committers and contributors Status: In Progress, Pinged the helpdesk. Issue of progressing an issue has been resolved for Gene, Al to try to progress a story before we close.
[ACTION] Al to updated the LTD to discuss sampling and details of PDV – Done.
[ACTION] Maryam to follow up with Trevor about setting up a server where moongen can run connected back to back to the DUT – In Progress We can setup a board back to back with the DUT, will be kicked off in the next couple of days.
-
-
JIRA
[ACTION] Maryam and Billy to add any generic/feature stories.
[ACTION] Gene to add any CI stories
[ACTION] Al to add any
spec update stories that we think we need to lock down for release B.
[ACTION] Andrew, Brian and Rick to add any traffic gen integration stories they would like to see as part of release B.
[ACTION] Maryam to update wiki with additional goals for Release B.
CI Integration
There was a missing link for the PVVP deployment scenario, L2FWD requires DNAT - → DONE – merged to master
CI Plan – Gene will publish soon.
* We should be able to hook up a server back to back with our DUT so that we can setup Moongen for the POD.
Sept 2nd 2015
[ACTION] Maryam to follow up with Aric about JIRA emails/updated to committers and contributors Status: Open, Pinged Aric, waiting for response. Will ping him again and CC Gene and Al
[ACTION] Al to updated the LTD to discuss sampling and details of PDV – In progress.
[ACTION] Maryam to follow up with Trevor about setting up a server where moongen can run connected back to back to the DUT - Open
-
-
[ACTION] Volunteer for Test Projects status update tomorrow – to give status update for vsperf. – Al Volunteered, Thank you Al.
Traffic Gen integration update
Xena:
RFC test was successfully deployed under Linux. [ACTION] Dan to give a demo next week.
Moongen:
Andrew tested 40Gb adapter and it initializes successfully but sw rate control for the 40Gb NIC still needs to be implemented for the adapter. Moongen typically uses HW/SW rate control.
Andrew experimented with the latency measurements but this has not been integrated with the RFC2544 script yet.
bidirectional traffic generation is also a feature that needs to be looked at.
Spirent: environment has been setup and ramp up on the scripts is in progress.
Gene suggested setting up a traffic gen working group for folks who are trying to integrate a traffic generator with VSPERF. If people are interested in participating please reply to the minutes [ACTION] Gene to send out some more info.
CI Integration
There was a missing link for the PVVP deployment scenario, L2FWD requires DNAT - → JIRA ticket added for this, code to be submitted soon.
The CI plan to be published soon and some commits not long after.
New tests
We need to start addressing some of the testing gaps in our test specification and look at new deployment scenarios and any other test categories that we need to develop tests for, either from a standard or from a relevant use case.
I implore the VSPERF community to start adding tests to the test_
spec and pushing to gerrit for review. [ACTION] VSPERF Community. Tests can be reviewed or discussed at the weekly call.
This is our current list of tests:
Throughput tests
Test ID: LTD.Throughput.RFC2544.PacketLossRatio
Test ID: LTD.Throughput.RFC2544.PacketLossRatioFrameModification
Test ID: LTD.Throughput.RFC2544.Profile
Test ID: LTD.Throughput.RFC2544.SystemRecoveryTime
Test ID: LTD.Throughput.RFC2544.BackToBackFrames
Test ID: LTD.Throughput.RFC2889.Soak
Test ID: LTD.Throughput.RFC2889.SoakFrameModification
Test ID: LTD.Throughput.RFC6201.ResetTime
Test ID: LTD.Throughput.RFC2889.MaxForwardingRate
Test ID: LTD.Throughput.RFC2889.ForwardPressure
Test ID: LTD.Throughput.RFC2889.AddressCachingCapacity [ACTION] Maryam add a JIRA ticket to define a new category for this test.
Test ID: LTD.Throughput.RFC2889.AddressLearningRate [ACTION] Maryam add a JIRA ticket to define a new category for this test.
Test ID: LTD.Throughput.RFC2889.ErrorFramesFiltering [ACTION] Maryam add a JIRA ticket to define a new category for this test.
Test ID: LTD.Throughput.RFC2889.BroadcastFrameForwarding [ACTION] Maryam add a JIRA ticket to define a new category for this test.
Packet Latency tests
Test ID: LTD.PacketLatency.InitialPacketProcessingLatency
Test ID: LTD.PacketDelayVariation.RFC3393.Soak
Scalability tests
Test ID: LTD.Scalability.RFC2544.0PacketLoss
Test ID: LTD.MemoryBandwidth.RFC2544.0PacketLoss.Scalability
Coupling between control path and datapath Tests
Test ID: LTD.CPDPCouplingFlowAddition
CPU and memory consumption
Test ID: LTD.CPU.RFC2544.0PacketLoss
August 26th 2015
[ACTION] Maryam to follow up with Aric about JIRA emails/updated to committers and contributors Status: Open, Pinged Aric, waiting for response. Will ping him again and CC Gene and Al
[ACTION] Al to updated the LTD to discuss sampling and details of PDV.
[ACTION] Maryam to setup a POD3 reservation wiki. Status: Closed.
* POD 3 Scheduling – Booking wiki Setup https://wiki.opnfv.org/wiki/pod3_booking_calendar
Only VSPERF contributors and committers should be requesting access to the POD to run vsperf tests.
The POD is not setup for performance testing right now, it’s a validation environment – will need to tweak BIOS and grub configuration as well as RAM on the system to set it up for performance.
August 19th 2015
Recognitions/Shout outs
Al Morton, Pierre Lynch, Bogdan Tenea , and Trevor Cooper for driving and supporting the RFC5481 PDV collection with IXIA.
Actions
[ACTION] Al update LTD and IETF draft with the correct terminology in terms of throughput/maximum forwarding rate for benchmarking methodology. Status: Closed,
[ACTION] Gene to share abstracts for OPNFV Summit/talk names – Status: Closed, abstract in the minutes below.
[ACTION] Maryam PVP Demo to be setup on POD 3 for people to be able to run – Status: Closed, Please see POD3 update below
[ACTION] Maryam to follow up with Aric about JIRA emails/updated to committers and contributors Status: Open
-
Brahmaputra target release date for February 2nd, 2016
Please review the backlog to see if there are stories that are missing that need to be added in.
August 12th 2015
Actions
-
[ACTION] Gene to share abstracts for OPNFV Summit/talk names.
PDV to be collected in RFC5481 format - investigate if it is possible to get the delay times for individual packets so we can calculate ourselves or it this is supported by traffic generators - Status: Open – See below
[ACTION] Maryam to get the model of the IXIA chassis connected to POD3 – Closed. Chassis is Optixia XM2, Version: IxNetwork 7.30.917.27 GA, IxOS 6.70.1050.16 GA .
[ACTION] Andrew Theurer to check availability for a Moongen Demo/readout – closed Demo/Readout August 12th 2015
[ACTION] Maryam PVP Demo to be setup on POD 3 for people to be able to run – In progress
August 5th 2015
Actions
TV to update Packet loss ratio tests acceptable loss % based on the test duration. Status: Closed.
-
Andrew (Xena) offered a readout on the Xena traffic generator –[ACTION] Billy to follow up - Open.
PDV to be collected in RFC5481 format - investigate if it is possible to get the delay times for individual packets so we can calculate ourselves or it this is supported by traffic generators - Status: Open
Al has been investigating this and it is possible with some IXIA equipment which have a HW capture buffer.
[ACTION] Maryam to get the model of the IXIA chassis connected to POD3.
Andrew Theurer joined in the discussion and mentioned that it is possible Moongen could also be extended/modified for this purpose. Andrew gave a good overview of the Red Hat contributions to Moongen and future work, please see Moongen update below.
[ACTION] Andrew Theurer to check availability for a Moongen Demo/readout.
Moongen
Moongen is a fully scriptable high-speed packet generator built on DPDK and LuaJIT. It can saturate a 10 Gbit/s connection with 64 byte packets on a single CPU core while executing user-provided Lua scripts for each packet. Multi-core support allows for even higher rates. It also features precise and accurate timestamping and rate control. (
https://github.com/emmericp/MoonGen)
Andrew Theurer mentioned that Red Hat are looking at modifying/extending it in an attempt to emulate/conform to the VSPERF LTD. This will lower the barrier of entry to VSPERF and is useful for prototyping/beta testing.
Andrew has been looking at, modifying and contributing to Moongen (modifying existing tests, and added an implementation for a 10Gb adapter, trying to enable a 40Gb adapter at the moment).
Red Hat are interested in implementing the VSPERF Spec in Moongen and integrating it with VSPERF.
IETF Draft LTD Summary
[Action] update LTD and IETF draft with the correct terminology in terms of throughput/maximum forwarding rate for benchmarking methodology – Al volunteered.
Put this on hold and try to implement the tests and getting results – to build a compelling story.
Should aim continually update the
spec in the IETF meeting cycles.
-
vnfs: Enable PVP using vhost-user – to be reworked
test_
spec: Add LTD.CPDPCouplingFlowAddition
test_
spec: Clarify LTD.CPU.RFC2544.0PacketLoss
test_
spec: Clarify LTD.Throughput.RFC2889.BroadcastFrameForwarding – Merged to master
src/dpdk: Enable building of vhost-user in src/dpdk. – merged to master
Support vswitch selection from the command line. – merged to master
Add a simple performance test that sends a continuous stream – merged to master
test_
spec: Clarify LTD.PacketLatency.InitialPacketProcessingLatency – Merged to master
Vanilla OVS support implementation – Merged to master
July 29th 2015
Actions
TV to update Packet loss ratio tests acceptable loss % based on the test duration. Status: open (commit permission issues)
-
PDV to be collected in RFC5481 format - Maryam to investigate if this it is possible to get the delay times for individual packets so we can calculate ourselves or it this is supported by traffic generators - Status: Open
Andrew (Xena) offered a readout on the Xena traffic generator – Billy to follow up - Open.
Rick to follow up with Dan on Moongen.
IETF Draft LTD Summary
Maryam & Al presented the draft to IETF on Thur 23rd (
https://wiki.opnfv.org/_media/wiki/benchmarking_vswitches_in_opnfv_1.6_aspect_4_3.pdf), presentation was very well received and the BMWG is intrested in seeing how our work progresses. We got some invaluable feedback from Scott Bradner regarding the terminology we are using when we refer to throughput and maximum forwarding rate for our soak tests. For Throughput you need to stop transmitting for a period of time to the DUT, then measure what the device is pushing. Whereas in the soak tests what we are measuring is the DUT max forwarding rate.
-
test_
spec: Clarify LTD.Throughput.RFC2889.BroadcastFrameForwarding
vnfs: Enable PVP using vhost-user
test_
spec: Clarify LTD.PacketLatency.InitialPacketProcessingLatency
src/dpdk: Enable building of vhost-user in src/dpdk.
test_
spec: Add LTD.CPDPCouplingFlowAddition
Support vswitch selection from the command line.
Add a simple performance test that sends a continuous stream
July 22nd 2015
Test definitions under review
Slowpath Latency & Fastpath Latency: good input from all contributers. Billy to submit clarified versions - Open.
PDV to be collected in RFC5481 format – Maryam to investigate if this it is possible to get the delay times for individual packets so we can calculate ourselves or it this is supported by traffic generators – Status: Open
Open discussion
IMIX traffic (IPv4, TCP and IPv6) and Mobility MIX in OPNFV – Investigation underway
Andrew (Xena) offered a readout on the Xena traffic generator - Open.
July 15th 2015
Actions From Last week
Billy to draft a new test for profiling packet loss. Status: closed, merged to master.
Add Test report section to every test case in the LTD and specify the fields that should be reported. We should also expand the LTD to report platform specific info that we should gather. Status: closed, merged to master.
Future action – Cover SDN flow controller tests. Add a Jira ticket for this, Maryam. Status: Closed, VSPERF-54.
TV to update Packet loss ratio tests acceptable loss % based on the test duration. Status: open
-
Test definitions under review:
PDV to be collected in RFC5481 format – Maryam to investigate if this it is possible to get the delay times for individual packets so we can calculate ourselves or it this is supported by traffic generators – Status: Open
July 8th 2015
Actions From Last week
-
Add Test report section to every test case in the LTD and specify the fields that should be reported. We should also expand the LTD to report platform specific info that we should gather. Status: In progress, Link:
https://gerrit.opnfv.org/gerrit/#/c/926/
Future action – Cover SDN flow controller tests. Add a Jira ticket for this, Maryam. Status: open, need to check if this is covered by Yardstick
TV to update Packet loss ratio tests acceptable loss % based on the test duration. Status: open
-
Test definitions under review:
PDV to be collected in RFC5481 format – Maryam to investigate if this it is possible to get the delay times for individual packets so we can calculate ourselves or it this is supported by traffic generators – Status: Open
Open discussion:
IMIX traffic (IPv4, TCP and IPv6) and Mobility MIX in OPNFV – Investigation underway
The LTD should specifically mention the tuples that should be used when matching a flow (variations include: (inport, outport), Network 5 tuple, Network 13 tuple) – Status: Closed – A flow classification entry has been added to the LTD.
July 1st 2015
-
Add Test report section to every test case in the LTD and specify the fields that should be reported. We should also expand the LTD to report platform specific info that we should gather. Status: In progress, Link:
https://gerrit.opnfv.org/gerrit/#/c/926/
Future action – cover SDN flow controller tests. Add a Jira ticket for this, Maryam. Status: open.
TV to update Packet loss ratio tests acceptable loss % based on the test duration. Status: open
-
TOIT development update
Features under development:
Back to back test ready to be pushed to gerrit.
Bidirectional functionality ready to be pushed to gerrit.
xunit parsing patch to be pushed to gerrit.
Frame modification tests – Have a working solution, needs some more testing before integrating Openflow rules into VSPERF.
Enabling OVS kernel Datapath (still on hold), pvp and pvvp deployments (still ironing out some issues.)
POD3 is being setup to allow folks to login and run VSPERF (CentOS 7 has been installed, we are just configuring the system in use for VSPERF).
Gurpreet update on integrating the Spirent traffic generator with VSPERF – Open.
IETF Draft LTD Summary
Al has pushed the latest version of the draft to gerrit – some final touches to be made.
Isolation and reporting should be included in the presentation.
Should be finalized Friday 03 July 2015.
DPDK 2.0 Build
Maryam will publish the basic build fixing patches → Status: Done, patches merged onto master
TV is to push any additional fixes – Status: Open.
Open discussion:
IMIX traffic (IPv4, TCP and IPv6) and Mobility MIX in OPNFV – Investigation underway
The LTD should specifically mention the tuples that should be used when matching a flow (variations include: (inport, outport), Network 5 tuple, Network 13 tuple) – Status: Open
June 24th 2015
Actions From Last week
Gurpreet to publish slides on “Overture vSwitch Performance” to the project wiki.
Gene to submit JIRA bug for relative paths issue with VSPERF. Maryam to take this as Gene is on holiday. Status: closed.
Maryam to update ltd to indicate RFC2889 address caching and learning tests are applicable to MAC learning switches only. Status: closed.
Future action – cover SDN flow controller tests. Add a Jira ticket for this, Maryam. Status: open.
Billy to draft a new test for profiling packet loss. Status: open.
TV to update Packet loss ratio tests acceptable loss % based on the test duration. Status: open
Al to publish Isolating Functions in the System Under Test on the wiki. Status: open
Add Test report section to every test case in the LTD and specify the fields that should be reported. We should also expand the LTD to report platform specific info that we should gather.
-
Update the meeting link or remove from the wiki, Status:Done.
TOIT development update
Features under development: Final testing of Back to back test is in progress, xunit parsing patch to be posted soon, frame modification tests (we’re still exploring how to implement this test), enabling OVS kernel Datapath (On hold), pvp and pvvp deployments (ironing out some issues.)
POD3 is being setup to allow folks to login and run VSPERF (encountered some remote control issues and we are working on resolving this).
Gurpreet update on POD11?
Gurpreet update on integrating the Spirent traffic generator with VSPERF – Gurpreet has ramped up on the code and gone throught the High level design document and it’s just implementation that remains.
Open discussion:
IMIX traffic (IPv4, TCP and IPv6) – we should expand the LTD to include IMIX traffic to tests as a traffic type.
The LTD should specifically mention the tuples that should be used when matching a flow (variations include: (inport, outport), Network 5 tuple, Network 13 tuple)
June 17th 2015
Actions From Last week
Gene to submit JIRA bug for relative paths issue with VSPERF. AR Maryam to take this as Gene is on holiday.
-
Maryam to investigate RFC2889 and whether it’s simple related to MAC learning. If it is, tests should indicate that they are only applicable to vswitches that support MAC learning. → Update ltd to indicate they are applicable to MAC learning switches.
Investigation CLOSED, the 2 tests related to address caching and address learning are related to MAC learning frames.
OPEN: Maryam to update test cases with the relevant comments.
Future action – cover SDN flow controller tests. Add a Jira ticket for this, Maryam
TOIT development update
Features under development: Back to back tests, frame modification tests, enabling OVS kernel Datapath, pvp and pvvp deployments.
POD3 is being setup to allow folks to login and run VSPERF.
Gurpreet mentioned we might be able to also get setup on POD11.
Gurpreet is looking at integrating the Spirent traffic generator with VSPERF.
Ideally we want to find a POD where we are connected to all the VSPERF supported traffic generators.
IETF Draft LTD Summary
Really good progress being made here, Al added in a 3x4 matrix within which we can track the coverage of our tests and identify gaps.
The draft has been merged to master.
DPDK 2.0 Build
TV is making good progress on this front.
validated the suggested fixes from Maryam.
currently has connectivity issues with gerrit and can’t push fixes. ACTION TV to contact opnfv-helpdesk@rt.linuxfoundation.org
* Readout
Isolating Functions in the System Under Test: Al led a discussion on how we can try to get consistent test results without too much variation.
ACTION: Slides to be published on the wiki.
Action Maryam work with Al on making additions to the diagrams.
June 10th 2015
VSPERF (TOIT) Update
Code has been +2’d and merged to master.
We are looking at getting POD3 in the Portland Lab setup for folks to be able to go in and run VSPERF and provide feedback on the framework.
We need to investigate if we can setup a test bed in another OPNFV lab that has Spirent equipment. Gene mentioned that the Huawei Lab could be a suitable candidate.
Gene is looking at integrating a SW packet generator with VSPERF. Gurpreet mentioned that Spirent could do the same.
IETF Summary LTD Draft
Good progress being made here, Al is going to push a revised version of the Draft.
We need to add in to our
spec and this draft how we plan on isolating our measurements so that the results of tests can be repeatable.
We need to put some slides together for the Next IETF meeting and socialize our work in order to get it adopted as a working group draft.
Actions
Gene to submit JIRA bug for relative paths issue with VSPERF.
-
Maryam to investigate RFC2889 and whether it’s simple related to MAC learning. If it is, tests should indicate that they are only applicable to vswitches that support MAC learning.
Maryam to look into how we can achieve/implement the
RFC 2889 tests.
Billy to draft a new test for profiling packet loss.
Packet loss ratio tests
Tests that run for a short period of time should be updated to test 0% packet loss.
Tests that run for a long period of time (soak tests) should have the 10^-7% packet loss ratio.
A new test to be drafted to give a packet loss profile for a vSwitch.
June 3rd 2015
May 27th 2015
AOB
Gene Snider asked if TOIT would support other traffic generators. Billy said this was a core requirement of TOIT.
Gurpreet asked for some time to present some information of VNF testing at the next meeting.
[AR] Gurpreet to contact Billy to agree presentation duration.
May 20th 2015
Maryam is out of office till June 8th, 2015 Billy O Mahony is providing cover on VSPERF
May 13th 2015
Open Gerrit Reviews
TestSpec: Add LTD.Latency.Slowpath
testspec: cleanup + update tests with new metrics
TestSpec: Add RFC2889 Broadcast Frame Latency test
TestSpec: Modify soak tests to measure PDV
TestSpec: Modify soak tests to measure PDV
TestSpec: Modify LTD access permissions
I encourage VSPERF committers and contributors to review these and provide their feedback.
Thanks Sothy for the very informative readout on the Performance of OVDK + port and flow mirroring.
May 6th 2015
TOIT Update
1st development stream (Implementing the stubbed out skeleton framework) Almost complete
2nd development stream (porting the existing low level components like vswitchs and vnfs… to the new directory structure while maintaining functionality/the ability to run tests) – only one component left.
3rd development stream (path-finding porting the existing framework from python 2.7 to 3.x while maintaining the ability to run tests etc…) – Done.
LTD Summary for IETF
Al Morton gave an update on his investigation for a tool to convert markdown to XML, but for all the simplicity markdown gives you it adds the complexity of all sorts of tools and converters which outweigh its benefit.
Going to stick to XML for the IETF draft summary.
The summary will serve as an informational tool to show/indicate what we’ve been doing with the IETF RFCs, which is selecting the RFCs that are most relevant to performance testing of a virtual switch. These RFCs include:
Actions:
Which is the most appropriate/applicable for our case?
I Suggest we review LTD.Throughput.RFC2544.SoakFrameModification as the first item for discussion.
April 29th 2015
Attendees: Eric Dubé, Maryam Tahhan, Al Morton, Billy O’Mahony, TV Rao, gsingh, Dave Neary
Issues with GotoMeeting resulted in a meeting over IRC. So only a couple of topics covered this Week.
Action Items
April 22nd 2015
Open Questions
LTD Summary for IETF:
Are we at a point where we can start this?
How do we get started?
What’s the best way to do this? My suggestion would be etherpad, or a draft patch in gerrit that we can work on, Any recommendations for a way of working on this?
vloop_vnf
Does it bypass the network stack in the guest?
Are we concerned that the network stack is a bottle neck?
Does code exist for this? If so can we push to gerrit for review?
Is it a proprietary VNF, or is it something we plan on developing?
Action Items
Project Update
General discussion
The generated benchmark reports need to document the minute details of the system setup including the vNIC version as performance can vary greatly between different versions.
Spirent’s virtual traffic generator supports
RFC testing – great enabling tool, but important to note we can’t get the same performance measurement accuracy as the HW generator.
Need to keep the p→v use case that bypasses the vswitch (pNIC to vNIC) on our minds also.
April 8th 2015
Project Update
TOIT:
We had a quick walkthrough the TOIT High Level Design (HLD) document.
-
Work is continuing on the HLD to define the interfaces and the various classes, this will be published shortly for review.
Merged Tests to test
spec:
Throughput.RFC2889.ForwardingRate
Throughput.RFC2889.ForwardPressure
Throughput.RFC2889.AddressCachingCapacity
Throughput.RFC2889.AddressLearningRate
Throughput.RFC2889.ErrorFramesFiltering
Throughput.RFC2889.BroadcastFrameForwarding
Opens:
Need some more clarification on [JIRA] (VSPERF-39) Add simple L2 forwarding module
Item for Discussion: How we can advertise our work to other benchmarking communities (ETSI_NFV, IETF Benchmarking Methodology Group)?
A new regular slot to feature in the weekly meeting going forward, this slot will be focused around what else is going on in the benchmarking community (ETSI_NFV, IETF Benchmarking Methodology Group). Al Morton kindly agreed to lead the discussions on this.
Suggested agenda Items for next week:
Eugene Snider to lead a discussion around a plug n play packet generator.
Clarification on [JIRA] (VSPERF-39) Add simple L2 forwarding module
Item for Discussion: How we can advertise our work to other benchmarking communities (ETSI_NFV, IETF Benchmarking Methodology Group)?
Patch submission process walkthrough.
March 25th 2015
Project Update
Merged Tests:
Throughput.RFC2544.SystemRecoveryTime
Throughput.RFC2544.BackToBackFrames
Throughput.RFC2544.ResetTime
Throughput.RFC2544.Soak
Throughput.RFC2544.SoakFrameModification
March 11th 2015
Project Update
New Project Topic and email tag: VSPERF. (based on teh JIRA project)
2 tests pushed to master:
Three tests have been rebased and open for review:
TOIT
Vote on promoting Al Morton from contributor to committer has passed.
Branching Models:
CI validation
JIRA:
Feb/11/2015
Mailing List
TOIT
Code is being cleaned up and prep'ed for release
Slides to be sent out by Stephen
Feedback given that tests should have a configurable duration.
Admin/Infrastructure point of contact for the project: Maryam Tahhan
Patch reviews
One +1 for the intro. Need another committer to +1 to progress that test
spec section to the next phase
Review cycle to be reduced to 2 days rather than 3 days
Features to be tested patch rework started today by Maryam Tahhan
-
Patch reviews and rework should be conducted in the order specified in the table (top to bottom).
Project Leader
Feb/04/2015
Minutes:
Project updates:
Initial code reviews for test
spec are open. The test
spec has been broken down into sections and committed to gerrit for review.
-
-
Tools:
Opens:
General Wiki updates:
Coding Guidelines : OPNFV Coding Guidelines
OPNFV Developer Resource Page : Getting started with your account, Gerrit and Git
HW Spec (specified above)
Open Gerrit reviews (specified above)
Governance model wording (specified above)
-
Jan/21/2015
Jan/14/2015
Attendees: Approx. 25 attendees
Notes:
Opens
[Action]: Mike to put in place
IRC channel for next week’s meeting
Where do we store artifacts like VM images? Consensus that they really do not belong in GIT….
[Action]: Mike to raise this at Performance and Test group meeting on how to resolve as it won’t be a problem specific to this project.
Raised today (Jan 15th); P&G group have taken an action to make a proposal on how to handle this project-wide
Project Development Tools – what candidates?
Code review: Gerrit… no objections
Bug tracking:
Could use GitHub bug tracker (assuming we are building on GitHub – need to confirm with Linux Foundation) Alternative could be to use Bugzilla.
Bryan Sullivan has experience of GitHub Bug tracker – seems to be pretty effective +1
Bugzilla is acceptable fallback if GitHub bug tracker not available (Bryan again).
Need to ensure we can enforce a mechanism that makes it clearly visible what bugs are being fixed (Gene Snider)
Ideally we should not re-invent the wheel, let’s use as much default from Linux Foundation as possible, unless consensus exists that it’s not acceptable
Test Specification draft
Maryam presented the structure of the test
spec
Bryan: is there anything there from QoS perspective? No….Need to get feedback from community here.
Al Morton (AT&T): Happy to contribute to some parts of this. Mike to add Al as a contributor [done]
Al to send out links to some IETF benchmarking/performance RFCs and drafts [done]
Governance structure
We need to agree a Governance and Review structure!
We are doing some trailblazing here from OPNFV perspective which may serve the benefit of the whole community
[Action]: Maryam to provide a detailed proposal to the mailing list on a Governance Model
