Minutes for Thursday g-API session

Edited by Lingli Deng based on notes taken by Bob Monkman and feedback from Bill Fischofer and Keith Wiles

Bob Monkman Bill Fischofer Keith Wiles Fu Qiao (CM) Srini Addepalli (FreeScale) Subha Venkataraman (Freescale) Bharat Mota (Freescale) Yu Chang (Huawei) Abdel rabi (Altera) Hu Yun Chao (Huawei) Zhou Nan (Huawei) Kin-Yip Liu (Cavium) Brian Skerry (intel) Prakash Kanthi (DataTap Solutions)

Session was delayed until 3:40 and so we had only slightly less than 2 hours. Skipped High level requirements to go over 2 presentations:

1. ODP overview and how it relates to g-API from ODP team perspective.
2. Keith’s updated DPDK perspective g-API presentation

We had presentation, some comments and questions in discussion afterwards.

It is suggested that ODP was created from the start to have precisely the same goal as g-API. Application portability with leverage of differentiated accelerators for different vendors

It is emphasized as the ODP team considers it fundamentally important that representative competitors work together to arrive at the g-API recommendations in the end. And the case for ODP is pretty promising. 30 months in the making and key aspect of ODP is that competitors and partners alike, from OEMs (Cisco, Nokia, ZTE, Huawei), SoC vendors (Applied Micro, AMD, Broadcom, Cavium, EZChip, Freescale, Kalray) and ISVs such as MontaVista, Enea worked in concert to define and agree the APIs, create implementations and a set of 10+ demos over time. Even Intel has been represented, first from OEMs who have a stake in DPDK and Intel Architecture solutions, and now through Wind River just ramping up participation thru acquisition.

Question came up as to whether ODP-DPDK could be leveraged for its virtio support to enable the proposed virtio option for binary compatibility across a given Instruction Set Architecture.

• Bill answered that, in theory, it could be, although this is not tested- compiled or ported to different CPU archs. "Today we have PowerPC, MIPS and a ARM port to the Gene-X platform. We do not have access to the SoC vendor acceleration hardware since none of the SDK’s are open, but the changes would be very small."

There was a discussion raised by Keith on the fact that ODP layer is open source, but few SoC implementations are published on the opendataplane.org site.

• Bob noted that this is does not have to be the case, but since unlike DPDK, which is in its core software-centric, the SoC specific implementations leverage hardware that is unique to every SoC and may reveal sensitive silicon details if fully published.
• Keith clarified that DPDK is not purely software-centric as the design allows for leveraging hardware as it does support acceleration of crypto, compression, classification/steering etc. has its own hardware designs for crypto, compression and switching. The API caller may choose to one of multiple implementations – either hardware or software. "As an example, we have crypto that is accelerated both via instructions as well as via offload accelerators – whether they be QAT or Nitrox. The only reason we have software designs is the power capability of the IA core allows us to perform some operations in software equally effectively without having to use external hardware. We don’t assume that the platform has hardware accelerators – but we can leverage them if they are there. Adding more support to DPDK to leverage things like external memory and event systems is a long term goal."
• Kin-Yip argued that it was not necessarily mandatory that implementations not be published. Matter of choice.
• Keith opined that it should be possible to publish considerably more without revealing proprietary information (paraphrased).
• Bill remarked that releasing the SDK source should not be an IP (Intellectual Property) problem. Because SDK software is not the secret sauce or IP of the product, the hardware is. “But they are allowed to release whatever source code they wish as no one can force them into doing anything. For a while, there were a few poll-mode drivers on DPDK that were not public, or were binary only. So secret sauce, even if it is in software can be accommodated."

It was presented that ODP team had in very simple cases reduced ODP-DPDK overhead to “negligible levels”, as is already the case for several SoC implementations. Also, there is a stated goal to ensure this is less than 2% for most cases and PoCs were in development to examine ODP performance with real world use cases.

• Keith is concerned that this proposal would suffer a performance hit and break compatiblity for all of the current DPDK customers, as would be required to rewrite their VNF applications to address the portability issues. “I would expect most would continue down the current DPDK path and then we have a fragmented world. Having ODP/SoC SDK at the driver layer only makes sense as this is the correct location for these types of drivers plus not effecting the performance of DPDK applications. If the ODP layer is truly not effecting performance then running ODP as a PMD would not effect the SoC performance as well.” *

The first slide was a rework of the earlier agreed VNF Application in Guest from OPNFV mini summit at NFV WC。

• Mention of ODP was removed and Abdel Rabi of Altera expressed confusion by this.
• Keith clarified that this is just a DPDK perspective and does not imply that ODP or other Acceleration Core were not supported. "Later in the slides I would show where ODP/SoC SDK’s would be addressed."
• Bob commented that the optics of this positioning creates controversy however and it would be better to show project drawings like this as they were agreed to not exclude candidate ACs.

Keith showed a new slide emphasizing their desire to preserve DPDK investment by those who had created integrations with DPDK.

• Clarification from Huawei was asked regarding statement that DPDK supports hardware acceleration with software acceleration option.
  • Keith responded that "with high performing cores today, it would be easier to use software as the setup time involved in handing the data off to hardware would/could be to high compared to software just doing the work. On lesser performing cores then hardware would be required as the operations in software would be to slow otherwise."
  • There was not necessarily agreement on this. Note: This would seem to be counter to NFV PoC #19 where the Intel CPU + Axxia SoC IPSec and DPI acceleration showed 10x performance gains in workflow with 90%+ reduction in jitter.
  • Keith suggested that this be left for further clarification from Magnus later. "Maybe Magnus can address this one as he was the designer of the software for the Axxia product."

There was new concrete proposal for folding certain ODP support into DPDK. "Extending DPDK with ODP features gives us the best of both worlds!"

1. Configuration: Combine the two configuration models
   1. DPDK supports only PCI configuration model
   2. ODP supports only SoC configuration model
2. Memory manager: Combine the two memory manager models
   1. ODP supports a external memory manager for SoC controlled memory
   2. DPDK support a high performance memory manager for devices without memory managers
3. Run-to-Completion and Event based model: Combine the two run models
   1. DPDK uses Run-to-Completion for high performance
   2. ODP uses Event-based model for programming model

"The phase one design would be to have ODP as a set of PMDs to allow Intel, MIPS, PowerPC and ARM systems to use any configuration they so desire to meet their system goals."

• Bob commented that the proposed SoC model in DPDK, whereby SoCs are implemented as PMD drivers (including using ODP APIs is desired) seems to relegate SoC to plug-in NIC cards with Intel x86 Host CPUs.
• Keith clarified that for VNF applications running on IA systems it does mean these SoC’s would be plugin cards, but in systems like ARM platforms with the SoC integrated on the board then NO it does not need to be a plugin card to that system. "The goal is to be able to run DPDK on any platform and unify the world by addressing the above issues in DPDK."
• Bob feared there was a fundamental lack of understanding the ODP is not a driver abstraction interface alone. "It is an application interface that is its full mode significantly contributes to a scalable many-core application scheduling model. ODP as a PMD does not really make sense and SoCs as PMD are viewed as perhaps applicable in only one use case, not all. This is a point of contention that would need further study, but the ODP team is convinced this would add unnecessary overhead. But ODP could and probably should add a NIC interface concept that is compatible with PMD, for external NICs at least."
• Keith responded that it seemed his proposal would apply to a number of cases. And DPDK is already able to run multiple devices at the same time, plus SoC device if needed. "I understand the ODP is not a driver here and I agree it will never be a driver. What ODP is today is an abstraction layer for any given SoC vendor SDK. By utilizing ODP as a abstraction layer for a number of SoC’s it allows DPDK to support any SoC using a PMD that understands the ODP API and abstraction."

DPDK and ODP teams can assess the viability of supporting both memory manager models in their respective projects.

It was agreed that data plane applications are generally RTC fundamentally.

• Bob commented on RTC model v Event-based Model. He clarified that ODP supported a simple polled mode operation, without queue/event structure to be close to DPDK model if desired _and_ an event-based advanced model.
• Keith stated that DPDK currently uses the RTC model only, because "DPDK started with IA systems where a single core is sufficient for a single task, and it does not make sense to use many cores to handle a single task with an event model reduces the performance. If DPDK were to be run on a different system with lower performance cores then it would be reasonable to have an event based system and the desire to add this support to DPDK."

Keith wrapped up proposed concrete next steps of adding support in DPDK for the above features and requested help with a standard DPDK implementation for ARM.

It was agreed that the only way forward was to produce prototypes examining these sorts of options. It was also suggested that the DPDK and ODP teams really need to directly discuss and explore convergence in general.

Bob confirmed the ODP is adding better DPDK support in ODP and will likewise produce some prototype/POCs for assessment of the project. This can be considered part of the Implementation phase where proposals are presented for meeting the requirements of DPACC and the group can evaluate and go from there.

Keith suggested rather than working with DPDK as the SDK to ODP, the team should focus on adding the missing parts to DPDK to enable the ODP/SoC vendors.

• "To be frank, I do not see the need to continue the work with DPDK as the SDK to ODP. This model is not going to be acceptable as too many DPDK users would be required to change and would effect performance on IA platforms. Even if ODP met the 2% performance goal it would still be a hard sell to our customer who already get 100% performance for their applications."
• "With that stated, I would suggest we focus on adding the missing parts to DPDK to enable the ODP/SoC vendors. Meaning we need a port of DPDK to an ARM SoC platform we can all agree on plus we can agree the port is correctly done to not effect ARM/SoC performance. Using the ODP PMD layer will allow the SoC vendors to keep their IP private and still produce an open source solution."
• "We can then move forward in the DPACC with a common goal for the VNF applications."