• Key points
  • General APIs are the ultimate goal for this project, but we need to figure out first the layering first, and then dive into the APIs between different layers.
  • In the same time, the general APIs to top of the SAL is considered to be agnostic to the framework design and encouraged to be worked on in parallel with framework layering.
• Raw recordings:

Fahd: The two framework proposals (SAL or AAL) are not so different from the API model perspective. These frameworks are different implementations of the general APIs. If the API is well-defined, it should not matter what the underlying framework proposal is. Do we want to support multiple frameworks? Or we want to have one unified framework? What is the goal, to have the API defined? Is possible to allow other people also work on the framework for the APIs this group defined?

Keith: I tend to agree that the two documents are close. Just that my approach is more standardized approach, by using Virtio etc. instead of trying to create new APIs. And see if that it is going to work for us, and define what the layers are going to be. But the thing I was trying to cover in that document is to define what the layers are and what the usecases are, from the acceleration/NFV perspective. And I see you are defining the APIs. The APIs to me are some of the last things we need to do. But we all have to agree on what set of components are to this thing, and how they layer together. If we don't do that, we'll have to many different things coming up and people see the APIs dont' synchronize the idea/philosophy we are trying to push forward. The document of AAL seems more implementation detail, while mine is more like how the layers are. I think you are right, but I believe we need to think a little bit higher level than just APIs. APIs are good, but we also need to understand what the APIs we need, and what the APIs in between.

Fahd: Exactly. There are some APIs that will be exposed to NFV developers, right? Those are the APIs you want to document, and don't need implemenation detail, whether you are using AAL via SR-IOV or using dpdk. I guess my question is, if you define the APIs well enough, the underlying framework should be agnostic, whether is dpdk or not.

Keith: That is definitely the goal, to define such APIs to be agnostic to the software layers underneath. (lost connection)

Fahd: The other thing that you do not addressed is that different models seem to have different requirements on API designs, like the flow programming feature of accelerators. (lost connection)

Keith: We cannot specify API that is hardware centric. It has to be generic enough. And then we'll gonna think what is beneath it. There are different sets of APIs, crypto, dpi, etc. E.g. for crypto API, where it sits, how do people get access to it. What are the generic things we need to do with that crypto API. We can go off and find what the specific APIs we need. Same thing with the flow APIs. We have to stay high.

Fahd: Yes, I agree. In the higher layer, for flow API, I think you can say this is a flow API for IPsec, a flow API for SSL offload.

Keith: I think we are using flow for different meaning. For me, I use flow API to refer to the flow offloading/routing.

Fahd: That is what I mean, depending on what you think a flow API contains, whether or not the IPsec flow is completely offloaded. You have some IPsec APIs that addressing this flow programming model. And you also have other IPsec related APIs doing only crypto/decrypo stuff.

Keith: So, there is look-aside and there is inline accelerators. You are right, those are needed to be included to the API set, what it is, and what it looks like. If you looking at the latest slides I sent before the meeting, I am trying to define what the layers are. On Page 3, we are trying to define what virtio way is, what the software layers are, and then look at the APIs between them. On Page 4, the SAL in the VM is added. If we can define them and make them common, then we can swap out the components for something different. What is in them, is a separate problem, meaning if you would like to do inline or look-aside crypto, that is the implementation detail within the crypto API, not necessarily within this layering scheme, which is much higher layer.

• Key points:
  • The layering picture is meant to capture all the usecases possible, and some of the layers are acutally optional for specific usecases.
    • For instance, SAL in the VM might be optional for the generic application which has only virtio as its underneath path to the hardware device.
    • For the VM using SR-IOV to access the hardware/external switch directly, the vSwtich layer in the host is optional.
  • vHost-User is a user space implementation from dpdk and meant to bypass the kernel entirely.
  • Bottom SAL is representing dpdk/odp like interface and is meant for abstracting the interface between vHost-User to specific hardware drivers.
  • Remaining concerns about the necessity for the bottom SAL underneath vHost-User.
    • Further details needed by taking crypto as an example to decide the relationship between vHost-User and bottom SAL.
    • Subha tends to believe it should be the vendor-specific drivers under the vHost-User rather than the bottom SAL.
  • The current layering covers only the data path, the mgmt features of AAL proposal needs to be included for a complete framework.
    • Keith has concerns about AAL's API for VMs to probe device and link it up. Clarifications made by Ferran that AAL does not.
    • Further offline discussion and online clarifications of related work flow and interfaces are suggested.
  • To move on:
    • The framwork pictures needs to be redrawed to better illustrate different layering for specific usecases.
    • Keith and Ferran needs to have offline discussion to come up with an agreed layering, which include mgmt plane as well.
    • Concrete example, such as crypto, might be necessary to further explore the rationale of the layering design.
• Raw recordings:

Subha: Question for Slide 3. It is stated that "no changes required in the VM to support acceleration". I believe virtio has to be enhanced for acceleration.

Keith: That statement only meant for network traffic.

Fahd: On Slide 4, with SAL, you have defined underneath the VNF and above the virtio seems to abstract everything underneath, that I think probably everybody is in an agreement that it needs to be there. I guess that one that is in the bottom, is the one that I am not sure that I completely understand yet. The one above the device.

Keith: In pictures that we don't show here, there is a case where some accelerated applications sitting on top of virtio++ without a SAL in the VM. Therefore, you relegate those pieces of acceleration in the host, which is in the SAL below it.

Subha: What is the relationship between SAL and DPDK/ODP?

Keith: SAL is a term I created to not just say DPDK or ODP.

Subha: Is it the same kind of functionality?

Keith: Correct. It is basically an abstraction. Look at ODP or DPDK, they are all trying to abstract the hardware and make the application as generic as possible. In this particular case, the bottom SAL is mostly dealing with the vSwitch and possibly accelerating applications in the user space. But the VMs themselves, they need something as well. And that also helps to abstract hardware (e.g. the SAL can directly access the hardware via SR-IOV or some other kind of mechanism). It is not required to have SR-IOV like mechanism to access the hardware directly, we need to make sure virtio supports everything as well for the class VM #2 in the picture.

Lingli: Do we have consensus over the proposed layering framework in this picture? Any further comments or questions for Keith to clarify?

Saikrishna: According to HW's proposal, there is also a management capability. I think they has to come together to form a complete framework.

Keith: One of the philosophy that I believe is that the NFV needs to export its needs, not necessarily in terms of "I need this type of hardware, I need that type of hardware.", but in general terms that "I need hardware for crypto acceleration or acceleration of some type" , not that "I need to plug up some type of device", which is really the functionality of orchestrator or the VIM within the system. Is that what you meant?

Saikrishna: (lost connection) The assumption is that there is the accelerator and there is mechanism to communicate to it.

Keith: Yes, this is what we discussed over one of the earlier meetings. Those need to be included in and I could not put it all on one slide. We need to come up with how it works. It seems to me that AAL has more specific knowledge than is needed. And I am worried that they really should only be metadata. They should not be going out and probing things. It just should be coming down and say "hey, this is what I need", and underneath it, in the VM layer something comes out and goes out to fix it all and get it all set up, based upon the orchestration's requests.

Subha: I have the same question. For the dynamica programmable accelerators, there is no need to probe the device before the allocation.

Keith: From what I understand, people want to have dynamic devices. That's a good goal. I don't want the VNF to find it and link it in. That is not his job. Let's try the static example first, and then add dynamic ones. Seaward: There is two layers of SAL for VM#4 in this picture. Do you mean that the upper layer of SAL does not work?

Keith: Well, it does not mean it does not work. It means if the accelerated application wants to get direct access to specific device (not sharing with other VMs) through that SAL in the VM, but everything else may go down the standard io interface and use the bottom SAL.

Seaward: Another question. In the previous meeting, you mentioned that virtio is expected to be extended for the requirements of application API? But according to the current spec of virtio, I see virtio is only for bus-based device types, not for other device types?

Keith: Virtio is a hardware specification. But the way we implement it with vHost-User, it is somewhat a software specification now, right? We are not talking directly to the hardware anymore, we only talk to the vHost implementation of virtio, and trying to expose those pieces to the VM. So you could represent different device ids to support the other standard devices. Maybe it turns out that virtio is not the answer, but I don't want to do something completely different without first investigating virtio to make sure it cannot work for us. If it can not, maybe we should start working on another piece of software that sits besides virtio. And extend virtio or other software to include some features of AAL. There are good features in it, but I don't think all are neccessary for what we need to define.

Fahd: Keith, can you help me understand what piece of AAL are of concern to you? Keith: Well, the ones VNFs have to going out and probing devices, trying to search for the device and attach to it. That does not seem to be appropriate. Right now, I cannot think of all the pieces in the AAL. I have to read it again.

Fahd: Yes. I think we need to established a concensus over whether we should support that capability or not. That VMs can probe the device or not.

Keith: I have already stated that VMs should not be probing the devices. It really should state its requirements interms of acceleration capability (e.g. throughput) and the orchestration to figure out whether or not it can support those requirements.

Saikrishna: I think maybe we need to go through HW's proposal one more time. In my understanding, there is no such requirement for the VMs to probe devices.

Keith: I agree I need to reread it again. (lost connection)

Unknown: Do you have any usecases where the VNFs don't request for specific device for acceleration and have the orchestrator to settle down which to use?

Keith: Take crypto for example, the device a VM needs for crypto, all it needs is data encrypted or decrypted, (lost connection).

(Ferran makes clarifications about AAL is not proposing device probing from the VMs, Keith needs further reading to better understand the proposal.)

Keith: It sounds like there is not much difference between the two proposals. We need to come up with a single picture that we both agree on and then figure out if that will lead us forward.

Subha: One question with this picture. If you look at VM2 and Vm3, it looks like that they are using the vSwitch on top of the bottom SAL. Is that a must?

Keith: The vSwitch is opitonal. It does not have to be there. It is for VM2VM traffic acceleration, unless you have local/external device.

Subha: You think the vHost-User should be able to talk directly to the device. Is that correct?

Keith: The vHost-User does not talk directly the device, it can talk to the SAL layer. I believe that the SAL layer has to be there to abstract the hardware-software relationship.

Subha: Are you talking about dpdk/odp kind of like interface?

Keith: Yes.

Subha: Why would that be needed?

Keith: You could buid-in the knowledge to the vHost-User to be able to access the hardware/software as you like, but I am trying to separate software into layers that could create abstractions at those layer boundaries so that we don't have to make vHost-User also know about hardware. We can just make it know about an API of a vSwitch or SAL, and then the SAL knows about all the specific hardware. You can also combine the SAL and vSwitch into one box, it is actually two conceptually two boxes. It is probably that vHost might melt into one. But I want to make us keep the distinction from the logical point of view that they are separate, so that we can abstract APIs between them.

Subha: I think that under vHost-User, it should be the vendor-specific driver come into the picture. (lost connection)

Fahd: External switch case seems not included in the proposed layering scheme?

Keith: Is that VM1 or VM0? For VM3, it can also pass through the vSwitch and access external switch as well. We have to account for it in a generic system.

Fhad: Can you modify the diagram for difference devices?

Keith: Yes. (switching to the latest slides deck for mid-layer considerations, Slide 4)

Seaward: In this picture, the vHost-User seems to be the backend driver for virtio, right?

Keith: Correct.

Seaward: But to my understanding, backend drivers usually are located in the kernerl space, rather than user space.

Keith: Yes, the standard vHost is in the kernel. And this is an implemenation from dpdk that sits in user space. Some people call it vHost-User.

Seaward: But you still need something in the kernel to do the similar things?

Keith: No, we don't want to touch the kernel. We want to bypass the kernel at all.

Subha: vHost is part of the linux distribution, virtio may be able to send all the information to the vHost-User process.

Keith: I recall somebody suggest that the bottom SAL can be eliminated. Although it is possbile, but I strongly disagree it can be, if you would like to maintain a software-hardware abstraction.

Subha: I think more detail of the layers might help, if you take the crypto example, to walk across the API requirements at each layer. In that way, it will give you some concrete ideas of what we need to do. From this point, it would be much easier to figure out what the vHost-User should be doing, and what the SAL should be doing.

Keith: Yes, once we all agree that these are the layers for the generic frameowrk, we can start drilling down into specific details.

Subha: I think the accelerated application in the VM can also access directly to the virtio interface. Is that possible if somebody really wants to do that? Or do we mandate that the application has to come via SAL?

Keith: The SAL in the VM can also be optional. For generic applications, maybe the virtio is the only path it has.

Keith: I will try to redraw the picture, and I need to talk to Ferran as well to make sure that we agree on that picture. And I need to review the AAL slides again to try to understand it. And hopefully, on that picture, we can say that this is the layering we want, and we start to put some worlds around them and so forth to nail it down. And then we can go on and dive into the details. Does that seem like a reasonable direction?

Lingli: Yes. We will have further discussion over this probably offline. And there is also text input from Ferran for AAL's work flows at: https://etherpad.opnfv.org/p/dpacc_framework_flow, which might be helpful too.

Lingli(on behalf of Vic): There is also another contribution on the eatherpad for gap analysis over openstack, which combines previous contributions from Howard and Srini and welcomes further review and feedback at: https://etherpad.opnfv.org/p/dpacc_framework_gap_openstack

Vic: (tried to give an brief introduction but failed due to bad connection)

Lingli: For the general API defintions above the SAL in the VM, as it is parallel to the framework layering and really important for realizing the goal of dapcc, would encourage more input and discussion shortly.