Wiki

This page is read only. You can view the source, but not change it. Ask your administrator if you think this is wrong.


====== Base Platform and Environment ======

  * Hardware: Latest Intel Server Platform.
  * Software: Based on Real-Time Linux https://rt.wiki.kernel.org/index.php/Main_Page.

Please refer to [[nfv-kvm-test]] for details.

====== Configuration ======

A right configuration is critical for improving the NFV performance/latency. Even working on the same codebase, different configrations can make completely different performance/latency result.

There are many combinations of configurations, from hardware configuration to Operating System configuration and application level configuration. And there is no one simple configuration that works for every case. To tune a specific scenario, it's important to know the behaviors of different configurations and their impact.
===== Platform Configuration =====

Some hardware features can be configured through firmware interface(like BIOS) but others may not be configurable (e.g. SMI on most platforms).

  * __Power management__: Most power management related features save power on the expensive of latency. These features includes: Intel®Turbo Boost Technology, Enhanced Intel®SpeedStep, Processor C state and P state.Normarlly they should be disabled. But depending on the real-time application design and latency requirements,there might be some featues can be enabled if the impact on deterministic execution of workload is small. 

  * __Hyper-Threading__: The logic cores that share resource with other logic cores can introduce latency so the feature is recommended to disable for realtime case.

  * __Legacy USB Support/Port 60/64 Emulation__: These features involve some emulation in firmware and can introduce randome latency. It is recommended to disable.

  * __SMI__: System Management Interrupt runs outside of the kernel code and can potentially cause latency. It is a pity there is no simple way to disable it. Some vendors may provide related switches in BIOS but most machines would not have.

===== Operating System Configuration =====

  * __CPU isolation__:  To achieve deterministic latency, dedicated CPUs should be allocated for realtime application. This can be achieved by isolating cpus from kernel scheduler. Please refer to http://lxr.free-electrons.com/source/Documentation/kernel-parameters.txt#L1608 for more information.

  * __Memory allocation__:  Memory shoud be reserved for realtime application and usually hugepage should be used to reduce page faut/TLB miss.

  * __IRQ affinity__: All the non-realtime IRQs should affinitized to non realtime CPUs to reduce the impact on realtime CPUs. Some OS distributions contain irqbalance deamon which balences the IRQs among all the cores dynamically. It should be disabled as well.

  * __Device assignment for VM__: If device is used in a VM, then device passthru is desirable. In this case, IOMMU should be enabled.

  * __Tickless__: Frequent tick cause latency. CONFIG_NOHZ_FULL should be enabled in linux kernel. With CONFIG_NOHZ_FULL, the physical CPU will trigger much less tick timer interrupt(currently, 1 tick per second). This can reduce latency because each host timer interrupt triggers VM exit from guest to host and cause performance/latency impact.

  * __TSC__: Mark TSC clock source as reliable. A TSC clock source that is thought as unreliable causes kernel to continuous to enable clock source watchdog to check if TSC frequency is still correct. On latest Intel platform with Constant TSC/Invariant TSC/Synchronized TSC, the TSC is reliable already hence the watchdog is useless but cause latency.

  * __Idle__: The poll option forced a polling idle loop that can slightly improve the performance of waking up an idle CPU.

  * __RCU_NOCB__: RCU is a kernel synchronization mechanism. Refer to http://lxr.free-electrons.com/source/Documentation/RCU/whatisRCU.txt for more information. With RCU_NOCB, the impact from RCU to the VNF will be reduced.

  * __Disable the RT throttling__: RT Throttling is a Linux kernel mechanism that occurs when a process or thread uses 100% of the core, leaving no resources for the Linux scheduler to execute the kernel/housekeeping tasks. RT Throttling increases the latency so should be disabled.

  * __NUMA configuration__: To achieve the best latency. CPU/Memory and device allocated for realtime application/VM should be in the same NUMA node.



====== Performance/Latency Tuning ======
  * timers: do not raise softirq unconditionally
A softirq is raised even when the active timer queue is empty which causes lots of context switches. In our case there is no timer user at present so the optimization can help us to count down the latency. [[https://gerrit.opnfv.org/gerrit/gitweb?p=kvmfornfv.git;a=commit;h=6d8b23062207c761e4606d1eae0e43cf36141a44 | See code change]].

  * Make vfio MSI interrupt be non-threaded
Threaded irq can help reduce interrupt latency because it avoids locking interrupt too long  in interrupt handler. But if the interrupt handler itself does not take much time just like vfio for which the only thing to do is inject the interrupt to guest which can be really fast. In such case threaded irq would cost time to do the context switch between irq thread and  interrupt handler. Another point is in NFV scenario such realtime interrupt(like DPDK interrupt)  is almost the highest priority, so making such interrupt non-threaded would certainly benefit the highest application. [[https://gerrit.opnfv.org/gerrit/gitweb?p=kvmfornfv.git;a=commit;h=a233b3fef0ef0048071145eb233becffbdf96d0f | See code change]].

  * Cache Allocation Technology(CAT) enabling
Last leve cache(LLC) contention is a key resource contention for memory intensive workloads running on the same socket. Intel CAT can be used to partition LLC among realtime/non-realtime apps/VMs.
Wiki

User Tools

Site Tools

Page Tools
