====== Test Environment ======
The KVM4NFV testing, especially the latency, requires special test environment setup, which includes BIOS setup, kernel configuration, kernel parameter and run time environment.
===== Hardware Platform Description =====
As currently there is no test bed assigned to kvm4nfv project yet. This section describes the hardware platform environment that is used to conduct the baseline performance data collection. In future, this will be updated for testbed.
==== CPU Info ====
processor : 35
vendor_id : GenuineIntel
cpu family : 6
model : 63
model name : Intel(R) Xeon(R) CPU E5-2699 v3 @ 2.30GHz
stepping : 2
microcode : 0x2d
cpu MHz : 2294.795
cache size : 46080 KB
physical id : 1
siblings : 18
core id : 27
cpu cores : 18
apicid : 118
initial apicid : 118
fpu : yes
fpu_exception : yes
cpuid level : 15
wp : yes
flags : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall nx pdpe1gb rdtscp lm constant_tsc arch_perfmon pebs bts rep_good nopl xtopology nonstop_tsc aperfmperf eagerfpu pni pclmulqdq dtes64 monitor ds_cpl vmx smx est tm2 ssse3 fma cx16 xtpr pdcm pcid dca sse4_1 sse4_2 x2apic movbe popcnt tsc_deadline_timer aes xsave avx f16c rdrand lahf_lm abm arat epb pln pts dtherm tpr_shadow vnmi flexpriority ept vpid fsgsbase tsc_adjust bmi1 avx2 smep bmi2 erms invpcid cqm xsaveopt cqm_llc cqm_occup_llc
bugs :
bogomips : 4595.54
clflush size : 64
cache_alignment : 64
address sizes : 46 bits physical, 48 bits virtual
power management:
==== CPU Topology ====
[nfv@otcnfv02 ~]$ lscpu
Architecture: x86_64
CPU op-mode(s): 32-bit, 64-bit
Byte Order: Little Endian
CPU(s): 36
On-line CPU(s) list: 0-35
Thread(s) per core: 1
Core(s) per socket: 18
Socket(s): 2
NUMA node(s): 2
Vendor ID: GenuineIntel
CPU family: 6
Model: 63
Model name: Intel(R) Xeon(R) CPU E5-2699 v3 @ 2.30GHz
Stepping: 2
CPU MHz: 2294.795
BogoMIPS: 4595.54
Virtualization: VT-x
L1d cache: 32K
L1i cache: 32K
L2 cache: 256K
L3 cache: 46080K
NUMA node0 CPU(s): 0-17
NUMA node1 CPU(s): 18-35
==== NIC ====
[nfv@otcnfv02 ~]$ lspci |grep Ether
03:00.0 Ethernet controller: Intel(R) Ethernet Controller X540-AT2 (rev 01)
03:00.1 Ethernet controller: Intel(R) Ethernet Controller X540-AT2 (rev 01)
81:00.0 Ethernet controller: Intel(R) Ethernet Controller X710 for 10GbE SFP+ (rev 01)
81:00.1 Ethernet controller: Intel(R) Ethernet Controller X710 for 10GbE SFP+ (rev 01)
81:00.2 Ethernet controller: Intel(R) Ethernet Controller X710 for 10GbE SFP+ (rev 01)
81:00.3 Ethernet controller: Intel(R) Ethernet Controller X710 for 10GbE SFP+ (rev 01)
===== BIOS Setup =====
Carefully BIOS setup is important to achieve real time latency. Different platform has different BIOS setup, below is the important BIOS setting on the platform collecting the baseline performance data.
CPU Power and Performance
CPU C-State
C1E Autopromote
Processor C3
Processor C6
Select Memory RAS
NUMA Optimized
Cluster-on-Die
Patrol Scrub
Demand Scrub
Correctable Error <10>
Intel(R) Hyper-Threading
Active Processor Cores
Execute Disable Bit
Intel(R) Virtualization Technology
Intel(R) TXT
Enhanced Error Containment Mode
USB Controller
USB 3.0 Controller
Legacy USB Support
Port 60/64 Emulation
===== Software Environment Setup =====
We need setup both the host/guest environment, to reduce any noise. https://gerrit.opnfv.org/gerrit/#/c/2161/ gives detailed implementation on how to setup the environment setup. Below are descriptions of the work done there.
==== Kernel Parameter ====
__Kernel configuration__: kernel/arch/x86/configs/opnfv.config.
__Host kernel boot line example__: isolcpus=11-15,31-35 nohz_full=11-15,31-35 rcu_nocbs=11-15,31-35 iommu=pt intel_iommu=on default_hugepagesz=1G hugepagesz=1G mce=off idle=poll intel_pstate=disable processor.max_cstate=1 pcie_asmp=off tsc=reliable
__Guest kernel boot line example__: isolcpus=1 nohz_full=1 rcu_nocbs=1 mce=off idle=poll default_hugepagesz=1G hugepagesz=1G
Please refer to [[nfv-kvm-tuning]] for more explanation.
==== Run-time Environment Setup ====
Not only special kernel parameter needed, also run-time environment adjustment is needed. Below are some BKMs.
Please refer to [[nfv-kvm-tuning]] for more explanation.
====== Test Description ======
Currently there are three tests are executed for the KVM4NFV project. Please notice, as the integration with the test project is WIP, these test are executed on an Intel lab, instead of OPNFV test bed.
1. Cyclictest
This is a common real time benchmark. Please refer to https://rt.wiki.kernel.org/index.php/Cyclictest for more information.
2. Device Interrupt Latency Test
This is a custom test where a FPGA card produce interrupt every 1ms. The latency from the interrupt invoked to the interrupt serviced will be mersured.
3. L2 Packet Forwarding
This is only used live migration now, although it's planned to be used for real time latency test in future.
====== Performance Data ======
We collected baseline performance data, i.e. the performance data without any enhancement, but with all the system configuration applied already.
===== Baseline Performance Data =====
1. CyclicTest
# Min Latencies: 00005
# Avg Latencies: 00007
# Max Latencies: 00032
# Histogram Overflows: 00002
# Histogram Overflow at cycle number:
# Thread 0: 45337966 50108448
2. Device Interrupt Latency
Latency is Min: 7.22us Max: 55.28us Avg: 8.93us
Jitter is Min: 963.20us Max: 1040.68us Avg: 1002.62us
===== Latest Performance Data =====
This is the test result at 11/09/2015.
1. CyclictTest
# Total: 086400000
# Min Latencies: 00006
# Avg Latencies: 00006
# Max Latencies: 00011
# Histogram Overflows: 00000
# Histogram Overflow at cycle number:
# Thread 0:
2. Device Interrupt Latency
Period is 1000 us
Latency is Min: 4.11us Max: 9.11us Avg: 5.06us
Jitter is Min: 997.09us Max: 1008.23us Avg: 1002.62us
===== Latest Live Migration Performance Data =====
For live migration performance data, please refer to https://etherpad.opnfv.org/p/live_migration_optimization_data for information.