Wiki

This page is read only. You can view the source, but not change it. Ask your administrator if you think this is wrong.
======== Project: QTIP-Platform Performance Benchmarking =======

==== Project Information ====

  * Project Name: QTIP
  * Project Repository: QTIP
  * Primary Contact: Vikram Dham
  * Project Lead: Vikram Dham
  * Project Category: Integration & Testing
  * Etherpad:  https://etherpad.opnfv.org/p/Qtip
  * Jira:  https://jira.opnfv.org/browse/QTIP
  * Gerrit: https://gerrit.opnfv.org/gerrit/#/admin/projects/qtip
  * GIT: https://git.opnfv.org/cgit/qtip/
  * Mailing list tag: [QTIP]

==== Qtip Meeting Information ====

  * IRC: #opnfv-qtip
  * Schedule for C Release: TBD
==== Description: ====

  * Qtip is a performance benchmark suite for OPNFV platform
  * QTIP aims to benchmark OPNFV platforms through a "Bottom up" approach, testing baremetal components first.
  * Emphassis on platform performance through quantitative benchmarks rather than platform validation




Qtip project is a "Bottom-Up" approach in characterizing and benchmarking OPNFV Platform. Proper characterization of OPNFV platforms is a critical component to understand how well OPNFV (and by implication upstream components) perform in realistic deployment scenarios, to provide feedback to design teams, to provide platform level performance data to higher layer developers, users, and community at large. 

Qtip also aims to build necessary testing and benchmarking tools that will be needed to fulfill its goals, to automate when possible, and to provide these tools and automation software to upper layer and other testing projects in OPNFV.
==== Scope: ====

The overall problem this project tries to solve is the general characterization of an OPNFV platform. It will focus on general performance questions that are common to the platform itself, or applicable to multiple OPNFV use cases.
QTIP will provide the capability to quantify a platform's performance behavior in a standardized, rigorous, and open way, and a well-documented methodology to reproduce the results by anyone interested
  
Main activities include:

  * Identify components of OPNFV platforms that are important for VNF applications: [[http://www.etsi.org/deliver/etsi_gs/NFV-PER/001_099/001/01.01.01_60/gs_nfv-per001v010101p.pdf|NFV PER 001]]
  * These components can be divided into Computing, Networking and Storage characteristics
  * Identify benchmarks to evaluate the performance of these components
  * Develop/ leverage tools to benchmark these components
  * Identify the test case scenarios in which to run these benchmarks
  * Automate the configuration/execution of tests along with the collection of test results
  * Represent test results in a standardized format for performance comparison between varying platform configuration
  * Develop a flexible framework to add testcases

  




In order to achieve the above objectives, we intend to use both open source, and non-open source commercial tools  under guidelines developed by the community. 

Such sophisticated testing methodology and harness, once developed, will first be applied to quantify a platform's behavior from lower layer going up (Bottom-up Approach). We believe understanding lower layer behavior is a necessary prerequisite step of understanding more complex upper layer behavior with more interconnecting parts. Examples of such platform level sub-systems may include (for illustration only): CPU, Memory, NICs, Storage, Switching, Hypervisors, containers, Host OS, Guest OS, vSwitch/vForwarding, TCP/IP, base OpenStack, base ODL/SDN and so on.

==== Tests cases: ====
  - [[Compute test suite]]:
      * [[Dhrystone on host nodes]]
      * [[Dhrystone on virtual machines]]
      * [[Whetstone on multiple host compute nodes]]
      * [[Whetstone on virtual machines hosted on differnet compute nodes]]
      * [[Whetsone on Compute node vs virtual machines]]
      * [[RAMspeed on host machine]]
      * [[RAMspeed on virtual machine]]  
      * [[OpenSSL speed on host machine]]
      * [[OpenSSL speed on Virtual machine]]
      * [[nDPI on host node]]
      * [[nDPI on Virtual machine]]
    - [[Storage test suite]]
      * [[50 percent Rand/Read File storage benchmark using FIO for multiple compute nodes]]
      * [[50 percent Rand/Read File storage benchmark using FIO for  virtual machines]]
    - [[Network test suite]]
      * [[TCP Iperf troughput test across VMs on a single compute node]]
      * [[TCP Iperf throughput test across VMs on separate compute nodes]]
      * [[TCP Iperf throughput test across Multiple compute nodes]]
    - [[ Complete QTIP testsuite]]
==== Sample Tests: ====

**Computing Benchmarks**

Running QTIP benchmarks on different machines configured with different components would evaluate the influence of these components on the computing performance.
For example, comparing these benchmarks on baremetal machines with different CPUs would help evaluate the performance of the CPU. Same approach can be used to test memory performance.
Comparing the performance of these benchmarks on a baremetal machine vs a VM running on the same baremetal machine would help analyze the overhead of the hypervisor.

Some of the computing benchmarks include:

  * Dhrystone Benchmarks
  * Whetstone Benchmarks
  * Cachememory benchmark
  * Memory bandwidth benchmark
  * OpenSSL speed benchmark
   
**Networking Benchmarks**

More details within the coming week.

**Storage Benchmarks**

These benchmarks can be used to compare storage peformance for disks mounted locally  as well on a network. The different storage components  within different platforms can be compared using these benchmarks to get quantitative results for storage performance.

  * File I/O benchmark
  * Block Read/Write Benchmark


==== Dependencies: ====



  * QTIP will take the infrastructure configuration defined by the Pharos project as reference upon which to run the tests [[pharos|Pharos Project: Community Test Infrastructure]]
  * Octopus for Continuous Integration and Development [[octopus:project_proposal|Octopus CI]]
  * Releng/ Test collection project [[collection_of_test_results|Test Collection]] for collecting test results


**Software**

  * OPNFV Based Arno Installation (OpenStack)
  * OpenSource benchmark tools e.g unixbench
  * Python 2.7
  * Ansible 1.9.2




**Related projects**

  * Testbed Infrastructure (Pharos) on which the Platform Performance Benchmarking system will be built. [[opnfv_testbed_infrastructure|OPNFV Project Proposal: Testbed Infrastructure]]
  * Bootstrap/GetStarted project that produce OPNFV software builds (or equivalent).[[get_started:get_started_project_proposal|Project: Bootstrap/Get started!]], [[[project_proposals:apex_opnfv|Appex]]
  * Infrastructure verification (YardStick) [[yardstick|Project: Yardstick - Infrastructure Verification]] deals with Infrastructure validation of OPNFV platforms. QTIP aims to benchmark the performance of components within OPNFV platforms for a quantitative analysis and doesn't deal with validation of the platform.
  * Testing methodology has a rich set of industry standards, e.g. ETSI NFV ISG, IETF, and so on. As we develop or adopt testing methodologies, we will reference these standards whenever applicable.


==== Reference POD ====

For Brahamputra release, QTIP would be calculating three indices.

  * The Compute Suite Index
  * The Storage Suite Index
  * The Network Suite Index

These indices are calculated by comparing QTIP results with the QTIP reference results obtained at Dell's OPNFV Lab's POD 3

[[Reference POD hardware details]]

[[Reference POD QTIP results]]


==== Committers and Contributors: ====



Committers:

  * Vikram Dham <vikram@nvirters.com>
  * Wenjing Chu, <wenjing_chu@dell.com> 
  * Shrinivas Chavan, <shrinivas.chavan@ericsson.com>
  * Sheng-Ann Yu <sheng-ann.yu@ericsson.com>
  * Iben Rodriguez <Iben.Rodriguez@vmsec.com>
  * Cooper, Trevor <trevor.cooper@intel.com>
  * Michael Haugh <mhaugh@clearpathnet.com>
  * Richomme, Morgan <morgan.richomme@orange.com> 
  * Nauman Ahad <nauman.ahad@xflowresearch.com>
  * Pang Hao <shamrock.pang@huawei.com>

Contributors:
  * 'Palani Chinnakannan (pals)' <pals@cisco.com>
  * aihua.li <aihua.li@huawei.com>
  * Tien, Michael <Michael_Tien@Dell.com>
  * Mofassir ul Islam Arif <mofassir.arif@xflowresearch.com>
==== Planned deliverables ====

  * QTIP benchmarking suite
  * Qtip documentations: mostly on wiki for test info, methodologies, references, user guides
  * One or more operational Qtip system on community testbeds
  * OPNFV script source tools for automation
  * Integration of results within 
  * Repository for Test Suite and Results 

==== Proposed Release Schedule: ====

  * Aligned with OPNFV Release Plan 2.

==== Meeting Minutes: ====

  * 09/03/2015 - http://ircbot.wl.linuxfoundation.org/meetings/opnfv-qtip/2015/opnfv-qtip.2015-09-03-20.23.txt

==== Documents: ====

  * QTIP revised scope: {{:qtip_20th_july.pptx|}}
  * QTIP 0.1: {{:qtip_0.1_6th_august.pptx|}}
  * QTIP meeting 13th August Slide deck: {{::qtip_13th_aug_meeting.pptx|}}
  * QTIP presentation to Testing Weekly Meeting 3rd September Slide deck: {{::opnfv_qtip_status_09032015.pptx|}}
Wiki

User Tools

Site Tools

Page Tools
