User Tools

Site Tools


opnfv_functional_testing

Differences

This shows you the differences between two versions of the page.

Link to this comparison view

Both sides previous revision Previous revision
Next revision
Previous revision
opnfv_functional_testing [2015/07/08 12:50]
Morgan Richomme
opnfv_functional_testing [2016/03/01 08:48]
Ryota Mibu [Key Project Facts]
Line 1: Line 1:
 ====== Project: OPNFV - Base system functionality testing (Functest) ====== ====== Project: OPNFV - Base system functionality testing (Functest) ======
  
-  * //Proposed name for the project:// **OPNFV – Base system functionality testing** +  * Project NameFunctest \\ 
-  * //Proposed name for the repository:// **functest** +  * Project Repository: functest ​\\ 
-  * //Project ​Categories:// Integration & Testing +  * Project ​Category: Integration & Testing ​\\ 
-  * //project Etherpad:// https://​etherpad.opnfv.org/​p/​functiontestrelease1activities +  * Project ​IRC chan: #​opnfv-testperf ​\\ 
-  * //IRC chan:// #​opnfv-testperf +  * [[functest_meeting|Team meeting ​page]] \\ 
-  * //[[functest_meeting|weekly team meeting]]:// every Tuesday (4PM UTC) +  * [[functest_team|Functest ​Team]] ​\\ 
-  * //[[functest_team|Team]]://​+ 
 +^ {{:​information.png?​70|}} ^ **Quick links** \\ [[https://jira.opnfv.org/​secure/​RapidBoard.jspa?​rapidView=59&​view=detail&​selectedIssue=FUNCTEST-55|Functest Kanban board]] \\ [[https://​jira.opnfv.org/​secure/​Dashboard.jspa?​selectPageId=10611|Functest Jira Dashboard]] \\ [[https://​wiki.opnfv.org/​functextnexttaks|Functest Brahmaputra page]] \\ [[opnfv_functional_testing#​Testcases|Testcases]]\\ [[opnfv_functional_testing#​Videos|Functest videos]] ^ 
  
  
Line 14: Line 15:
 Project “OPNFV – Base system functionality testing” will provide comprehensive testing methodology,​ test suites and test cases to test and verify OPNFV Platform functionality that covers the  VIM and NFVI components. ​ Project “OPNFV – Base system functionality testing” will provide comprehensive testing methodology,​ test suites and test cases to test and verify OPNFV Platform functionality that covers the  VIM and NFVI components. ​
  
-This project uses a "tops-down" approach that will start with chosen ETSI NFV use-case/s and open source VNFs for the functional testing. The approach taken will be to +This project uses a "top-down" approach that will start with chosen ETSI NFV use-case/s and open source VNFs for the functional testing. The approach taken will be to 
   * break down the use-case into simple operations and functions required. ​   * break down the use-case into simple operations and functions required. ​
   * specify necessary network topologies ​   * specify necessary network topologies ​
-  * develop necessary test traffic ​and traffic profiles. ​Ideally VNFs will be Open Source however proprietary VNFs may also be used as needed. ​+  ​* develop [[https://​wiki.opnfv.org/​traffic_profile_management|traffic profiles]] 
 +  ​* develop necessary test traffic 
 +    * Ideally VNFs will be Open Source however proprietary VNFs may also be used as needed. ​
  
 This project will develop test suites that cover detailed functional test cases, test methodologies and platform configurations which will be documented and maintained in a repository for use by other OPNFV testing projects and the community in general. ​ Developing test suites will also help lay the foundation for a test automation framework that in future can be used by the continuation integration (CI) project (Octopus). We envisage that certain VNF deployment use-cases could be automatically tested as an optional step of the CI process. This project will develop test suites that cover detailed functional test cases, test methodologies and platform configurations which will be documented and maintained in a repository for use by other OPNFV testing projects and the community in general. ​ Developing test suites will also help lay the foundation for a test automation framework that in future can be used by the continuation integration (CI) project (Octopus). We envisage that certain VNF deployment use-cases could be automatically tested as an optional step of the CI process.
Line 29: Line 32:
   * Automate test with CI    * Automate test with CI 
  
 +A dedicated page for all the testing projects (Functest, Yardstick, Pharos, ...) is available [[testing|here]].
  
 ===== Scope ===== ===== Scope =====
Line 51: Line 55:
   * [[https://​wiki.openstack.org/​wiki/​Rally|Rally]] (benchmark, Tempest)   * [[https://​wiki.openstack.org/​wiki/​Rally|Rally]] (benchmark, Tempest)
   * [[http://​robotframework.org/​|Robot framework]]   * [[http://​robotframework.org/​|Robot framework]]
 +  * [[https://​wiki.onosproject.org/​display/​ONOS/​TestON+Tutorial|Teston framework]]
  
 Functional tests shall be Functional tests shall be
-  * independant ​from the installer (Fuel, Foreman/​Puppet,​ Juju,​..) ​+  * independent ​from the installer (Fuel, Foreman/​Puppet,​ Juju,​..) ​
   * automated and integrated in CI   * automated and integrated in CI
- 
  
  
 ===== Functest versus Release ===== ===== Functest versus Release =====
  
-[[functest_release_1| Functest for Release 1 ]]+[[functest_release_1| Functest for ARNO (Release 1]]
  
-[[FuncTextNextTaks|functest ​for Release 2]].+[[functest_release_2Functest ​for BRAHMAPUTRA (Release 2]]
  
 +[[FuncTextNextTaks| Functest Next tasks]].
  
 +===== Testcases =====
  
-===== Dependencies =====+^Testcase ^Availability ^Comment ^ 
 +| vPing | Arno | basic hello world example.| 
 +| ODL | Arno | Using robotframework,​ 3 pending errors due to ODL bug in Arno. |  
 +| Tempest | Arno | Smoke suite considered. Still troubleshooting to be done. | 
 +| Rally | Arno | based on default Rally scenario. Still troubleshooting to be done. | 
 +| vIMS | Brahmaputra | candidate testcase using Cloudify for VNF management. [[https://​wiki.opnfv.org/​ip_multimedia_subsystem|pre-study]] | 
 +| ONOS | Brahmaputra | candidate using teston tool | 
 +| SFC | Brahmaputra | candidate | 
 +| BGPVPN | Brahmaputra | candidate | 
 +| Policy | Brahmaputra | candidate | 
 +| vPE | ? | [[https://​wiki.opnfv.org/​virtual_provider_edge|pre-study]]| 
 +| vPE perfo | ? | [[https://​wiki.opnfv.org/​vpe_performance_testing|pre-study]] | 
 +| vEPC | ? | | 
 +| vCDN | ? | |
  
-  * The project is a contributor to project “Octopus”. +[[https://wiki.opnfv.org/vnf_deployment_test_cases|Study on VNF deployment]]
-  * The project leverages ​ "Boot Strap/Get Started"​ +
-  * The project relies on the following upstream projects: +
-    * OpenStack Juno release: Components: Nova, Glance, Keystone, Horizon, Neutron, Ceilometer, Heat +
-    * OpenDaylight Helium release: Components: MDSAL, OVSDB, RESTCONF, ML2 plugin/ODL Neutron drivers +
-    * Installer: TBD +
-    * Puppet (for instance configuration) +
-    * QEMU/KVM +
-    * OpenWRT (as example VNF - for routing, firewall, NAT) +
-    * Snort (as example VNF - for IDS) +
-    * Linux   ​Ubuntu 14.04/Centos 7 distribution+
  
 +===== Videos =====
 +
 +[[https://​www.youtube.com/​watch?​v=7IO6t3C_OFA | Functest demo for Arno SR0]]
 +
 +===== Tips =====
 +
 +==== Short story on Functest versus the other test projects and the installers ====
 +
 +First of all we try to be as agnostic as possible.
 +
 +we see OPNFV as a black box.
 +
 +We automatically run functest-{installer}-master after a successful fresh installation of {installer}
 +In functest (and in yardstick) we have internal test cases and companion test projects
 +the functest companion projects dealing with functional testing are
 +  * doctor
 +  * copper
 +  * ovno
 +  * policyTest
 +  * promise
 +  * sdnvpn
 +
 +companion project means that we wil onboard them in Functest CI process. After a fresh install, functest will be started (a docker file) running several tests one after the others (e.g. https://​build.opnfv.org/​ci/​view/​functest/​job/​functest-fuel-master/​lastSuccessfulBuild/​consoleText) including companion tests in the near future.
 +each test may need specific env, so functest can install tooling and triggers scripts to setup env
 +we will also create a filtering mechanism to run tests only when possible (assuming that by default it is always possible) but no need to run odl suite towards an OPNFV including onos
 +
 +In functest we developed ​ the code for our internal tests (scripts, tests, integration of upstream test suites,...)
 +
 +The companion projects are in charge of their scripts, their code, their scenario...
 +we mutualize the CI env (running on target CI production labs), the tooling (some projects need for instance to run upstream ODL suite, functest install once robot and mininet for all of them, Rally may test module used by feature project (eg; congress/​copper)),​ the reporting, the documentation
 +
 +Then jenkins functest-{installer} job will produce results that will be pushed in the Test collection DB
 +before Release B we will look carefully to the jobs (success criteria for Functest for B-Release is 4 successfull run on CI of the onboarded projects)
 +  * if errors in integration or internal cases => we will blame ourselves
 +  * if erros in companion projects => we will report the issues to the member of the feature projects
 +
 +So you can see that we did not do specific stuff for 1 installer.
 +We have some adaptations of course in our scripts (e.g. to retrieve the credentials - not the same way on the different installers) but we do not create a plugin for a given installers, we have a docker file.
 +Functest (as yardstick) docker file could even be almost used towards any OPNFV-like solution (some adaptations on the env setup) ​
 +
 +
 +==== Git ====
 +
 +Dealing with Branches
 +
 +Best practice => Cherry Pick from Master to Stable useful patches as described in [[releases:​stablebranch|Stable Branch]]
 +
 +If you forgot that (it may happen and we experienced that for SR1) as Master and stable may be considered as the same, you may have to merge Master on Stable...and then winter is coming...so a procedure that works:
 +  ​
 +   * git clone ssh://<​Your_ID_assuming_you_have_the_merge_rights>​@gerrit.opnfv.org:​29418/​functest functest-clean
 +   * git checkout stable/arno
 +   * git merge master
 +   * git push origin HEAD:​refs/​for/​stable/​arno.
  
 ---- ----
Line 89: Line 151:
     * Frank Brockners (fbrockne@cisco.com)     * Frank Brockners (fbrockne@cisco.com)
     * Sajeev Manikkoth (sajeevmanikkoth@gmail.com)     * Sajeev Manikkoth (sajeevmanikkoth@gmail.com)
-    * Jun Li (matthew.lijun@huawei.com) 
     * Sean Chen (s.chen@huawei.com)     * Sean Chen (s.chen@huawei.com)
     * Rajeev Seth (rseth@sonusnet.com)     * Rajeev Seth (rseth@sonusnet.com)
     * Kevin Riley (kriley@sonusnet.com)     * Kevin Riley (kriley@sonusnet.com)
     * Justin Hart (jhart@sonusnet.com) ​     * Justin Hart (jhart@sonusnet.com) ​
 +    * Ryota Mibu (r-mibu@cq.jp.nec.com)
  
  
 Want to Join: [[https://​wiki.opnfv.org/?​ns=%3A&​image=%3Aopnfv-_functest.pdf&​do=media&​tab_files=files&​tab_details=view| know more on Functest]] Want to Join: [[https://​wiki.opnfv.org/?​ns=%3A&​image=%3Aopnfv-_functest.pdf&​do=media&​tab_files=files&​tab_details=view| know more on Functest]]
- ==== Planned deliverables ==== 
- 
-Project deliverable:​ 
-The project delivers the following components: 
-  
-  * Documentation of Test Suites and test cases that cover the OPNFV platform functionality testing including pass/fail criteria. 
-  * Test software and scripts for testing OPNFV Platform functionality that is essentially broken down into two main suites as follows: 
-    * OPNFV Platform Smoke test suite: ​ A sanity testing suite for basic verification of the platform. 
-    * OPNFV Platform Regression test suite: A comprehensive collection of detailed test suites. 
-  * Automated Test Framework based on Tempest/​Robot/​Jenkins,​ with necessary scripts and tools to automatically test and verify OPNFV functionality 
- 
-=== Proposed Release Schedule ==== 
  
-OPNFV release #1.+{{:​devtest_journey.pdf|Tester Journey}}
opnfv_functional_testing.txt · Last modified: 2016/03/01 08:48 by Ryota Mibu