[[http://docs.openstack.org/developer/tempest/overview.html|Tempest]] is the OpenStack reference functional testing suite It is run through Rally ===== Summary ===== ^ Tempest ^ LF POD1 (Fuel) ^ LF POD2 (Foreman) ^ | smoke | 27 KO on 105 tests (21/05)\\ 26 KO on 100 (22/05)\\ 14 KO on 106 (03/06) \\ 13 KO on 96 (04/06 #38) | 26 NOK on 97 tests (13/05 - manual) \\ 32 NOK on 105 tests (22/05 - manual) \\ 41 NOK on 92 tests (#34) \\ 39/104 (#35) \\ 41/130 \\ 34/95 \\ 41/102 \\ 39/107 \\ 43/99 (#40) \\ 0/0 (#45) \\ 13/90 (#50) \\ 37/98 (04/06 #54) | | all | | 243 KO on 875 (5/05) | ===== Error details ===== === Error categories === #1. Issues due to bugs in the test itself #2. Issues due to installation of the system/component #3. Issues due to configuration of the system/component #4. Issues due to failures/bugs/etc. in a component === POD 1 === ^ ID ^ Error ^ Associated tests ^ Comments ^ | 1 | tempest_lib.exceptions.BadRequest: Bad request \\ Details: {u'message': u'Multiple possible networks found, use a Network ID to be more specific.', u'code': 400} | tempest.api.compute.servers.test_attach_interfaces.AttachInterfacesTestJSON.test_add_remove_fixed_ip \\ tempest.api.compute.servers.test_create_server.ServersTestJSON \\ tempest.api.compute.servers.test_server_actions.ServerActionsTestJSON \\ tempest.api.compute.servers.test_create_server.ServersTestManualDisk \\ tempest.api.compute.servers.test_server_addresses.ServerAddressesTestJSON \\ tempest.api.volume.test_volumes_actions.VolumesV1ActionsTest \\ tempest.api.volume.test_volumes_actions.VolumesV2ActionsTest \\ tempest.scenario.test_volume_boot_pattern.TestVolumeBootPatternV2.test_volume_boot_pattern \\ tempest.scenario.test_volume_boot_pattern.TestVolumeBootPattern.test_volume_boot_pattern \\ tempest.scenario.test_server_basic_ops.TestServerBasicOps.test_server_basicops | | | 2 | tempest_lib.exceptions.UnexpectedResponseCode: Unexpected response code received | tempest.api.network.test_networks.BulkNetworkOpsIpV6TestJSON.test_bulk_create_delete_port \\ tempest.api.network.test_networks.BulkNetworkOpsIpV6TestJSON.test_bulk_create_delete_subnet \\ tempest.api.network.test_networks.NetworksTestJSON.test_create_update_delete_network_subnet | | | 8 | testtools.matchers._impl.MismatchError: u'desc2-1852962374' != u'description-229608591' | tempest.api.identity.admin.v3.test_services.ServicesTestJSON.test_create_update_get_service | | === POD 2 === ^ ID ^ Error ^ Associated tests ^ Comments ^ | 1 | tempest_lib.exceptions.BadRequest: Bad request \\ Details: {u'message': u'Multiple possible networks found, use a Network ID to be more specific.', u'code': 400} | tempest.api.volume.test_volumes_actions.VolumesV2ActionsTest \\ tempest.api.volume.test_volumes_actions.VolumesV1ActionsTest \\ tempest.scenario.test_volume_boot_pattern.TestVolumeBootPattern.test_volume_boot_pattern \\ tempest.scenario.test_volume_boot_pattern.TestVolumeBootPatternV2.test_volume_boot_pattern \\ tempest.scenario.test_server_basic_ops.TestServerBasicOps.test_server_basicops \\ tempest.api.compute.servers.test_create_server.ServersTestJSON \\ tempest.api.compute.servers.test_attach_interfaces.AttachInterfacesTestJSON.test_add_remove_fixed_ip \\ tempest.api.compute.servers.test_server_actions.ServerActionsTestJSON \\ tempest.api.compute.servers.test_server_addresses.ServerAddressesTestJSON | | | 2 | tempest_lib.exceptions.Conflict: An object with that identifier already exists \\ Details: {u'message': u"Quota exceeded for resources: ['network']", u'type': u'OverQuota', u'detail': u''} | tempest.api.network.test_networks.NetworksIpV6TestJSON.test_create_update_delete_network_subnet \\ tempest.api.network.test_networks.BulkNetworkOpsIpV6TestJSON.test_bulk_create_delete_subnet \\ tempest.api.network.test_floating_ips.FloatingIPTestJSON \\ tempest.api.network.test_ports.PortsIpV6TestJSON.test_create_port_in_allowed_allocation_pools \\ tempest.api.network.test_ports.PortsIpV6TestJSON.test_create_port_with_no_securitygroups \\ tempest.api.network.test_routers.RoutersIpV6Test.test_add_remove_router_interface_with_subnet_id | | | 3 | ValueError: need more than 1 value to unpack \\ Response - Headers: {'date': 'Tue, 12 May 2015 13:05:08 GMT', 'status': '204', 'connection': 'close', 'content-length': '0', 'x-openstack-request-id': 'req-08ba5b6f-47a8-4c0b-adda-50b4dc2fc920'} | tempest.scenario.test_network_basic_ops.TestNetworkBasicOps.test_network_basic_ops | | | 4 | tempest_lib.exceptions.UnexpectedResponseCode: Unexpected response code received \\ Details: 503 \\ NeutronError": {"message": "Unable to create the network. No available network found in maximum allowed attempts." | tempest.api.network.test_routers.RoutersIpV6Test.test_add_multiple_router_interfaces \\ tempest.api.network.test_networks.BulkNetworkOpsTestJSON.test_bulk_create_delete_subnet tempest.api.network.test_networks.BulkNetworkOpsTestJSON.test_bulk_create_delete_port \\ tempest.api.network.test_networks.BulkNetworkOpsIpV6TestJSON.test_bulk_create_delete_network | | | 5 | tempest.exceptions.InvalidConfiguration: Invalid Configuration \\ Details: Not enough users available for multi-user testing | tempest.api.compute.security_groups.test_security_group_rules.SecurityGroupRulesTestJSON \\ tempest.api.compute.security_groups.test_security_groups.SecurityGroupsTestJSON \\ tempest.api.compute.servers.test_server_actions.ServerActionsTestJSON \\ tempest.api.compute.servers.test_create_server.ServersTestJSON \\ tempest.api.compute.flavors.test_flavors.FlavorsV2TestJSON \\ tempest.api.compute.servers.test_create_server.ServersTestManualDisk \\ tempest.api.compute.servers.test_attach_interfaces.AttachInterfacesTestJSON \\ tempest.api.compute.servers.test_server_addresses.ServerAddressesTestJSON | | | 6 | tempest_lib.exceptions.ServerFault: Got server fault \\ Details: create_port_postcommit failed. | tempest.api.network.test_routers.RoutersIpV6Test.test_add_remove_router_interface_with_port_id | | | 7 | tempest_lib.exceptions.UnexpectedResponseCode: Unexpected response code received \\ Details: 503 | tempest.api.network.test_networks.NetworksTestJSON | | | 9 | Invalid credentials | tempest.api.identity.admin.v3.test_regions.RegionsTestJSON \\ tempest.api.identity.admin.v3.test_endpoints.EndPointsTestJSON \\ tempest.api.identity.admin.v3.test_groups.GroupsV3TestJSON \\ tempest.api.identity.admin.v3.test_trusts.TrustsV3TestJSO \\ tempest.api.identity.admin.v3.test_roles.RolesV3TestJSON \\ tempest.api.identity.admin.v3.test_policies.PoliciesTestJSON \\ tempest.api.identity.admin.v3.test_credentials.CredentialsTestJSON \\ tempest.api.identity.admin.v3.test_services.ServicesTestJSON \\tempest.api.identity.admin.v3.test_domains.DomainsTestJSON | | === Error follow up === ^ Issue ID ^ Category ID ^ Solution/workaround ^ | 1 | #3 or #1 | These tests do not precise the network when creating instance. it automatically retrieves the first (and unique) network. If several networks have been created, the test is failed. http://lists.openstack.org/pipermail/openstack/2014-August/008736.html. We should either precise the network (and complete test coding) or ensure that there is only 1 possible network. to be tested with only 1 available network.. | | 2 | #3 | The default value of network seems to be 10 (default value). This value shall be exceeded and triggers errors (parallel processing) (http://eavesdrop.openstack.org/irclogs/%23openstack-lbaas/%23openstack-lbaas.2015-04-29.log.html), either increase default value and or review cleaning | | 3 | #3 or #1 | In many cases, the tests scenario tries to retrieve values from tables that may not exist (python error). These tables concern network and/or floating ip creation. This test will pass if issues ID #1 & #2 are solved. | | 4 | #3? | not reproducible anmore | | 5 | #3? | linked to netwok (only on POD2) | | 6 | #3? | linked to netwok (only on POD2) | | 7 | ? | | | 8 | ? | not reproducible (replaced by #9) | | 9 | #3 | appeared 22/5 linked to identity.admin.v3, problem comes from non definition of admin_domain_name, adding this value set to default prevents any error on v3. A priori no exportable variable, shall be done in the tempest.conf that is automatically generated or v3 identity tests shall be ignored. | ==== references ==== [[List of Tempest smoke tests]] [[https://etherpad.opnfv.org/p/functest-lfpod1-testresults|Tempest details POD1]] [[https://etherpad.opnfv.org/p/functest-lfpod2-testresults|Tempest details POD2]] [[https://etherpad.opnfv.org/p/functest-ericssonpod-testresults |Tempest detail Ericsson POD]]