Tempest is the OpenStack reference functional testing suite
It is run through Rally
Tempest | LF POD1 (Fuel) | LF POD2 (Foreman) |
---|---|---|
smoke | 27 KO on 105 tests (21/05) 26 KO on 100 (22/05) 14 KO on 106 (03/06) 13 KO on 96 (04/06 #38) | 26 NOK on 97 tests (13/05 - manual) 32 NOK on 105 tests (22/05 - manual) 41 NOK on 92 tests (#34) 39/104 (#35) 41/130 34/95 41/102 39/107 43/99 (#40) 0/0 (#45) 13/90 (#50) 37/98 (04/06 #54) |
all | 243 KO on 875 (5/05) |
#1. Issues due to bugs in the test itself #2. Issues due to installation of the system/component #3. Issues due to configuration of the system/component #4. Issues due to failures/bugs/etc. in a component
ID | Error | Associated tests | Comments |
---|---|---|---|
1 | tempest_lib.exceptions.BadRequest: Bad request Details: {u'message': u'Multiple possible networks found, use a Network ID to be more specific.', u'code': 400} | tempest.api.compute.servers.test_attach_interfaces.AttachInterfacesTestJSON.test_add_remove_fixed_ip tempest.api.compute.servers.test_create_server.ServersTestJSON tempest.api.compute.servers.test_server_actions.ServerActionsTestJSON tempest.api.compute.servers.test_create_server.ServersTestManualDisk tempest.api.compute.servers.test_server_addresses.ServerAddressesTestJSON tempest.api.volume.test_volumes_actions.VolumesV1ActionsTest tempest.api.volume.test_volumes_actions.VolumesV2ActionsTest tempest.scenario.test_volume_boot_pattern.TestVolumeBootPatternV2.test_volume_boot_pattern tempest.scenario.test_volume_boot_pattern.TestVolumeBootPattern.test_volume_boot_pattern tempest.scenario.test_server_basic_ops.TestServerBasicOps.test_server_basicops | |
| tempest_lib.exceptions.UnexpectedResponseCode: Unexpected response code received | tempest.api.network.test_networks.BulkNetworkOpsIpV6TestJSON.test_bulk_create_delete_port tempest.api.network.test_networks.BulkNetworkOpsIpV6TestJSON.test_bulk_create_delete_subnet tempest.api.network.test_networks.NetworksTestJSON.test_create_update_delete_network_subnet | |
8 | testtools.matchers._impl.MismatchError: u'desc2-1852962374' != u'description-229608591' | tempest.api.identity.admin.v3.test_services.ServicesTestJSON.test_create_update_get_service |
ID | Error | Associated tests | Comments |
---|---|---|---|
1 | tempest_lib.exceptions.BadRequest: Bad request Details: {u'message': u'Multiple possible networks found, use a Network ID to be more specific.', u'code': 400} | tempest.api.volume.test_volumes_actions.VolumesV2ActionsTest tempest.api.volume.test_volumes_actions.VolumesV1ActionsTest tempest.scenario.test_volume_boot_pattern.TestVolumeBootPattern.test_volume_boot_pattern tempest.scenario.test_volume_boot_pattern.TestVolumeBootPatternV2.test_volume_boot_pattern tempest.scenario.test_server_basic_ops.TestServerBasicOps.test_server_basicops tempest.api.compute.servers.test_create_server.ServersTestJSON tempest.api.compute.servers.test_attach_interfaces.AttachInterfacesTestJSON.test_add_remove_fixed_ip tempest.api.compute.servers.test_server_actions.ServerActionsTestJSON tempest.api.compute.servers.test_server_addresses.ServerAddressesTestJSON | |
| Details: {u'message': u"Quota exceeded for resources: ['network']", u'type': u'OverQuota', u'detail': u''} | tempest.api.network.test_networks.NetworksIpV6TestJSON.test_create_update_delete_network_subnet tempest.api.network.test_networks.BulkNetworkOpsIpV6TestJSON.test_bulk_create_delete_subnet tempest.api.network.test_floating_ips.FloatingIPTestJSON tempest.api.network.test_ports.PortsIpV6TestJSON.test_create_port_in_allowed_allocation_pools tempest.api.network.test_ports.PortsIpV6TestJSON.test_create_port_with_no_securitygroups tempest.api.network.test_routers.RoutersIpV6Test.test_add_remove_router_interface_with_subnet_id | |
3 | ValueError: need more than 1 value to unpack Response - Headers: {'date': 'Tue, 12 May 2015 13:05:08 GMT', 'status': '204', 'connection': 'close', 'content-length': '0', 'x-openstack-request-id': 'req-08ba5b6f-47a8-4c0b-adda-50b4dc2fc920'} | tempest.scenario.test_network_basic_ops.TestNetworkBasicOps.test_network_basic_ops | |
4 | tempest_lib.exceptions.UnexpectedResponseCode: Unexpected response code received Details: 503 NeutronError": {"message": "Unable to create the network. No available network found in maximum allowed attempts." | tempest.api.network.test_routers.RoutersIpV6Test.test_add_multiple_router_interfaces tempest.api.network.test_networks.BulkNetworkOpsTestJSON.test_bulk_create_delete_subnet tempest.api.network.test_networks.BulkNetworkOpsTestJSON.test_bulk_create_delete_port tempest.api.network.test_networks.BulkNetworkOpsIpV6TestJSON.test_bulk_create_delete_network | |
| Details: Not enough users available for multi-user testing | tempest.api.compute.security_groups.test_security_group_rules.SecurityGroupRulesTestJSON tempest.api.compute.security_groups.test_security_groups.SecurityGroupsTestJSON tempest.api.compute.servers.test_server_actions.ServerActionsTestJSON tempest.api.compute.servers.test_create_server.ServersTestJSON tempest.api.compute.flavors.test_flavors.FlavorsV2TestJSON tempest.api.compute.servers.test_create_server.ServersTestManualDisk tempest.api.compute.servers.test_attach_interfaces.AttachInterfacesTestJSON tempest.api.compute.servers.test_server_addresses.ServerAddressesTestJSON | |
6 | tempest_lib.exceptions.ServerFault: Got server fault Details: create_port_postcommit failed. | tempest.api.network.test_routers.RoutersIpV6Test.test_add_remove_router_interface_with_port_id | |
7 | tempest_lib.exceptions.UnexpectedResponseCode: Unexpected response code received Details: 503 | tempest.api.network.test_networks.NetworksTestJSON | |
| | tempest.api.identity.admin.v3.test_regions.RegionsTestJSON tempest.api.identity.admin.v3.test_endpoints.EndPointsTestJSON tempest.api.identity.admin.v3.test_groups.GroupsV3TestJSON tempest.api.identity.admin.v3.test_trusts.TrustsV3TestJSO tempest.api.identity.admin.v3.test_roles.RolesV3TestJSON tempest.api.identity.admin.v3.test_policies.PoliciesTestJSON tempest.api.identity.admin.v3.test_credentials.CredentialsTestJSON tempest.api.identity.admin.v3.test_services.ServicesTestJSON \\tempest.api.identity.admin.v3.test_domains.DomainsTestJSON |
Issue ID | Category ID | Solution/workaround |
---|---|---|
1 | #3 or #1 | These tests do not precise the network when creating instance. it automatically retrieves the first (and unique) network. If several networks have been created, the test is failed. http://lists.openstack.org/pipermail/openstack/2014-August/008736.html. We should either precise the network (and complete test coding) or ensure that there is only 1 possible network. to be tested with only 1 available network.. |
2 | #3 | The default value of network seems to be 10 (default value). This value shall be exceeded and triggers errors (parallel processing) (http://eavesdrop.openstack.org/irclogs/%23openstack-lbaas/%23openstack-lbaas.2015-04-29.log.html), either increase default value and or review cleaning |
3 | #3 or #1 | In many cases, the tests scenario tries to retrieve values from tables that may not exist (python error). These tables concern network and/or floating ip creation. This test will pass if issues ID #1 & #2 are solved. |
4 | #3? | not reproducible anmore |
5 | #3? | linked to netwok (only on POD2) |
6 | #3? | linked to netwok (only on POD2) |
7 | ? | |
8 | ? | not reproducible (replaced by #9) |
9 | #3 | appeared 22/5 linked to identity.admin.v3, problem comes from non definition of admin_domain_name, adding this value set to default prevents any error on v3. A priori no exportable variable, shall be done in the tempest.conf that is automatically generated or v3 identity tests shall be ignored. |