User Tools

Site Tools


r1_tempest

This is an old revision of the document!


Table of Contents

Tempest is the OpenStack reference functional testing suite

It is run through Rally

Summary

Tempest LF POD1 (Fuel) LF POD2 (Foreman)
smoke 27 KO on 105 tests (21/05)
26 Ko on 100 (22/05)
26 KO on 97 tests (13/05)
32 KO on 105 tests (22/05)
all 243 KO on 875 (5/05)

Error details

Error categories

#1. Issues due to bugs in the test itself
#2. Issues due to installation of the system/component
#3. Issues due to configuration of the system/component
#4. Issues due to failures/bugs/etc. in a component

POD 1

ID Error Associated tests Comments
1 tempest_lib.exceptions.BadRequest: Bad request
Details: {u'message': u'Multiple possible networks found, use a Network ID to be more specific.', u'code': 400}
tempest.api.compute.servers.test_attach_interfaces.AttachInterfacesTestJSON.test_add_remove_fixed_ip
tempest.api.compute.servers.test_server_actions.ServerActionsTestJSON
tempest.api.volume.test_volumes_actions.VolumesV2ActionsTest
tempest.api.volume.test_volumes_actions.VolumesV1ActionsTest
tempest.scenario.test_volume_boot_pattern.TestVolumeBootPattern.test_volume_boot_pattern
tempest.api.compute.servers.test_create_server.ServersTestJSON
tempest.scenario.test_volume_boot_pattern.TestVolumeBootPatternV2.test_volume_boot_pattern
tempest.api.compute.servers.test_create_server.ServersTestManualDisk
tempest.scenario.test_server_basic_ops.TestServerBasicOps.test_server_basicops
tempest.api.compute.servers.test_server_addresses.ServerAddressesTestJSON
2 tempest_lib.exceptions.Conflict: An object with that identifier already exists
Details: Quota exceeded for resources: ['network']", u'type': u'OverQuota', u'detail'
tempest.api.network.test_networks.BulkNetworkOpsTestJSON.test_bulk_create_delete_subnet
tempest.api.network.test_networks.BulkNetworkOpsIpV6TestJSON.test_bulk_create_delete_subnet
tempest.api.network.test_routers.RoutersTest.test_add_remove_router_interface_with_port_id
tempest.api.network.test_networks.BulkNetworkOpsTestJSON.test_bulk_create_delete_port
tempest.api.network.test_networks.BulkNetworkOpsTestJSON.test_bulk_create_delete_network
tempest.api.network.test_networks.BulkNetworkOpsIpV6TestJSON.test_bulk_create_delete_port
tempest.api.network.test_networks.BulkNetworkOpsIpV6TestJSON.test_bulk_create_delete_network
tempest.api.network.test_routers.RoutersTest.test_add_multiple_router_interfaces
tempest.api.network.test_ports.PortsIpV6TestJSON.test_create_port_with_no_securitygroups
tempest.api.network.test_networks.NetworksTestJSON
tempest.api.network.test_routers.RoutersTest.test_add_remove_router_interface_with_subnet_id
3 ValueError: need more than 1 value to unpack tempest.scenario.test_network_basic_ops.TestNetworkBasicOps.test_network_basic_ops
8 testtools.matchers._impl.MismatchError: u'desc2-1852962374' != u'description-229608591' tempest.api.identity.admin.v3.test_services.ServicesTestJSON.test_create_update_get_service
9 Invalid credentials tempest.api.identity.admin.v3.test_regions.RegionsTestJSON
tempest.api.identity.admin.v3.test_endpoints.EndPointsTestJSON
tempest.api.identity.admin.v3.test_groups.GroupsV3TestJSON
tempest.api.identity.admin.v3.test_trusts.TrustsV3TestJSO
tempest.api.identity.admin.v3.test_roles.RolesV3TestJSON
tempest.api.identity.admin.v3.test_policies.PoliciesTestJSON
tempest.api.identity.admin.v3.test_credentials.CredentialsTestJSON
tempest.api.identity.admin.v3.test_services.ServicesTestJSON \\tempest.api.identity.admin.v3.test_domains.DomainsTestJSON

POD 2

ID Error Associated tests Comments
1 tempest_lib.exceptions.BadRequest: Bad request
Details: {u'message': u'Multiple possible networks found, use a Network ID to be more specific.', u'code': 400}
tempest.api.volume.test_volumes_actions.VolumesV2ActionsTest
tempest.api.volume.test_volumes_actions.VolumesV1ActionsTest
tempest.scenario.test_volume_boot_pattern.TestVolumeBootPattern.test_volume_boot_pattern
tempest.scenario.test_volume_boot_pattern.TestVolumeBootPatternV2.test_volume_boot_pattern
tempest.scenario.test_server_basic_ops.TestServerBasicOps.test_server_basicops
2 tempest_lib.exceptions.Conflict: An object with that identifier already exists
Details: {u'message': u"Quota exceeded for resources: ['network']", u'type': u'OverQuota', u'detail': u''}
tempest.api.network.test_networks.NetworksIpV6TestJSON.test_create_update_delete_network_subnet
tempest.api.network.test_networks.BulkNetworkOpsIpV6TestJSON.test_bulk_create_delete_subnet
tempest.api.network.test_floating_ips.FloatingIPTestJSON
tempest.api.network.test_ports.PortsIpV6TestJSON.test_create_port_in_allowed_allocation_pools
tempest.api.network.test_ports.PortsIpV6TestJSON.test_create_port_with_no_securitygroups
tempest.api.network.test_routers.RoutersIpV6Test.test_add_remove_router_interface_with_subnet_id
3 ValueError: need more than 1 value to unpack
Response - Headers: {'date': 'Tue, 12 May 2015 13:05:08 GMT', 'status': '204', 'connection': 'close', 'content-length': '0', 'x-openstack-request-id': 'req-08ba5b6f-47a8-4c0b-adda-50b4dc2fc920'}
tempest.scenario.test_network_basic_ops.TestNetworkBasicOps.test_network_basic_ops
4 tempest_lib.exceptions.UnexpectedResponseCode: Unexpected response code received
Details: 503
NeutronError": {"message": "Unable to create the network. No available network found in maximum allowed attempts."
tempest.api.network.test_routers.RoutersIpV6Test.test_add_multiple_router_interfaces
tempest.api.network.test_networks.BulkNetworkOpsTestJSON.test_bulk_create_delete_subnet tempest.api.network.test_networks.BulkNetworkOpsTestJSON.test_bulk_create_delete_port
tempest.api.network.test_networks.BulkNetworkOpsIpV6TestJSON.test_bulk_create_delete_network
5 tempest.exceptions.InvalidConfiguration: Invalid Configuration
Details: Not enough users available for multi-user testing
tempest.api.compute.security_groups.test_security_group_rules.SecurityGroupRulesTestJSON
tempest.api.compute.security_groups.test_security_groups.SecurityGroupsTestJSON
tempest.api.compute.servers.test_server_actions.ServerActionsTestJSON
tempest.api.compute.servers.test_create_server.ServersTestJSON
tempest.api.compute.flavors.test_flavors.FlavorsV2TestJSON
tempest.api.compute.servers.test_create_server.ServersTestManualDisk
tempest.api.compute.servers.test_attach_interfaces.AttachInterfacesTestJSON
tempest.api.compute.servers.test_server_addresses.ServerAddressesTestJSON
6 tempest_lib.exceptions.ServerFault: Got server fault
Details: create_port_postcommit failed.
tempest.api.network.test_routers.RoutersIpV6Test.test_add_remove_router_interface_with_port_id
7 tempest_lib.exceptions.UnexpectedResponseCode: Unexpected response code received
Details: 503
tempest.api.network.test_networks.NetworksTestJSON
9 Invalid credentials tempest.api.identity.admin.v3.test_regions.RegionsTestJSON
tempest.api.identity.admin.v3.test_endpoints.EndPointsTestJSON
tempest.api.identity.admin.v3.test_groups.GroupsV3TestJSON
tempest.api.identity.admin.v3.test_trusts.TrustsV3TestJSO
tempest.api.identity.admin.v3.test_roles.RolesV3TestJSON
tempest.api.identity.admin.v3.test_policies.PoliciesTestJSON
tempest.api.identity.admin.v3.test_credentials.CredentialsTestJSON
tempest.api.identity.admin.v3.test_services.ServicesTestJSON \\tempest.api.identity.admin.v3.test_domains.DomainsTestJSON

Error follow up

Issue ID Category ID Solution/workaround
1 #3 or #1 These tests do not precise the network when creating instance. it automatically retrieves the first (and unique) network. If several networks have been created, the test is failed. http://lists.openstack.org/pipermail/openstack/2014-August/008736.html. We should either precise the network (and complete test coding) or ensure that there is only 1 possible network. to be tested with only 1 available network..
2 #3 The default value of network seems to be 10 (default value). This value shall be exceeded and triggers errors (parallel processing) (http://eavesdrop.openstack.org/irclogs/%23openstack-lbaas/%23openstack-lbaas.2015-04-29.log.html), either increase default value and or review cleaning
3 #3 or #1 In many cases, the tests scenario tries to retrieve values from tables that may not exist (python error). These tables concern network and/or floating ip creation. This test will pass if issues ID #1 & #2 are solved.
4 #3? linked to netwok (only on POD2)
5 #3? linked to netwok (only on POD2)
6 #3? linked to netwok (only on POD2)
7 ?
8 ?
9 ? appeared 22/5 linked to identity.admin.v3

references

r1_tempest.1432288735.txt.gz · Last modified: 2015/05/22 09:58 by Morgan Richomme