CSIT Sporadic failures - tempest.scenario.test_server_basic_ops.TestServerBasicOps.test_server_basic_ops
Description
Environment
Operating System: All
Platform: All
Activity
JamO Luhrsen November 1, 2017 at 11:48 PM
looks like pike needs an extra config in tempest.conf for this test to pass. here is a patch
to try for that. once merged and we see this test case passing reliably we can close this
issue:
JamO Luhrsen October 24, 2017 at 4:36 AM
I have been working on a patch that will pause the tempest test cases before teardown to collect debugs. I have something
in the sandbox for this bug that may help. See here:
that should take you to this specific test_server_basic_ops test case. If you drill down in to it you'll see two keywords
OpenStackOperations . Show Debugs
OpenStackOperations . Get Test Teardown Debugs
Those are executed at the point where the test has failed, but before tempest does it's clean up. You can find
some better debugs/details in there (flow dumps, openstack cli outputs, etc)
this is all in the sandbox and that log will be gone in aprox 5 days from now. Eventually my patch will be done and
we'll see this running in releng where the logs stay for 6 months.
Chetan October 7, 2017 at 12:09 AMEdited
From the logs, it looks more like a openstack issue(not an odl issue). The error been thrown during Floating IP creation(neutron floating-ip create) in this use-case. From q-svc logs, I see there is no floating-ip JSON request been sent to ODL.
Since for the full tempest suite, the external-network and external-subnet been created earlier, and floating-ip been are carved out from this subnet. Strange to see only in this TC, fip creation fails and been successful with other tempest run.
I do hit with this same issue locally with Pike + CarbonSR2 while other tempest TC involving FIP creation getting passed.
JamO Luhrsen September 27, 2017 at 7:12 PM
another example [0] with it's relevant karaf log [1], neutron server log [2] and tempest log [3]
[0] https://logs.opendaylight.org/releng/jenkins092/netvirt-csit-1node-openstack-pike-upstream-stateful-carbon/27/log.html.gz#s1-s2-s1-t16
[1] https://logs.opendaylight.org/releng/jenkins092/netvirt-csit-1node-openstack-pike-upstream-stateful-carbon/27/odl1_karaf.log.gz
[2] https://logs.opendaylight.org/releng/jenkins092/netvirt-csit-1node-openstack-pike-upstream-stateful-carbon/27/control_1/q-svc.log.2017-09-27-132116.gz
[3] https://logs.opendaylight.org/releng/jenkins092/netvirt-csit-1node-openstack-pike-upstream-stateful-carbon/27/tempest/tempest_output_tempest.scenario.test_server_basic_ops.TestServerBasicOps.test_server_basic_ops.log.gz
looks like a FIP was not found in this job:
https://logs.opendaylight.org/releng/jenkins092/netvirt-csit-1node-openstack-pike-upstream-stateful-nitrogen/3/log.html.gz
tempest snippet:
tempest.scenario.test_server_basic_ops.TestServerBasicOps.test_server_basic_ops[compute,id-7fff3fb3-91d8-4fd0-bd7d-0204f1f180ba,network,smoke]
----------------------------------------------------------------------------------------------------------------------------------------------
Captured traceback:
~~~~~~~~~~~~~~~~~~~
Traceback (most recent call last):
File "tempest/common/utils/_init_.py", line 89, in wrapper
return f(self, *func_args, **func_kwargs)
File "tempest/scenario/test_server_basic_ops.py", line 145, in test_server_basic_ops
self.verify_ssh(keypair)
File "tempest/scenario/test_server_basic_ops.py", line 60, in verify_ssh
self.fip = self.create_floating_ip(self.instance)['ip']
File "tempest/scenario/manager.py", line 624, in create_floating_ip
create_floating_ip(pool=pool_name)['floating_ip'])
File "tempest/lib/services/compute/floating_ips_client.py", line 65, in create_floating_ip
resp, body = self.post(url, post_body)
File "tempest/lib/common/rest_client.py", line 270, in post
return self.request('POST', url, extra_headers, headers, body, chunked)
File "tempest/lib/services/compute/base_compute_client.py", line 48, in request
method, url, extra_headers, headers, body, chunked)
File "tempest/lib/common/rest_client.py", line 659, in request
self._error_checker(resp, resp_body)
File "tempest/lib/common/rest_client.py", line 765, in _error_checker
raise exceptions.NotFound(resp_body, resp=resp)
tempest.lib.exceptions.NotFound: Object not found
Details: {u'message': u'Floating IP pool not found.', u'code': 404}