Hello,
I've got a an issue similar to the one discussed at
https://github.com/jtriley/StarCluster/issues/270. The full output from
my terminate commandline is:
starcluster -c config terminate smc2.0
StarCluster - (
http://star.mit.edu/cluster) (v. 0.95.5)
Software Tools for Academics and Researchers (STAR)
Please submit bug reports to starcluster_at_mit.edu
Terminate EBS cluster smc2.0 (y/n)? y
2014-10-29 14:32:50,042 >>> Running plugin speechmatics.CustomSetup
2014-10-29 14:32:50,127 >>> Running plugin
starcluster.plugins.users.CreateUsers
2014-10-29 14:32:50,256 >>> Running plugin starcluster.plugins.sge.SGEPlugin
2014-10-29 14:32:50,324 >>> Running plugin
starcluster.clustersetup.DefaultClusterSetup
2014-10-29 14:32:50,573 >>> Detaching volume vol-f75924f6 from master
2014-10-29 14:32:50,865 >>> Terminating node: master (i-5c350c57)
2014-10-29 14:32:51,116 >>> Waiting for cluster to terminate...
2014-10-29 14:32:52,345 >>> Removing placement group: _at_sc-smc2.0
2014-10-29 14:34:19,758 >>> Removing security group: _at_sc-smc2.0
2014-10-29 14:39:28,140 !!! ERROR - Failed to terminate cluster!
Traceback (most recent call last):
File
"/speechmatics/venv/local/lib/python2.7/site-packages/starcluster/commands/terminate.py",
line 87, in terminate
self._terminate_cluster(cl)
File
"/speechmatics/venv/local/lib/python2.7/site-packages/starcluster/commands/terminate.py",
line 64, in _terminate_cluster
cl.terminate_cluster()
File
"/speechmatics/venv/local/lib/python2.7/site-packages/starcluster/cluster.py",
line 1603, in terminate_cluster
self.ec2.delete_group(sg)
File
"/speechmatics/venv/local/lib/python2.7/site-packages/starcluster/awsutils.py",
line 268, in delete_group
ret_val = group.delete()
File
"/speechmatics/venv/local/lib/python2.7/site-packages/boto/ec2/securitygroup.py",
line 90, in delete
dry_run=dry_run
File
"/speechmatics/venv/local/lib/python2.7/site-packages/boto/ec2/connection.py",
line 3004, in delete_security_group
return self.get_status('DeleteSecurityGroup', params, verb='POST')
File
"/speechmatics/venv/local/lib/python2.7/site-packages/boto/connection.py",
line 1196, in get_status
raise self.ResponseError(response.status, response.reason, body)
EC2ResponseError: EC2ResponseError: 400 Bad Request
<?xml version="1.0" encoding="UTF-8"?>
<Response><Errors><Error><Code>DependencyViolation</Code><Message>resource
sg-f7079392 has a dependent
object</Message></Error></Errors><RequestID>2c366b92-a46d-4113-90f2-ed62d1fa3f9d</RequestID></Response>
2014-10-29 14:39:28,141 !!! ERROR - Use -f to forcefully terminate the
cluster
2014-10-29 14:39:28,141 !!! ERROR - DependencyViolation: resource
sg-f7079392 has a dependent object
Traceback (most recent call last):
File
"/speechmatics/venv/local/lib/python2.7/site-packages/starcluster/cli.py",
line 274, in main
sc.execute(args)
File
"/speechmatics/venv/local/lib/python2.7/site-packages/starcluster/commands/terminate.py",
line 101, in execute
self.terminate(cluster_name, force=self.opts.force)
File
"/speechmatics/venv/local/lib/python2.7/site-packages/starcluster/commands/terminate.py",
line 87, in terminate
self._terminate_cluster(cl)
File
"/speechmatics/venv/local/lib/python2.7/site-packages/starcluster/commands/terminate.py",
line 64, in _terminate_cluster
cl.terminate_cluster()
File
"/speechmatics/venv/local/lib/python2.7/site-packages/starcluster/cluster.py",
line 1603, in terminate_cluster
self.ec2.delete_group(sg)
File
"/speechmatics/venv/local/lib/python2.7/site-packages/starcluster/awsutils.py",
line 268, in delete_group
ret_val = group.delete()
File
"/speechmatics/venv/local/lib/python2.7/site-packages/boto/ec2/securitygroup.py",
line 90, in delete
dry_run=dry_run
File
"/speechmatics/venv/local/lib/python2.7/site-packages/boto/ec2/connection.py",
line 3004, in delete_security_group
return self.get_status('DeleteSecurityGroup', params, verb='POST')
File
"/speechmatics/venv/local/lib/python2.7/site-packages/boto/connection.py",
line 1196, in get_status
raise self.ResponseError(response.status, response.reason, body)
EC2ResponseError: EC2ResponseError: 400 Bad Request
<?xml version="1.0" encoding="UTF-8"?>
<Response><Errors><Error><Code>DependencyViolation</Code><Message>resource
sg-f7079392 has a dependent
object</Message></Error></Errors><RequestID>2c366b92-a46d-4113-90f2-ed62d1fa3f9d</RequestID></Response>
As the post at
https://awssolutions.wordpress.com/2012/10/17/deleting-aws-security-group-error-has-a-dependant-job/
suggested I went to network interfaces tab in AWS EC2 web console and
there were a number of network interfaces associated with security group
_at_sc-smc2.0. These network interfaces were not associated with any
instance. Only after removing the interfaces associated with this
security group the terminate command (with -f) removed the security
group successfully. It does not seem that any amount was going to fix
the issue. I had only master running for the last day and there were
about five network interfaces that I had to remove by hand.
Hope this may help to identify the underlying problem. This issue has
not happened before when terminating a cluster.
Best regards,
David
Received on Wed Oct 29 2014 - 11:14:59 EDT