failed to add/remove additional nodes

From: Ryan Golhar <no email>
Date: Thu, 3 Apr 2014 14:32:45 -0400

Hi all - I have a 50 node spot cluster running. I tried to add 10
additional nodes and at some point along the way it failed. Only 2 nodes
were added to the cluster, but they aren't getting SGE jobs. I tried
re-adding the nodes using '-x -a' but it fails. So I then tried to remove
the nodes, and that is failing as well. How do I fix this? Here's the
output:

[ec2-user_at_awsmicro plugins]$ starcluster removenode ngscluster node060

StarCluster - (http://star.mit.edu/cluster) (v. 0.9999)

Software Tools for Academics and Researchers (STAR)

Please submit bug reports to starcluster_at_mit.edu


>>> Running plugin tagger.TaggerPlugin

>>> Running plugin setupuserenv.SetupUserEnvironment

>>> Running plugin starcluster.plugins.users.CreateUsers

>>> Running plugin starcluster.plugins.sge.SGEPlugin

>>> Removing node060 from SGE

!!! ERROR - Error occured while running plugin
'starcluster.plugins.sge.SGEPlugin':

!!! ERROR - remote command 'source /etc/profile && qconf -dattr

!!! ERROR - hostgroup hostlist node060 _at_allhosts' failed with status 1:

!!! ERROR - error writing object "_at_allhosts" to spooling database


At this point, I have to go into the AWS web console and remove the nodes
myself as starcluster isn't able to.
Received on Thu Apr 03 2014 - 14:32:47 EDT