remove and add node errors
Hi Justin,
I'm dynamically resizing a cluster using the addnode and remove node and errors are frequently thrown. I then have to manually remove the node from /etc/hosts and also manually terminate the node in the AWS console. The attached bug report was generated from 0.93.1, but I'm seeing the same errors with 0.93.3. it looks to be an SSH authentication failure when accessing nodes, presumably to edit /etc/hosts?
thanks ./w
---------- CRASH DETAILS ----------
COMMAND: starcluster -c mycluster.config rn mycluster node028
2012-05-04 21:05:34,033 PID: 17610 config.py:551 - DEBUG - Loading config
2012-05-04 21:05:34,033 PID: 17610 config.py:118 - DEBUG - Loading file: mycluster.config
2012-05-04 21:05:34,057 PID: 17610 awsutils.py:54 - DEBUG - creating self._conn w/ connection_authenticator kwargs = {'proxy_user': None, 'proxy_pass': None, 'proxy_port': None, 'proxy': None, 'is_secure': True, 'path': '/', 'region': None, 'port': None}
2012-05-04 21:05:34,454 PID: 17610 cluster.py:665 - DEBUG - existing nodes: {}
2012-05-04 21:05:34,454 PID: 17610 cluster.py:673 - DEBUG - adding node i-c532d0a3 to self._nodes list
2012-05-04 21:05:34,454 PID: 17610 cluster.py:673 - DEBUG - adding node i-35749853 to self._nodes list
2012-05-04 21:05:34,454 PID: 17610 cluster.py:673 - DEBUG - adding node i-33749855 to self._nodes list
2012-05-04 21:05:34,455 PID: 17610 cluster.py:673 - DEBUG - adding node i-31749857 to self._nodes list
2012-05-04 21:05:34,474 PID: 17610 cluster.py:673 - DEBUG - adding node i-3f749859 to self._nodes list
2012-05-04 21:05:34,474 PID: 17610 cluster.py:673 - DEBUG - adding node i-3d74985b to self._nodes list
2012-05-04 21:05:34,475 PID: 17610 cluster.py:673 - DEBUG - adding node i-3b74985d to self._nodes list
2012-05-04 21:05:34,475 PID: 17610 cluster.py:673 - DEBUG - adding node i-3974985f to self._nodes list
2012-05-04 21:05:34,475 PID: 17610 cluster.py:673 - DEBUG - adding node i-07749861 to self._nodes list
2012-05-04 21:05:34,475 PID: 17610 cluster.py:673 - DEBUG - adding node i-05749863 to self._nodes list
2012-05-04 21:05:34,475 PID: 17610 cluster.py:673 - DEBUG - adding node i-03749865 to self._nodes list
2012-05-04 21:05:34,475 PID: 17610 cluster.py:673 - DEBUG - adding node i-01749867 to self._nodes list
2012-05-04 21:05:34,476 PID: 17610 cluster.py:673 - DEBUG - adding node i-0f749869 to self._nodes list
2012-05-04 21:05:34,476 PID: 17610 cluster.py:673 - DEBUG - adding node i-0d74986b to self._nodes list
2012-05-04 21:05:34,476 PID: 17610 cluster.py:673 - DEBUG - adding node i-0b74986d to self._nodes list
2012-05-04 21:05:34,476 PID: 17610 cluster.py:673 - DEBUG - adding node i-0974986f to self._nodes list
2012-05-04 21:05:34,476 PID: 17610 cluster.py:673 - DEBUG - adding node i-17749871 to self._nodes list
2012-05-04 21:05:34,476 PID: 17610 cluster.py:673 - DEBUG - adding node i-15749873 to self._nodes list
2012-05-04 21:05:34,476 PID: 17610 cluster.py:673 - DEBUG - adding node i-13749875 to self._nodes list
2012-05-04 21:05:34,477 PID: 17610 cluster.py:673 - DEBUG - adding node i-11749877 to self._nodes list
2012-05-04 21:05:34,477 PID: 17610 cluster.py:673 - DEBUG - adding node i-1f749879 to self._nodes list
2012-05-04 21:05:34,477 PID: 17610 cluster.py:673 - DEBUG - adding node i-1d74987b to self._nodes list
2012-05-04 21:05:34,477 PID: 17610 cluster.py:673 - DEBUG - adding node i-1b74987d to self._nodes list
2012-05-04 21:05:34,477 PID: 17610 cluster.py:673 - DEBUG - adding node i-1974987f to self._nodes list
2012-05-04 21:05:34,478 PID: 17610 cluster.py:673 - DEBUG - adding node i-e7749881 to self._nodes list
2012-05-04 21:05:34,478 PID: 17610 cluster.py:673 - DEBUG - adding node i-e5749883 to self._nodes list
2012-05-04 21:05:34,478 PID: 17610 cluster.py:673 - DEBUG - adding node i-e3749885 to self._nodes list
2012-05-04 21:05:34,478 PID: 17610 cluster.py:673 - DEBUG - adding node i-e1749887 to self._nodes list
2012-05-04 21:05:34,478 PID: 17610 cluster.py:673 - DEBUG - adding node i-ef749889 to self._nodes list
2012-05-04 21:05:34,478 PID: 17610 cluster.py:681 - DEBUG - returning self._nodes = [<Node: master (i-c532d0a3)>, <Node: node001 (i-35749853)>, <Node: node002 (i-33749855)>, <Node: node003 (i-31749857)>, <Node: node004 (i-3f749859)>, <Node: node005 (i-3d74985b)>, <Node: node006 (i-3b74985d)>, <Node: node007 (i-3974985f)>, <Node: node008 (i-07749861)>, <Node: node009 (i-05749863)>, <Node: node010 (i-03749865)>, <Node: node011 (i-01749867)>, <Node: node012 (i-0f749869)>, <Node: node013 (i-0d74986b)>, <Node: node014 (i-0b74986d)>, <Node: node015 (i-0974986f)>, <Node: node016 (i-17749871)>, <Node: node017 (i-15749873)>, <Node: node018 (i-13749875)>, <Node: node019 (i-11749877)>, <Node: node020 (i-1f749879)>, <Node: node021 (i-1d74987b)>, <Node: node022 (i-1b74987d)>, <Node: node023 (i-1974987f)>, <Node: node024 (i-e7749881)>, <Node: node025 (i-e5749883)>, <Node: node026 (i-e3749885)>, <Node: node027 (i-e1749887)>, <Node: node028 (i-ef749889)>]
2012-05-04 21:05:34,554 PID: 17610 cluster.py:665 - DEBUG - existing nodes: {u'i-35749853': <Node: node001 (i-35749853)>, u'i-33749855': <Node: node002 (i-33749855)>, u'i-07749861': <Node: node008 (i-07749861)>, u'i-0974986f': <Node: node015 (i-0974986f)>, u'i-ef749889': <Node: node028 (i-ef749889)>, u'i-c532d0a3': <Node: master (i-c532d0a3)>, u'i-0b74986d': <Node: node014 (i-0b74986d)>, u'i-e3749885': <Node: node026 (i-e3749885)>, u'i-1b74987d': <Node: node022 (i-1b74987d)>, u'i-0d74986b': <Node: node013 (i-0d74986b)>, u'i-e1749887': <Node: node027 (i-e1749887)>, u'i-1f749879': <Node: node020 (i-1f749879)>, u'i-3d74985b': <Node: node005 (i-3d74985b)>, u'i-3974985f': <Node: node007 (i-3974985f)>, u'i-15749873': <Node: node017 (i-15749873)>, u'i-3b74985d': <Node: node006 (i-3b74985d)>, u'i-11749877': <Node: node019 (i-11749877)>, u'i-1d74987b': <Node: node021 (i-1d74987b)>, u'i-1974987f': <Node: node023 (i-1974987f)>, u'i-05749863': <Node: node009 (i-05749863)>, u'i-17749871': <Node: node016 (i-17749871)>, u'i-3f749859': <Node: node004 (i-3f749859)>, u'i-e5749883': <Node: node025 (i-e5749883)>, u'i-13749875': <Node: node018 (i-13749875)>, u'i-0f749869': <Node: node012 (i-0f749869)>, u'i-31749857': <Node: node003 (i-31749857)>, u'i-01749867': <Node: node011 (i-01749867)>, u'i-03749865': <Node: node010 (i-03749865)>, u'i-e7749881': <Node: node024 (i-e7749881)>}
2012-05-04 21:05:34,554 PID: 17610 cluster.py:668 - DEBUG - updating existing node i-c532d0a3 in self._nodes
2012-05-04 21:05:34,554 PID: 17610 cluster.py:668 - DEBUG - updating existing node i-35749853 in self._nodes
2012-05-04 21:05:34,554 PID: 17610 cluster.py:668 - DEBUG - updating existing node i-33749855 in self._nodes
2012-05-04 21:05:34,554 PID: 17610 cluster.py:668 - DEBUG - updating existing node i-31749857 in self._nodes
2012-05-04 21:05:34,554 PID: 17610 cluster.py:668 - DEBUG - updating existing node i-3f749859 in self._nodes
2012-05-04 21:05:34,554 PID: 17610 cluster.py:668 - DEBUG - updating existing node i-3d74985b in self._nodes
2012-05-04 21:05:34,555 PID: 17610 cluster.py:668 - DEBUG - updating existing node i-3b74985d in self._nodes
2012-05-04 21:05:34,555 PID: 17610 cluster.py:668 - DEBUG - updating existing node i-3974985f in self._nodes
2012-05-04 21:05:34,555 PID: 17610 cluster.py:668 - DEBUG - updating existing node i-07749861 in self._nodes
2012-05-04 21:05:34,555 PID: 17610 cluster.py:668 - DEBUG - updating existing node i-05749863 in self._nodes
2012-05-04 21:05:34,555 PID: 17610 cluster.py:668 - DEBUG - updating existing node i-03749865 in self._nodes
2012-05-04 21:05:34,555 PID: 17610 cluster.py:668 - DEBUG - updating existing node i-01749867 in self._nodes
2012-05-04 21:05:34,555 PID: 17610 cluster.py:668 - DEBUG - updating existing node i-0f749869 in self._nodes
2012-05-04 21:05:34,556 PID: 17610 cluster.py:668 - DEBUG - updating existing node i-0d74986b in self._nodes
2012-05-04 21:05:34,556 PID: 17610 cluster.py:668 - DEBUG - updating existing node i-0b74986d in self._nodes
2012-05-04 21:05:34,556 PID: 17610 cluster.py:668 - DEBUG - updating existing node i-0974986f in self._nodes
2012-05-04 21:05:34,556 PID: 17610 cluster.py:668 - DEBUG - updating existing node i-17749871 in self._nodes
2012-05-04 21:05:34,556 PID: 17610 cluster.py:668 - DEBUG - updating existing node i-15749873 in self._nodes
2012-05-04 21:05:34,556 PID: 17610 cluster.py:668 - DEBUG - updating existing node i-13749875 in self._nodes
2012-05-04 21:05:34,556 PID: 17610 cluster.py:668 - DEBUG - updating existing node i-11749877 in self._nodes
2012-05-04 21:05:34,557 PID: 17610 cluster.py:668 - DEBUG - updating existing node i-1f749879 in self._nodes
2012-05-04 21:05:34,557 PID: 17610 cluster.py:668 - DEBUG - updating existing node i-1d74987b in self._nodes
2012-05-04 21:05:34,557 PID: 17610 cluster.py:668 - DEBUG - updating existing node i-1b74987d in self._nodes
2012-05-04 21:05:34,557 PID: 17610 cluster.py:668 - DEBUG - updating existing node i-1974987f in self._nodes
2012-05-04 21:05:34,557 PID: 17610 cluster.py:668 - DEBUG - updating existing node i-e7749881 in self._nodes
2012-05-04 21:05:34,557 PID: 17610 cluster.py:668 - DEBUG - updating existing node i-e5749883 in self._nodes
2012-05-04 21:05:34,557 PID: 17610 cluster.py:668 - DEBUG - updating existing node i-e3749885 in self._nodes
2012-05-04 21:05:34,557 PID: 17610 cluster.py:668 - DEBUG - updating existing node i-e1749887 in self._nodes
2012-05-04 21:05:34,558 PID: 17610 cluster.py:668 - DEBUG - updating existing node i-ef749889 in self._nodes
2012-05-04 21:05:34,558 PID: 17610 cluster.py:681 - DEBUG - returning self._nodes = [<Node: master (i-c532d0a3)>, <Node: node001 (i-35749853)>, <Node: node002 (i-33749855)>, <Node: node003 (i-31749857)>, <Node: node004 (i-3f749859)>, <Node: node005 (i-3d74985b)>, <Node: node006 (i-3b74985d)>, <Node: node007 (i-3974985f)>, <Node: node008 (i-07749861)>, <Node: node009 (i-05749863)>, <Node: node010 (i-03749865)>, <Node: node011 (i-01749867)>, <Node: node012 (i-0f749869)>, <Node: node013 (i-0d74986b)>, <Node: node014 (i-0b74986d)>, <Node: node015 (i-0974986f)>, <Node: node016 (i-17749871)>, <Node: node017 (i-15749873)>, <Node: node018 (i-13749875)>, <Node: node019 (i-11749877)>, <Node: node020 (i-1f749879)>, <Node: node021 (i-1d74987b)>, <Node: node022 (i-1b74987d)>, <Node: node023 (i-1974987f)>, <Node: node024 (i-e7749881)>, <Node: node025 (i-e5749883)>, <Node: node026 (i-e3749885)>, <Node: node027 (i-e1749887)>, <Node: node028 (i-ef749889)>]
2012-05-04 21:05:34,558 PID: 17610 clustersetup.py:496 - INFO - Removing node node028 (i-ef749889)...
2012-05-04 21:05:34,558 PID: 17610 clustersetup.py:498 - INFO - Removing node028 from SGE
2012-05-04 21:05:34,559 PID: 17610 ssh.py:75 - DEBUG - loading private key /vol/sites/shared/ssh/mycluster.rsa
2012-05-04 21:05:34,559 PID: 17610 ssh.py:160 - DEBUG - Using private key /vol/sites/shared/ssh/mycluster.rsa (rsa)
2012-05-04 21:05:34,559 PID: 17610 ssh.py:97 - DEBUG - connecting to host ec2-107-20-104-139.compute-1.amazonaws.com on port 22 as user root
2012-05-04 21:05:47,013 PID: 17610 ssh.py:179 - DEBUG - creating sftp connection
2012-05-04 21:05:47,868 PID: 17610 ssh.py:536 - DEBUG - root_at_master modified "@allhosts" in host group list
2012-05-04 21:05:48,024 PID: 17610 ssh.py:536 - DEBUG - root_at_master modified "all.q" in cluster queue list
2012-05-04 21:05:48,104 PID: 17610 ssh.py:536 - DEBUG - root_at_master removed "node028" from configuration list
2012-05-04 21:05:48,183 PID: 17610 ssh.py:536 - DEBUG - root_at_master removed "node028" from execution host list
2012-05-04 21:05:48,183 PID: 17610 ssh.py:75 - DEBUG - loading private key /vol/sites/shared/ssh/mycluster.rsa
2012-05-04 21:05:48,184 PID: 17610 ssh.py:160 - DEBUG - Using private key /vol/sites/shared/ssh/mycluster.rsa (rsa)
2012-05-04 21:05:48,184 PID: 17610 ssh.py:97 - DEBUG - connecting to host ec2-50-16-114-124.compute-1.amazonaws.com on port 22 as user root
2012-05-04 21:06:03,597 PID: 17610 clustersetup.py:348 - INFO - Updating SGE parallel environment 'orte'
2012-05-04 21:06:03,601 PID: 17610 ssh.py:75 - DEBUG - loading private key /vol/sites/shared/ssh/mycluster.rsa
2012-05-04 21:06:03,602 PID: 17610 ssh.py:160 - DEBUG - Using private key /vol/sites/shared/ssh/mycluster.rsa (rsa)
2012-05-04 21:06:03,602 PID: 17610 ssh.py:97 - DEBUG - connecting to host ec2-184-72-178-185.compute-1.amazonaws.com on port 22 as user root
2012-05-04 21:06:03,602 PID: 17610 ssh.py:75 - DEBUG - loading private key /vol/sites/shared/ssh/mycluster.rsa
2012-05-04 21:06:03,603 PID: 17610 ssh.py:160 - DEBUG - Using private key /vol/sites/shared/ssh/mycluster.rsa (rsa)
2012-05-04 21:06:03,603 PID: 17610 ssh.py:97 - DEBUG - connecting to host ec2-23-20-152-32.compute-1.amazonaws.com on port 22 as user root
2012-05-04 21:06:03,603 PID: 17610 ssh.py:75 - DEBUG - loading private key /vol/sites/shared/ssh/mycluster.rsa
2012-05-04 21:06:03,604 PID: 17610 ssh.py:160 - DEBUG - Using private key /vol/sites/shared/ssh/mycluster.rsa (rsa)
2012-05-04 21:06:03,604 PID: 17610 ssh.py:97 - DEBUG - connecting to host ec2-174-129-82-72.compute-1.amazonaws.com on port 22 as user root
2012-05-04 21:06:03,605 PID: 17610 ssh.py:75 - DEBUG - loading private key /vol/sites/shared/ssh/mycluster.rsa
2012-05-04 21:06:03,605 PID: 17610 ssh.py:160 - DEBUG - Using private key /vol/sites/shared/ssh/mycluster.rsa (rsa)
2012-05-04 21:06:03,605 PID: 17610 ssh.py:97 - DEBUG - connecting to host ec2-184-72-80-161.compute-1.amazonaws.com on port 22 as user root
2012-05-04 21:06:03,606 PID: 17610 ssh.py:75 - DEBUG - loading private key /vol/sites/shared/ssh/mycluster.rsa
2012-05-04 21:06:03,630 PID: 17610 ssh.py:75 - DEBUG - loading private key /vol/sites/shared/ssh/mycluster.rsa
2012-05-04 21:06:03,631 PID: 17610 ssh.py:160 - DEBUG - Using private key /vol/sites/shared/ssh/mycluster.rsa (rsa)
2012-05-04 21:06:03,631 PID: 17610 ssh.py:97 - DEBUG - connecting to host ec2-107-22-132-97.compute-1.amazonaws.com on port 22 as user root
2012-05-04 21:06:03,632 PID: 17610 ssh.py:75 - DEBUG - loading private key /vol/sites/shared/ssh/mycluster.rsa
2012-05-04 21:06:03,632 PID: 17610 ssh.py:160 - DEBUG - Using private key /vol/sites/shared/ssh/mycluster.rsa (rsa)
2012-05-04 21:06:03,635 PID: 17610 ssh.py:97 - DEBUG - connecting to host ec2-174-129-141-34.compute-1.amazonaws.com on port 22 as user root
2012-05-04 21:06:03,633 PID: 17610 ssh.py:75 - DEBUG - loading private key /vol/sites/shared/ssh/mycluster.rsa
2012-05-04 21:06:03,636 PID: 17610 ssh.py:160 - DEBUG - Using private key /vol/sites/shared/ssh/mycluster.rsa (rsa)
2012-05-04 21:06:03,633 PID: 17610 threadpool.py:135 - DEBUG - unfinished_tasks = 28
2012-05-04 21:06:03,633 PID: 17610 ssh.py:75 - DEBUG - loading private key /vol/sites/shared/ssh/mycluster.rsa
2012-05-04 21:06:03,634 PID: 17610 ssh.py:75 - DEBUG - loading private key /vol/sites/shared/ssh/mycluster.rsa
2012-05-04 21:06:03,634 PID: 17610 ssh.py:75 - DEBUG - loading private key /vol/sites/shared/ssh/mycluster.rsa
2012-05-04 21:06:03,634 PID: 17610 ssh.py:75 - DEBUG - loading private key /vol/sites/shared/ssh/mycluster.rsa
2012-05-04 21:06:03,634 PID: 17610 ssh.py:75 - DEBUG - loading private key /vol/sites/shared/ssh/mycluster.rsa
2012-05-04 21:06:03,634 PID: 17610 ssh.py:75 - DEBUG - loading private key /vol/sites/shared/ssh/mycluster.rsa
2012-05-04 21:06:03,635 PID: 17610 ssh.py:75 - DEBUG - loading private key /vol/sites/shared/ssh/mycluster.rsa
2012-05-04 21:06:03,659 PID: 17610 ssh.py:160 - DEBUG - Using private key /vol/sites/shared/ssh/mycluster.rsa (rsa)
2012-05-04 21:06:03,659 PID: 17610 ssh.py:97 - DEBUG - connecting to host ec2-174-129-128-158.compute-1.amazonaws.com on port 22 as user root
2012-05-04 21:06:03,635 PID: 17610 ssh.py:75 - DEBUG - loading private key /vol/sites/shared/ssh/mycluster.rsa
2012-05-04 21:06:03,660 PID: 17610 ssh.py:160 - DEBUG - Using private key /vol/sites/shared/ssh/mycluster.rsa (rsa)
2012-05-04 21:06:03,660 PID: 17610 ssh.py:97 - DEBUG - connecting to host ec2-23-22-1-205.compute-1.amazonaws.com on port 22 as user root
2012-05-04 21:06:03,635 PID: 17610 ssh.py:75 - DEBUG - loading private key /vol/sites/shared/ssh/mycluster.rsa
2012-05-04 21:06:03,633 PID: 17610 ssh.py:75 - DEBUG - loading private key /vol/sites/shared/ssh/mycluster.rsa
2012-05-04 21:06:03,662 PID: 17610 ssh.py:160 - DEBUG - Using private key /vol/sites/shared/ssh/mycluster.rsa (rsa)
2012-05-04 21:06:03,662 PID: 17610 ssh.py:97 - DEBUG - connecting to host ec2-23-20-76-33.compute-1.amazonaws.com on port 22 as user root
2012-05-04 21:06:03,638 PID: 17610 ssh.py:160 - DEBUG - Using private key /vol/sites/shared/ssh/mycluster.rsa (rsa)
2012-05-04 21:06:03,663 PID: 17610 ssh.py:97 - DEBUG - connecting to host ec2-23-20-184-64.compute-1.amazonaws.com on port 22 as user root
2012-05-04 21:06:03,633 PID: 17610 ssh.py:75 - DEBUG - loading private key /vol/sites/shared/ssh/mycluster.rsa
2012-05-04 21:06:03,664 PID: 17610 ssh.py:160 - DEBUG - Using private key /vol/sites/shared/ssh/mycluster.rsa (rsa)
2012-05-04 21:06:03,664 PID: 17610 ssh.py:97 - DEBUG - connecting to host ec2-50-19-31-63.compute-1.amazonaws.com on port 22 as user root
2012-05-04 21:06:03,636 PID: 17610 ssh.py:97 - DEBUG - connecting to host ec2-174-129-46-13.compute-1.amazonaws.com on port 22 as user root
2012-05-04 21:06:03,640 PID: 17610 ssh.py:160 - DEBUG - Using private key /vol/sites/shared/ssh/mycluster.rsa (rsa)
2012-05-04 21:06:03,664 PID: 17610 ssh.py:97 - DEBUG - connecting to host ec2-50-16-154-247.compute-1.amazonaws.com on port 22 as user root
2012-05-04 21:06:03,637 PID: 17610 ssh.py:160 - DEBUG - Using private key /vol/sites/shared/ssh/mycluster.rsa (rsa)
2012-05-04 21:06:03,665 PID: 17610 ssh.py:97 - DEBUG - connecting to host ec2-107-21-141-210.compute-1.amazonaws.com on port 22 as user root
2012-05-04 21:06:03,639 PID: 17610 ssh.py:160 - DEBUG - Using private key /vol/sites/shared/ssh/mycluster.rsa (rsa)
2012-05-04 21:06:03,665 PID: 17610 ssh.py:97 - DEBUG - connecting to host ec2-174-129-158-44.compute-1.amazonaws.com on port 22 as user root
2012-05-04 21:06:03,639 PID: 17610 ssh.py:160 - DEBUG - Using private key /vol/sites/shared/ssh/mycluster.rsa (rsa)
2012-05-04 21:06:03,665 PID: 17610 ssh.py:97 - DEBUG - connecting to host ec2-23-20-38-132.compute-1.amazonaws.com on port 22 as user root
2012-05-04 21:06:03,658 PID: 17610 ssh.py:160 - DEBUG - Using private key /vol/sites/shared/ssh/mycluster.rsa (rsa)
2012-05-04 21:06:03,666 PID: 17610 ssh.py:97 - DEBUG - connecting to host ec2-23-22-28-250.compute-1.amazonaws.com on port 22 as user root
2012-05-04 21:06:03,661 PID: 17610 ssh.py:160 - DEBUG - Using private key /vol/sites/shared/ssh/mycluster.rsa (rsa)
2012-05-04 21:06:03,666 PID: 17610 ssh.py:97 - DEBUG - connecting to host ec2-23-20-118-235.compute-1.amazonaws.com on port 22 as user root
2012-05-04 21:06:03,677 PID: 17610 ssh.py:160 - DEBUG - Using private key /vol/sites/shared/ssh/mycluster.rsa (rsa)
2012-05-04 21:06:03,683 PID: 17610 ssh.py:536 - DEBUG - 8
2012-05-04 21:06:03,685 PID: 17610 ssh.py:75 - DEBUG - loading private key /vol/sites/shared/ssh/mycluster.rsa
2012-05-04 21:06:03,685 PID: 17610 ssh.py:160 - DEBUG - Using private key /vol/sites/shared/ssh/mycluster.rsa (rsa)
2012-05-04 21:06:03,685 PID: 17610 ssh.py:97 - DEBUG - connecting to host ec2-184-73-14-134.compute-1.amazonaws.com on port 22 as user root
2012-05-04 21:06:03,686 PID: 17610 ssh.py:97 - DEBUG - connecting to host ec2-50-17-70-72.compute-1.amazonaws.com on port 22 as user root
2012-05-04 21:06:05,032 PID: 17610 threadpool.py:135 - DEBUG - unfinished_tasks = 27
2012-05-04 21:06:53,674 PID: 17610 threadpool.py:135 - DEBUG - unfinished_tasks = 27
2012-05-04 21:07:13,497 PID: 17610 threadpool.py:135 - DEBUG - unfinished_tasks = 27
2012-05-04 21:07:35,363 PID: 17610 threadpool.py:135 - DEBUG - unfinished_tasks = 27
2012-05-04 21:08:08,902 PID: 17610 threadpool.py:135 - DEBUG - unfinished_tasks = 27
2012-05-04 21:08:13,154 PID: 17610 ssh.py:75 - DEBUG - loading private key /vol/sites/shared/ssh/mycluster.rsa
2012-05-04 21:08:13,646 PID: 17610 ssh.py:75 - DEBUG - loading private key /vol/sites/shared/ssh/mycluster.rsa
2012-05-04 21:08:13,647 PID: 17610 ssh.py:160 - DEBUG - Using private key /vol/sites/shared/ssh/mycluster.rsa (rsa)
2012-05-04 21:08:13,647 PID: 17610 ssh.py:97 - DEBUG - connecting to host ec2-23-22-14-9.compute-1.amazonaws.com on port 22 as user root
2012-05-04 21:08:13,937 PID: 17610 ssh.py:160 - DEBUG - Using private key /vol/sites/shared/ssh/mycluster.rsa (rsa)
2012-05-04 21:08:16,725 PID: 17610 ssh.py:97 - DEBUG - connecting to host ec2-107-22-99-46.compute-1.amazonaws.com on port 22 as user root
2012-05-04 21:08:16,533 PID: 17610 ssh.py:75 - DEBUG - loading private key /vol/sites/shared/ssh/mycluster.rsa
2012-05-04 21:08:16,726 PID: 17610 ssh.py:160 - DEBUG - Using private key /vol/sites/shared/ssh/mycluster.rsa (rsa)
2012-05-04 21:08:16,727 PID: 17610 ssh.py:97 - DEBUG - connecting to host ec2-107-21-149-111.compute-1.amazonaws.com on port 22 as user root
2012-05-04 21:08:16,727 PID: 17610 threadpool.py:135 - DEBUG - unfinished_tasks = 24
2012-05-04 21:08:16,732 PID: 17610 ssh.py:75 - DEBUG - loading private key /vol/sites/shared/ssh/mycluster.rsa
2012-05-04 21:08:17,719 PID: 17610 ssh.py:75 - DEBUG - loading private key /vol/sites/shared/ssh/mycluster.rsa
2012-05-04 21:08:17,720 PID: 17610 ssh.py:160 - DEBUG - Using private key /vol/sites/shared/ssh/mycluster.rsa (rsa)
2012-05-04 21:08:17,720 PID: 17610 ssh.py:97 - DEBUG - connecting to host ec2-174-129-113-38.compute-1.amazonaws.com on port 22 as user root
2012-05-04 21:08:17,721 PID: 17610 ssh.py:75 - DEBUG - loading private key /vol/sites/shared/ssh/mycluster.rsa
2012-05-04 21:08:25,659 PID: 17610 ssh.py:160 - DEBUG - Using private key /vol/sites/shared/ssh/mycluster.rsa (rsa)
2012-05-04 21:08:25,660 PID: 17610 ssh.py:97 - DEBUG - connecting to host ec2-107-21-190-97.compute-1.amazonaws.com on port 22 as user root
2012-05-04 21:08:25,660 PID: 17610 ssh.py:536 - DEBUG - 8
2012-05-04 21:08:17,721 PID: 17610 ssh.py:160 - DEBUG - Using private key /vol/sites/shared/ssh/mycluster.rsa (rsa)
2012-05-04 21:08:25,660 PID: 17610 ssh.py:97 - DEBUG - connecting to host ec2-50-17-144-100.compute-1.amazonaws.com on port 22 as user root
2012-05-04 21:08:25,661 PID: 17610 ssh.py:536 - DEBUG - 8
2012-05-04 21:08:25,662 PID: 17610 threadpool.py:135 - DEBUG - unfinished_tasks = 12
2012-05-04 21:08:17,722 PID: 17610 ssh.py:75 - DEBUG - loading private key /vol/sites/shared/ssh/mycluster.rsa
2012-05-04 21:08:25,664 PID: 17610 ssh.py:160 - DEBUG - Using private key /vol/sites/shared/ssh/mycluster.rsa (rsa)
2012-05-04 21:08:25,664 PID: 17610 ssh.py:97 - DEBUG - connecting to host ec2-184-73-69-75.compute-1.amazonaws.com on port 22 as user root
2012-05-04 21:08:25,745 PID: 17610 ssh.py:536 - DEBUG - 8
2012-05-04 21:08:31,193 PID: 17610 threadpool.py:135 - DEBUG - unfinished_tasks = 9
2012-05-04 21:08:37,036 PID: 17610 threadpool.py:135 - DEBUG - unfinished_tasks = 9
2012-05-04 21:08:59,527 PID: 17610 threadpool.py:135 - DEBUG - unfinished_tasks = 9
2012-05-04 21:09:26,072 PID: 17610 ssh.py:536 - DEBUG - 8
2012-05-04 21:09:29,940 PID: 17610 threadpool.py:135 - DEBUG - unfinished_tasks = 8
2012-05-04 21:09:39,433 PID: 17610 ssh.py:536 - DEBUG - 8
2012-05-04 21:09:40,449 PID: 17610 threadpool.py:135 - DEBUG - unfinished_tasks = 7
2012-05-04 21:09:46,765 PID: 17610 threadpool.py:135 - DEBUG - unfinished_tasks = 7
2012-05-04 21:09:46,977 PID: 17610 ssh.py:536 - DEBUG - 8
2012-05-04 21:09:47,766 PID: 17610 threadpool.py:135 - DEBUG - unfinished_tasks = 6
2012-05-04 21:09:48,767 PID: 17610 threadpool.py:135 - DEBUG - unfinished_tasks = 6
2012-05-04 21:09:49,019 PID: 17610 ssh.py:536 - DEBUG - 8
2012-05-04 21:09:49,235 PID: 17610 ssh.py:536 - DEBUG - 8
2012-05-04 21:09:49,768 PID: 17610 threadpool.py:135 - DEBUG - unfinished_tasks = 4
2012-05-04 21:09:49,794 PID: 17610 ssh.py:536 - DEBUG - 8
2012-05-04 21:09:49,829 PID: 17610 ssh.py:536 - DEBUG - 8
2012-05-04 21:09:50,770 PID: 17610 threadpool.py:135 - DEBUG - unfinished_tasks = 2
2012-05-04 21:09:50,943 PID: 17610 ssh.py:536 - DEBUG - 8
2012-05-04 21:09:51,771 PID: 17610 threadpool.py:135 - DEBUG - unfinished_tasks = 1
2012-05-04 21:09:52,545 PID: 17610 ssh.py:536 - DEBUG - 8
2012-05-04 21:09:52,806 PID: 17610 cli.py:266 - DEBUG - error occurred in job (id=-1276658832): failed to connect to host ec2-50-16-154-247.compute-1.amazonaws.com on port 22
Traceback (most recent call last):
File "/usr/local/lib/python2.7/dist-packages/starcluster/threadpool.py", line 31, in run
job.run()
File "/usr/local/lib/python2.7/dist-packages/starcluster/threadpool.py", line 58, in run
r = self.method(*self.args, **self.kwargs)
File "/usr/local/lib/python2.7/dist-packages/starcluster/clustersetup.py", line 351, in <lambda>
num_processors = sum(self.pool.map(lambda n: n.num_processors, nodes))
File "/usr/local/lib/python2.7/dist-packages/starcluster/node.py", line 169, in num_processors
'cat /proc/cpuinfo | grep processor | wc -l')[0])
File "/usr/local/lib/python2.7/dist-packages/starcluster/ssh.py", line 512, in execute
channel = self.transport.open_session()
File "/usr/local/lib/python2.7/dist-packages/starcluster/ssh.py", line 129, in transport
port=self._port, timeout=self._timeout)
File "/usr/local/lib/python2.7/dist-packages/starcluster/ssh.py", line 115, in connect
raise exception.SSHConnectionError(host, port)
SSHConnectionError: failed to connect to host ec2-50-16-154-247.compute-1.amazonaws.com on port 22
error occurred in job (id=-1343800464): failed to connect to host ec2-174-129-141-34.compute-1.amazonaws.com on port 22
Traceback (most recent call last):
File "/usr/local/lib/python2.7/dist-packages/starcluster/threadpool.py", line 31, in run
job.run()
File "/usr/local/lib/python2.7/dist-packages/starcluster/threadpool.py", line 58, in run
r = self.method(*self.args, **self.kwargs)
File "/usr/local/lib/python2.7/dist-packages/starcluster/clustersetup.py", line 351, in <lambda>
num_processors = sum(self.pool.map(lambda n: n.num_processors, nodes))
File "/usr/local/lib/python2.7/dist-packages/starcluster/node.py", line 169, in num_processors
'cat /proc/cpuinfo | grep processor | wc -l')[0])
File "/usr/local/lib/python2.7/dist-packages/starcluster/ssh.py", line 512, in execute
channel = self.transport.open_session()
File "/usr/local/lib/python2.7/dist-packages/starcluster/ssh.py", line 129, in transport
port=self._port, timeout=self._timeout)
File "/usr/local/lib/python2.7/dist-packages/starcluster/ssh.py", line 115, in connect
raise exception.SSHConnectionError(host, port)
SSHConnectionError: failed to connect to host ec2-174-129-141-34.compute-1.amazonaws.com on port 22
error occurred in job (id=-1259873424): failed to connect to host ec2-23-20-38-132.compute-1.amazonaws.com on port 22
Traceback (most recent call last):
File "/usr/local/lib/python2.7/dist-packages/starcluster/threadpool.py", line 31, in run
job.run()
File "/usr/local/lib/python2.7/dist-packages/starcluster/threadpool.py", line 58, in run
r = self.method(*self.args, **self.kwargs)
File "/usr/local/lib/python2.7/dist-packages/starcluster/clustersetup.py", line 351, in <lambda>
num_processors = sum(self.pool.map(lambda n: n.num_processors, nodes))
File "/usr/local/lib/python2.7/dist-packages/starcluster/node.py", line 169, in num_processors
'cat /proc/cpuinfo | grep processor | wc -l')[0])
File "/usr/local/lib/python2.7/dist-packages/starcluster/ssh.py", line 512, in execute
channel = self.transport.open_session()
File "/usr/local/lib/python2.7/dist-packages/starcluster/ssh.py", line 129, in transport
port=self._port, timeout=self._timeout)
File "/usr/local/lib/python2.7/dist-packages/starcluster/ssh.py", line 115, in connect
raise exception.SSHConnectionError(host, port)
SSHConnectionError: failed to connect to host ec2-23-20-38-132.compute-1.amazonaws.com on port 22
error occurred in job (id=-1368978576): failed to connect to host ec2-50-17-70-72.compute-1.amazonaws.com on port 22
Traceback (most recent call last):
File "/usr/local/lib/python2.7/dist-packages/starcluster/threadpool.py", line 31, in run
job.run()
File "/usr/local/lib/python2.7/dist-packages/starcluster/threadpool.py", line 58, in run
r = self.method(*self.args, **self.kwargs)
File "/usr/local/lib/python2.7/dist-packages/starcluster/clustersetup.py", line 351, in <lambda>
num_processors = sum(self.pool.map(lambda n: n.num_processors, nodes))
File "/usr/local/lib/python2.7/dist-packages/starcluster/node.py", line 169, in num_processors
'cat /proc/cpuinfo | grep processor | wc -l')[0])
File "/usr/local/lib/python2.7/dist-packages/starcluster/ssh.py", line 512, in execute
channel = self.transport.open_session()
File "/usr/local/lib/python2.7/dist-packages/starcluster/ssh.py", line 129, in transport
port=self._port, timeout=self._timeout)
File "/usr/local/lib/python2.7/dist-packages/starcluster/ssh.py", line 115, in connect
raise exception.SSHConnectionError(host, port)
SSHConnectionError: failed to connect to host ec2-50-17-70-72.compute-1.amazonaws.com on port 22
error occurred in job (id=-1327015056): failed to authenticate to host ec2-174-129-46-13.compute-1.amazonaws.com as user root
Traceback (most recent call last):
File "/usr/local/lib/python2.7/dist-packages/starcluster/threadpool.py", line 31, in run
job.run()
File "/usr/local/lib/python2.7/dist-packages/starcluster/threadpool.py", line 58, in run
r = self.method(*self.args, **self.kwargs)
File "/usr/local/lib/python2.7/dist-packages/starcluster/clustersetup.py", line 351, in <lambda>
num_processors = sum(self.pool.map(lambda n: n.num_processors, nodes))
File "/usr/local/lib/python2.7/dist-packages/starcluster/node.py", line 169, in num_processors
'cat /proc/cpuinfo | grep processor | wc -l')[0])
File "/usr/local/lib/python2.7/dist-packages/starcluster/ssh.py", line 512, in execute
channel = self.transport.open_session()
File "/usr/local/lib/python2.7/dist-packages/starcluster/ssh.py", line 129, in transport
port=self._port, timeout=self._timeout)
File "/usr/local/lib/python2.7/dist-packages/starcluster/ssh.py", line 108, in connect
raise exception.SSHAuthException(username, host)
SSHAuthException: failed to authenticate to host ec2-174-129-46-13.compute-1.amazonaws.com as user root
error occurred in job (id=-1301836944): failed to authenticate to host ec2-23-20-152-32.compute-1.amazonaws.com as user root
Traceback (most recent call last):
File "/usr/local/lib/python2.7/dist-packages/starcluster/threadpool.py", line 31, in run
job.run()
File "/usr/local/lib/python2.7/dist-packages/starcluster/threadpool.py", line 58, in run
r = self.method(*self.args, **self.kwargs)
File "/usr/local/lib/python2.7/dist-packages/starcluster/clustersetup.py", line 351, in <lambda>
num_processors = sum(self.pool.map(lambda n: n.num_processors, nodes))
File "/usr/local/lib/python2.7/dist-packages/starcluster/node.py", line 169, in num_processors
'cat /proc/cpuinfo | grep processor | wc -l')[0])
File "/usr/local/lib/python2.7/dist-packages/starcluster/ssh.py", line 512, in execute
channel = self.transport.open_session()
File "/usr/local/lib/python2.7/dist-packages/starcluster/ssh.py", line 129, in transport
port=self._port, timeout=self._timeout)
File "/usr/local/lib/python2.7/dist-packages/starcluster/ssh.py", line 108, in connect
raise exception.SSHAuthException(username, host)
SSHAuthException: failed to authenticate to host ec2-23-20-152-32.compute-1.amazonaws.com as user root
error occurred in job (id=-1243088016): failed to authenticate to host ec2-184-73-14-134.compute-1.amazonaws.com as user root
Traceback (most recent call last):
File "/usr/local/lib/python2.7/dist-packages/starcluster/threadpool.py", line 31, in run
job.run()
File "/usr/local/lib/python2.7/dist-packages/starcluster/threadpool.py", line 58, in run
r = self.method(*self.args, **self.kwargs)
File "/usr/local/lib/python2.7/dist-packages/starcluster/clustersetup.py", line 351, in <lambda>
num_processors = sum(self.pool.map(lambda n: n.num_processors, nodes))
File "/usr/local/lib/python2.7/dist-packages/starcluster/node.py", line 169, in num_processors
'cat /proc/cpuinfo | grep processor | wc -l')[0])
File "/usr/local/lib/python2.7/dist-packages/starcluster/ssh.py", line 512, in execute
channel = self.transport.open_session()
File "/usr/local/lib/python2.7/dist-packages/starcluster/ssh.py", line 129, in transport
port=self._port, timeout=self._timeout)
File "/usr/local/lib/python2.7/dist-packages/starcluster/ssh.py", line 108, in connect
raise exception.SSHAuthException(username, host)
SSHAuthException: failed to authenticate to host ec2-184-73-14-134.compute-1.amazonaws.com as user root
error occurred in job (id=-1318622352): failed to authenticate to host ec2-23-22-1-205.compute-1.amazonaws.com as user root
Traceback (most recent call last):
File "/usr/local/lib/python2.7/dist-packages/starcluster/threadpool.py", line 31, in run
job.run()
File "/usr/local/lib/python2.7/dist-packages/starcluster/threadpool.py", line 58, in run
r = self.method(*self.args, **self.kwargs)
File "/usr/local/lib/python2.7/dist-packages/starcluster/clustersetup.py", line 351, in <lambda>
num_processors = sum(self.pool.map(lambda n: n.num_processors, nodes))
File "/usr/local/lib/python2.7/dist-packages/starcluster/node.py", line 169, in num_processors
'cat /proc/cpuinfo | grep processor | wc -l')[0])
File "/usr/local/lib/python2.7/dist-packages/starcluster/ssh.py", line 512, in execute
channel = self.transport.open_session()
File "/usr/local/lib/python2.7/dist-packages/starcluster/ssh.py", line 129, in transport
port=self._port, timeout=self._timeout)
File "/usr/local/lib/python2.7/dist-packages/starcluster/ssh.py", line 108, in connect
raise exception.SSHAuthException(username, host)
SSHAuthException: failed to authenticate to host ec2-23-22-1-205.compute-1.amazonaws.com as user root
error occurred in job (id=-1377371280): failed to authenticate to host ec2-107-21-141-210.compute-1.amazonaws.com as user root
Traceback (most recent call last):
File "/usr/local/lib/python2.7/dist-packages/starcluster/threadpool.py", line 31, in run
job.run()
File "/usr/local/lib/python2.7/dist-packages/starcluster/threadpool.py", line 58, in run
r = self.method(*self.args, **self.kwargs)
File "/usr/local/lib/python2.7/dist-packages/starcluster/clustersetup.py", line 351, in <lambda>
num_processors = sum(self.pool.map(lambda n: n.num_processors, nodes))
File "/usr/local/lib/python2.7/dist-packages/starcluster/node.py", line 169, in num_processors
'cat /proc/cpuinfo | grep processor | wc -l')[0])
File "/usr/local/lib/python2.7/dist-packages/starcluster/ssh.py", line 512, in execute
channel = self.transport.open_session()
File "/usr/local/lib/python2.7/dist-packages/starcluster/ssh.py", line 129, in transport
port=self._port, timeout=self._timeout)
File "/usr/local/lib/python2.7/dist-packages/starcluster/ssh.py", line 108, in connect
raise exception.SSHAuthException(username, host)
SSHAuthException: failed to authenticate to host ec2-107-21-141-210.compute-1.amazonaws.com as user root
error occurred in job (id=-1268266128): failed to authenticate to host ec2-174-129-158-44.compute-1.amazonaws.com as user root
Traceback (most recent call last):
File "/usr/local/lib/python2.7/dist-packages/starcluster/threadpool.py", line 31, in run
job.run()
File "/usr/local/lib/python2.7/dist-packages/starcluster/threadpool.py", line 58, in run
r = self.method(*self.args, **self.kwargs)
File "/usr/local/lib/python2.7/dist-packages/starcluster/clustersetup.py", line 351, in <lambda>
num_processors = sum(self.pool.map(lambda n: n.num_processors, nodes))
File "/usr/local/lib/python2.7/dist-packages/starcluster/node.py", line 169, in num_processors
'cat /proc/cpuinfo | grep processor | wc -l')[0])
File "/usr/local/lib/python2.7/dist-packages/starcluster/ssh.py", line 512, in execute
channel = self.transport.open_session()
File "/usr/local/lib/python2.7/dist-packages/starcluster/ssh.py", line 129, in transport
port=self._port, timeout=self._timeout)
File "/usr/local/lib/python2.7/dist-packages/starcluster/ssh.py", line 108, in connect
raise exception.SSHAuthException(username, host)
SSHAuthException: failed to authenticate to host ec2-174-129-158-44.compute-1.amazonaws.com as user root
error occurred in job (id=-1385763984): failed to authenticate to host ec2-23-20-76-33.compute-1.amazonaws.com as user root
Traceback (most recent call last):
File "/usr/local/lib/python2.7/dist-packages/starcluster/threadpool.py", line 31, in run
job.run()
File "/usr/local/lib/python2.7/dist-packages/starcluster/threadpool.py", line 58, in run
r = self.method(*self.args, **self.kwargs)
File "/usr/local/lib/python2.7/dist-packages/starcluster/clustersetup.py", line 351, in <lambda>
num_processors = sum(self.pool.map(lambda n: n.num_processors, nodes))
File "/usr/local/lib/python2.7/dist-packages/starcluster/node.py", line 169, in num_processors
'cat /proc/cpuinfo | grep processor | wc -l')[0])
File "/usr/local/lib/python2.7/dist-packages/starcluster/ssh.py", line 512, in execute
channel = self.transport.open_session()
File "/usr/local/lib/python2.7/dist-packages/starcluster/ssh.py", line 129, in transport
port=self._port, timeout=self._timeout)
File "/usr/local/lib/python2.7/dist-packages/starcluster/ssh.py", line 108, in connect
raise exception.SSHAuthException(username, host)
SSHAuthException: failed to authenticate to host ec2-23-20-76-33.compute-1.amazonaws.com as user root
error occurred in job (id=-1394156688): failed to authenticate to host ec2-174-129-128-158.compute-1.amazonaws.com as user root
Traceback (most recent call last):
File "/usr/local/lib/python2.7/dist-packages/starcluster/threadpool.py", line 31, in run
job.run()
File "/usr/local/lib/python2.7/dist-packages/starcluster/threadpool.py", line 58, in run
r = self.method(*self.args, **self.kwargs)
File "/usr/local/lib/python2.7/dist-packages/starcluster/clustersetup.py", line 351, in <lambda>
num_processors = sum(self.pool.map(lambda n: n.num_processors, nodes))
File "/usr/local/lib/python2.7/dist-packages/starcluster/node.py", line 169, in num_processors
'cat /proc/cpuinfo | grep processor | wc -l')[0])
File "/usr/local/lib/python2.7/dist-packages/starcluster/ssh.py", line 512, in execute
channel = self.transport.open_session()
File "/usr/local/lib/python2.7/dist-packages/starcluster/ssh.py", line 129, in transport
port=self._port, timeout=self._timeout)
File "/usr/local/lib/python2.7/dist-packages/starcluster/ssh.py", line 108, in connect
raise exception.SSHAuthException(username, host)
SSHAuthException: failed to authenticate to host ec2-174-129-128-158.compute-1.amazonaws.com as user root
error occurred in job (id=-1310229648): failed to authenticate to host ec2-23-20-118-235.compute-1.amazonaws.com as user root
Traceback (most recent call last):
File "/usr/local/lib/python2.7/dist-packages/starcluster/threadpool.py", line 31, in run
job.run()
File "/usr/local/lib/python2.7/dist-packages/starcluster/threadpool.py", line 58, in run
r = self.method(*self.args, **self.kwargs)
File "/usr/local/lib/python2.7/dist-packages/starcluster/clustersetup.py", line 351, in <lambda>
num_processors = sum(self.pool.map(lambda n: n.num_processors, nodes))
File "/usr/local/lib/python2.7/dist-packages/starcluster/node.py", line 169, in num_processors
'cat /proc/cpuinfo | grep processor | wc -l')[0])
File "/usr/local/lib/python2.7/dist-packages/starcluster/ssh.py", line 512, in execute
channel = self.transport.open_session()
File "/usr/local/lib/python2.7/dist-packages/starcluster/ssh.py", line 129, in transport
port=self._port, timeout=self._timeout)
File "/usr/local/lib/python2.7/dist-packages/starcluster/ssh.py", line 108, in connect
raise exception.SSHAuthException(username, host)
SSHAuthException: failed to authenticate to host ec2-23-20-118-235.compute-1.amazonaws.com as user root
error occurred in job (id=-1285051536): failed to authenticate to host ec2-107-22-132-97.compute-1.amazonaws.com as user root
Traceback (most recent call last):
File "/usr/local/lib/python2.7/dist-packages/starcluster/threadpool.py", line 31, in run
job.run()
File "/usr/local/lib/python2.7/dist-packages/starcluster/threadpool.py", line 58, in run
r = self.method(*self.args, **self.kwargs)
File "/usr/local/lib/python2.7/dist-packages/starcluster/clustersetup.py", line 351, in <lambda>
num_processors = sum(self.pool.map(lambda n: n.num_processors, nodes))
File "/usr/local/lib/python2.7/dist-packages/starcluster/node.py", line 169, in num_processors
'cat /proc/cpuinfo | grep processor | wc -l')[0])
File "/usr/local/lib/python2.7/dist-packages/starcluster/ssh.py", line 512, in execute
channel = self.transport.open_session()
File "/usr/local/lib/python2.7/dist-packages/starcluster/ssh.py", line 129, in transport
port=self._port, timeout=self._timeout)
File "/usr/local/lib/python2.7/dist-packages/starcluster/ssh.py", line 108, in connect
raise exception.SSHAuthException(username, host)
SSHAuthException: failed to authenticate to host ec2-107-22-132-97.compute-1.amazonaws.com as user root
error occurred in job (id=-1335407760): failed to authenticate to host ec2-174-129-82-72.compute-1.amazonaws.com as user root
Traceback (most recent call last):
File "/usr/local/lib/python2.7/dist-packages/starcluster/threadpool.py", line 31, in run
job.run()
File "/usr/local/lib/python2.7/dist-packages/starcluster/threadpool.py", line 58, in run
r = self.method(*self.args, **self.kwargs)
File "/usr/local/lib/python2.7/dist-packages/starcluster/clustersetup.py", line 351, in <lambda>
num_processors = sum(self.pool.map(lambda n: n.num_processors, nodes))
File "/usr/local/lib/python2.7/dist-packages/starcluster/node.py", line 169, in num_processors
'cat /proc/cpuinfo | grep processor | wc -l')[0])
File "/usr/local/lib/python2.7/dist-packages/starcluster/ssh.py", line 512, in execute
channel = self.transport.open_session()
File "/usr/local/lib/python2.7/dist-packages/starcluster/ssh.py", line 129, in transport
port=self._port, timeout=self._timeout)
File "/usr/local/lib/python2.7/dist-packages/starcluster/ssh.py", line 108, in connect
raise exception.SSHAuthException(username, host)
SSHAuthException: failed to authenticate to host ec2-174-129-82-72.compute-1.amazonaws.com as user root
---------- SYSTEM INFO ----------
StarCluster: 0.93.1
Python: 2.7.2+ (default, Oct 4 2011, 20:03:08) [GCC 4.6.1]
Platform: Linux-3.0.0-14-virtual-i686-with-Ubuntu-11.10-oneiric
boto: 2.0
paramiko: 1.7.7.1 (George)
Crypto: 2.5
jinja2: 2.5.5
decorator: 3.3.1
Received on Thu May 10 2012 - 15:15:10 EDT
This archive was generated by
hypermail 2.3.0.