PID: 18373 config.py:515 - DEBUG - Loading config PID: 18373 config.py:108 - DEBUG - Loading file: /home/jpk37/.starcluster/config PID: 18373 cli.py:87 - ERROR - config file /home/jpk37/.starcluster/config does not exist PID: 18373 exception.py:211 - INFO - Config template written to /home/jpk37/.starcluster/config. Please customize this file. PID: 18551 config.py:515 - DEBUG - Loading config PID: 18551 config.py:108 - DEBUG - Loading file: /home/jpk37/.starcluster/config PID: 18551 cli.py:91 - ERROR - volume 'vol-b2ff16d8' not defined in config PID: 18563 config.py:515 - DEBUG - Loading config PID: 18563 config.py:108 - DEBUG - Loading file: /home/jpk37/.starcluster/config PID: 18563 config.py:515 - DEBUG - Loading config PID: 18563 config.py:108 - DEBUG - Loading file: /home/jpk37/.starcluster/config PID: 18563 awsutils.py:54 - DEBUG - creating self._conn w/ connection_authenticator kwargs = {'path': '/', 'region': None, 'port': None, 'is_secure': True} PID: 18563 start.py:167 - INFO - Using default cluster template: smallcluster PID: 18563 cluster.py:1333 - INFO - Validating cluster template settings... PID: 18563 start.py:180 - ERROR - settings for cluster template "smallcluster" are not valid: PID: 18563 cli.py:170 - ERROR - Account does not contain a key with keyname = blastkey.pem. PID: 18568 config.py:515 - DEBUG - Loading config PID: 18568 config.py:108 - DEBUG - Loading file: /home/jpk37/.starcluster/config PID: 18568 config.py:515 - DEBUG - Loading config PID: 18568 config.py:108 - DEBUG - Loading file: /home/jpk37/.starcluster/config PID: 18568 awsutils.py:54 - DEBUG - creating self._conn w/ connection_authenticator kwargs = {'path': '/', 'region': None, 'port': None, 'is_secure': True} PID: 18568 start.py:167 - INFO - Using default cluster template: smallcluster PID: 18568 cluster.py:1333 - INFO - Validating cluster template settings... PID: 18568 cluster.py:872 - DEBUG - Launch map: node001 (ami: ami-8cf913e5, type: t1.micro)... PID: 18568 cluster.py:1355 - INFO - Cluster template settings are valid PID: 18568 cluster.py:1228 - INFO - Starting cluster... PID: 18568 cluster.py:891 - INFO - Launching a 2-node cluster... PID: 18568 cluster.py:908 - INFO - Launching a 2-node cluster... PID: 18568 cluster.py:872 - DEBUG - Launch map: node001 (ami: ami-8cf913e5, type: t1.micro)... PID: 18568 cluster.py:919 - DEBUG - Launching master (ami: ami-8cf913e5, type: t1.micro) PID: 18568 cluster.py:919 - DEBUG - Launching node001 (ami: ami-8cf913e5, type: t1.micro) PID: 18568 awsutils.py:175 - INFO - Creating security group @sc-mycluster... PID: 18568 cli.py:157 - ERROR - UnsupportedOperation: AMI 'ami-8cf913e5' with an instance-store root device is not supported for the instance type 't1.micro'. PID: 18573 config.py:515 - DEBUG - Loading config PID: 18573 config.py:108 - DEBUG - Loading file: /home/jpk37/.starcluster/config PID: 18573 config.py:515 - DEBUG - Loading config PID: 18573 config.py:108 - DEBUG - Loading file: /home/jpk37/.starcluster/config PID: 18573 awsutils.py:54 - DEBUG - creating self._conn w/ connection_authenticator kwargs = {'path': '/', 'region': None, 'port': None, 'is_secure': True} PID: 18573 cluster.py:670 - DEBUG - existing nodes: {} PID: 18573 cluster.py:686 - DEBUG - returning self._nodes = [] PID: 18573 cluster.py:670 - DEBUG - existing nodes: {} PID: 18573 cluster.py:686 - DEBUG - returning self._nodes = [] PID: 18573 cli.py:170 - ERROR - Cluster with tag name mycluster already exists. If you want to reconfigure the existing instances use the 'restart' command: $ starcluster restart mycluster This will reboot all of the instances and configure the cluster starting from scratch. Otherwise either choose a different tag name, or terminate the existing cluster using: $ starcluster terminate mycluster PID: 18575 config.py:515 - DEBUG - Loading config PID: 18575 config.py:108 - DEBUG - Loading file: /home/jpk37/.starcluster/config PID: 18575 awsutils.py:54 - DEBUG - creating self._conn w/ connection_authenticator kwargs = {'path': '/', 'region': None, 'port': None, 'is_secure': True} PID: 18575 cluster.py:670 - DEBUG - existing nodes: {} PID: 18575 cluster.py:686 - DEBUG - returning self._nodes = [] PID: 18575 cli.py:170 - ERROR - No running nodes found PID: 18576 config.py:515 - DEBUG - Loading config PID: 18576 config.py:108 - DEBUG - Loading file: /home/jpk37/.starcluster/config PID: 18576 awsutils.py:54 - DEBUG - creating self._conn w/ connection_authenticator kwargs = {'path': '/', 'region': None, 'port': None, 'is_secure': True} PID: 18576 cluster.py:670 - DEBUG - existing nodes: {} PID: 18576 cluster.py:686 - DEBUG - returning self._nodes = [] PID: 18576 cluster.py:670 - DEBUG - existing nodes: {} PID: 18576 cluster.py:686 - DEBUG - returning self._nodes = [] PID: 18576 cluster.py:670 - DEBUG - existing nodes: {} PID: 18576 cluster.py:686 - DEBUG - returning self._nodes = [] PID: 18576 cluster.py:1196 - INFO - Removing @sc-mycluster security group PID: 18586 config.py:515 - DEBUG - Loading config PID: 18586 config.py:108 - DEBUG - Loading file: /home/jpk37/.starcluster/config PID: 18586 config.py:515 - DEBUG - Loading config PID: 18586 config.py:108 - DEBUG - Loading file: /home/jpk37/.starcluster/config PID: 18586 awsutils.py:54 - DEBUG - creating self._conn w/ connection_authenticator kwargs = {'path': '/', 'region': None, 'port': None, 'is_secure': True} PID: 18586 start.py:167 - INFO - Using default cluster template: smallcluster PID: 18586 cluster.py:1333 - INFO - Validating cluster template settings... PID: 18586 cluster.py:872 - DEBUG - Launch map: node001 (ami: ami-8cf913e5, type: m1.small)... PID: 18586 cluster.py:1355 - INFO - Cluster template settings are valid PID: 18586 cluster.py:1228 - INFO - Starting cluster... PID: 18586 cluster.py:891 - INFO - Launching a 2-node cluster... PID: 18586 cluster.py:908 - INFO - Launching a 2-node cluster... PID: 18586 cluster.py:872 - DEBUG - Launch map: node001 (ami: ami-8cf913e5, type: m1.small)... PID: 18586 cluster.py:919 - DEBUG - Launching master (ami: ami-8cf913e5, type: m1.small) PID: 18586 cluster.py:919 - DEBUG - Launching node001 (ami: ami-8cf913e5, type: m1.small) PID: 18586 awsutils.py:175 - INFO - Creating security group @sc-mycluster... PID: 18586 cluster.py:1048 - INFO - Waiting for cluster to come up... (updating every 30s) PID: 18586 cluster.py:670 - DEBUG - existing nodes: {} PID: 18586 cluster.py:678 - DEBUG - adding node i-97f26bf6 to self._nodes list PID: 18586 cluster.py:678 - DEBUG - adding node i-9bf26bfa to self._nodes list PID: 18586 cluster.py:686 - DEBUG - returning self._nodes = [, ] PID: 18586 cluster.py:1070 - INFO - Waiting for all nodes to be in a 'running' state... PID: 18586 cluster.py:670 - DEBUG - existing nodes: {u'i-97f26bf6': , u'i-9bf26bfa': } PID: 18586 cluster.py:673 - DEBUG - updating existing node i-97f26bf6 in self._nodes PID: 18586 cluster.py:673 - DEBUG - updating existing node i-9bf26bfa in self._nodes PID: 18586 cluster.py:686 - DEBUG - returning self._nodes = [, ] PID: 18586 cluster.py:670 - DEBUG - existing nodes: {u'i-97f26bf6': , u'i-9bf26bfa': } PID: 18586 cluster.py:673 - DEBUG - updating existing node i-97f26bf6 in self._nodes PID: 18586 cluster.py:673 - DEBUG - updating existing node i-9bf26bfa in self._nodes PID: 18586 cluster.py:686 - DEBUG - returning self._nodes = [, ] PID: 18586 cluster.py:670 - DEBUG - existing nodes: {u'i-97f26bf6': , u'i-9bf26bfa': } PID: 18586 cluster.py:673 - DEBUG - updating existing node i-97f26bf6 in self._nodes PID: 18586 cluster.py:673 - DEBUG - updating existing node i-9bf26bfa in self._nodes PID: 18586 cluster.py:686 - DEBUG - returning self._nodes = [, ] PID: 18586 cluster.py:670 - DEBUG - existing nodes: {u'i-97f26bf6': , u'i-9bf26bfa': } PID: 18586 cluster.py:673 - DEBUG - updating existing node i-97f26bf6 in self._nodes PID: 18586 cluster.py:673 - DEBUG - updating existing node i-9bf26bfa in self._nodes PID: 18586 cluster.py:686 - DEBUG - returning self._nodes = [, ] PID: 18586 cluster.py:670 - DEBUG - existing nodes: {u'i-97f26bf6': , u'i-9bf26bfa': } PID: 18586 cluster.py:673 - DEBUG - updating existing node i-97f26bf6 in self._nodes PID: 18586 cluster.py:673 - DEBUG - updating existing node i-9bf26bfa in self._nodes PID: 18586 cluster.py:686 - DEBUG - returning self._nodes = [, ] PID: 18586 cluster.py:670 - DEBUG - existing nodes: {u'i-97f26bf6': , u'i-9bf26bfa': } PID: 18586 cluster.py:673 - DEBUG - updating existing node i-97f26bf6 in self._nodes PID: 18586 cluster.py:673 - DEBUG - updating existing node i-9bf26bfa in self._nodes PID: 18586 cluster.py:686 - DEBUG - returning self._nodes = [, ] PID: 18586 cluster.py:670 - DEBUG - existing nodes: {u'i-97f26bf6': , u'i-9bf26bfa': } PID: 18586 cluster.py:673 - DEBUG - updating existing node i-97f26bf6 in self._nodes PID: 18586 cluster.py:673 - DEBUG - updating existing node i-9bf26bfa in self._nodes PID: 18586 cluster.py:686 - DEBUG - returning self._nodes = [, ] PID: 18586 cluster.py:670 - DEBUG - existing nodes: {u'i-97f26bf6': , u'i-9bf26bfa': } PID: 18586 cluster.py:673 - DEBUG - updating existing node i-97f26bf6 in self._nodes PID: 18586 cluster.py:673 - DEBUG - updating existing node i-9bf26bfa in self._nodes PID: 18586 cluster.py:686 - DEBUG - returning self._nodes = [, ] PID: 18586 cluster.py:1081 - INFO - Waiting for SSH to come up on all nodes... PID: 18586 ssh.py:61 - DEBUG - loading private key /home/jpk37/Downloads/blastkey.pem PID: 18586 ssh.py:68 - DEBUG - specified key does not end in either rsa or dsa, trying both PID: 18586 ssh.py:145 - DEBUG - Using private key /home/jpk37/Downloads/blastkey.pem (rsa) PID: 18586 ssh.py:82 - DEBUG - connecting to host ec2-107-20-63-150.compute-1.amazonaws.com on port 22 as user root PID: 18586 ssh.py:61 - DEBUG - loading private key /home/jpk37/Downloads/blastkey.pem PID: 18586 ssh.py:68 - DEBUG - specified key does not end in either rsa or dsa, trying both PID: 18586 ssh.py:145 - DEBUG - Using private key /home/jpk37/Downloads/blastkey.pem (rsa) PID: 18586 ssh.py:82 - DEBUG - connecting to host ec2-50-17-119-104.compute-1.amazonaws.com on port 22 as user root PID: 18586 cluster.py:1252 - INFO - The master node is ec2-107-20-63-150.compute-1.amazonaws.com PID: 18586 cluster.py:1253 - INFO - Setting up the cluster... PID: 18586 cluster.py:1127 - INFO - Attaching volume vol-b2ff16d8 to master node on /dev/sdz ... PID: 18586 cluster.py:1129 - DEBUG - resp = attaching PID: 18586 cluster.py:670 - DEBUG - existing nodes: {u'i-97f26bf6': , u'i-9bf26bfa': } PID: 18586 cluster.py:673 - DEBUG - updating existing node i-97f26bf6 in self._nodes PID: 18586 cluster.py:673 - DEBUG - updating existing node i-9bf26bfa in self._nodes PID: 18586 cluster.py:686 - DEBUG - returning self._nodes = [, ] PID: 18586 clustersetup.py:96 - INFO - Configuring hostnames... PID: 18586 threadpool.py:108 - DEBUG - unfinished_tasks = 2 PID: 18586 ssh.py:164 - DEBUG - creating sftp connection PID: 18586 ssh.py:164 - DEBUG - creating sftp connection PID: 18586 clustersetup.py:262 - INFO - Mounting EBS volume vol-b2ff16d8 on /db... PID: 18586 ssh.py:405 - DEBUG - /dev/sda1 on / type ext3 (rw) PID: 18586 ssh.py:405 - DEBUG - proc on /proc type proc (rw,noexec,nosuid,nodev) PID: 18586 ssh.py:405 - DEBUG - none on /sys type sysfs (rw,noexec,nosuid,nodev) PID: 18586 ssh.py:405 - DEBUG - none on /sys/kernel/debug type debugfs (rw) PID: 18586 ssh.py:405 - DEBUG - none on /sys/kernel/security type securityfs (rw) PID: 18586 ssh.py:405 - DEBUG - devtmpfs on /dev type devtmpfs (rw,mode=0755) PID: 18586 ssh.py:405 - DEBUG - none on /dev/pts type devpts (rw,noexec,nosuid,gid=5,mode=0620) PID: 18586 ssh.py:405 - DEBUG - none on /dev/shm type tmpfs (rw,nosuid,nodev) PID: 18586 ssh.py:405 - DEBUG - none on /var/run type tmpfs (rw,nosuid,mode=0755) PID: 18586 ssh.py:405 - DEBUG - none on /var/lock type tmpfs (rw,noexec,nosuid,nodev) PID: 18586 ssh.py:405 - DEBUG - none on /lib/init/rw type tmpfs (rw,nosuid,mode=0755) PID: 18586 ssh.py:405 - DEBUG - /dev/sda2 on /mnt type ext3 (rw) PID: 18586 ssh.py:405 - DEBUG - nfsd on /proc/fs/nfsd type nfsd (rw) PID: 18586 ssh.py:254 - DEBUG - new /etc/fstab after removing regex ( /db ) matches: # /etc/fstab: static file system information. # proc /proc proc nodev,noexec,nosuid 0 0 /dev/sda1 / ext3 defaults 0 0 /dev/sda2 /mnt auto defaults,comment=cloudconfig 0 0 /dev/sda3 none swap sw,comment=cloudconfig 0 0 PID: 18586 clustersetup.py:144 - INFO - Creating cluster user: sgeadmin (uid: 1001, gid: 1001) PID: 18586 threadpool.py:108 - DEBUG - unfinished_tasks = 2 PID: 18586 clustersetup.py:161 - DEBUG - user sgeadmin does not exist, creating... PID: 18586 clustersetup.py:161 - DEBUG - user sgeadmin does not exist, creating... PID: 18586 clustersetup.py:186 - INFO - Configuring scratch space for user: sgeadmin PID: 18586 threadpool.py:108 - DEBUG - unfinished_tasks = 2 PID: 18586 clustersetup.py:195 - INFO - Configuring /etc/hosts on each node PID: 18586 threadpool.py:108 - DEBUG - unfinished_tasks = 2 PID: 18586 ssh.py:254 - DEBUG - new /etc/hosts after removing regex (master|node001) matches: 127.0.0.1 localhost # The following lines are desirable for IPv6 capable hosts ::1 ip6-localhost ip6-loopback fe00::0 ip6-localnet ff00::0 ip6-mcastprefix ff02::1 ip6-allnodes ff02::2 ip6-allrouters ff02::3 ip6-allhosts PID: 18586 ssh.py:254 - DEBUG - new /etc/hosts after removing regex (master|node001) matches: 127.0.0.1 localhost # The following lines are desirable for IPv6 capable hosts ::1 ip6-localhost ip6-loopback fe00::0 ip6-localnet ff00::0 ip6-mcastprefix ff02::1 ip6-allnodes ff02::2 ip6-allrouters ff02::3 ip6-allhosts PID: 18586 clustersetup.py:314 - INFO - Configuring NFS... PID: 18586 ssh.py:405 - DEBUG - Rather than invoking init scripts through /etc/init.d, use the service(8) PID: 18586 ssh.py:405 - DEBUG - utility, e.g. service portmap start PID: 18586 ssh.py:405 - DEBUG - PID: 18586 ssh.py:405 - DEBUG - Since the script you are attempting to invoke has been converted to an PID: 18586 ssh.py:405 - DEBUG - Upstart job, you may also use the start(8) utility, e.g. start portmap PID: 18586 ssh.py:405 - DEBUG - * Exporting directories for NFS kernel daemon... PID: 18586 ssh.py:405 - DEBUG - ...done. PID: 18586 ssh.py:405 - DEBUG - * Starting NFS kernel daemon PID: 18586 ssh.py:405 - DEBUG - ...done. PID: 18586 ssh.py:405 - DEBUG - /dev/sda1 on / type ext3 (rw) PID: 18586 ssh.py:405 - DEBUG - proc on /proc type proc (rw,noexec,nosuid,nodev) PID: 18586 ssh.py:405 - DEBUG - none on /sys type sysfs (rw,noexec,nosuid,nodev) PID: 18586 ssh.py:405 - DEBUG - none on /sys/kernel/debug type debugfs (rw) PID: 18586 ssh.py:405 - DEBUG - none on /sys/kernel/security type securityfs (rw) PID: 18586 ssh.py:405 - DEBUG - devtmpfs on /dev type devtmpfs (rw,mode=0755) PID: 18586 ssh.py:405 - DEBUG - none on /dev/pts type devpts (rw,noexec,nosuid,gid=5,mode=0620) PID: 18586 ssh.py:405 - DEBUG - none on /dev/shm type tmpfs (rw,nosuid,nodev) PID: 18586 ssh.py:405 - DEBUG - none on /var/run type tmpfs (rw,nosuid,mode=0755) PID: 18586 ssh.py:405 - DEBUG - none on /var/lock type tmpfs (rw,noexec,nosuid,nodev) PID: 18586 ssh.py:405 - DEBUG - none on /lib/init/rw type tmpfs (rw,nosuid,mode=0755) PID: 18586 ssh.py:405 - DEBUG - /dev/sda2 on /mnt type ext3 (rw) PID: 18586 ssh.py:405 - DEBUG - nfsd on /proc/fs/nfsd type nfsd (rw) PID: 18586 clustersetup.py:304 - INFO - Mounting shares for node node001 PID: 18586 threadpool.py:108 - DEBUG - unfinished_tasks = 1 PID: 18586 ssh.py:405 - DEBUG - Rather than invoking init scripts through /etc/init.d, use the service(8) PID: 18586 ssh.py:405 - DEBUG - utility, e.g. service portmap start PID: 18586 ssh.py:405 - DEBUG - PID: 18586 ssh.py:405 - DEBUG - Since the script you are attempting to invoke has been converted to an PID: 18586 ssh.py:405 - DEBUG - Upstart job, you may also use the start(8) utility, e.g. start portmap PID: 18586 ssh.py:402 - DEBUG - command mount -t devpts none /dev/pts failed with status 32 PID: 18586 ssh.py:405 - DEBUG - mount: none already mounted or /dev/pts busy PID: 18586 ssh.py:405 - DEBUG - mount: according to mtab, none is already mounted on /dev/pts PID: 18586 ssh.py:254 - DEBUG - new /etc/fstab after removing regex ( /home | /opt/sge6 | /db ) matches: # /etc/fstab: static file system information. # proc /proc proc nodev,noexec,nosuid 0 0 /dev/sda1 / ext3 defaults 0 0 /dev/sda2 /mnt auto defaults,comment=cloudconfig 0 0 /dev/sda3 none swap sw,comment=cloudconfig 0 0 PID: 18586 threadpool.py:108 - DEBUG - unfinished_tasks = 1 PID: 18586 utils.py:61 - INFO - _setup_nfs took 0.234 mins PID: 18586 clustersetup.py:207 - INFO - Configuring passwordless ssh for root PID: 18586 node.py:396 - DEBUG - adding auth_key_contents PID: 18586 clustersetup.py:215 - INFO - Configuring passwordless ssh for sgeadmin PID: 18586 node.py:396 - DEBUG - adding auth_key_contents PID: 18586 node.py:404 - DEBUG - adding conn_pubkey_contents PID: 18586 ssh.py:402 - DEBUG - command pkill -9 sge failed with status 1 PID: 18586 ssh.py:402 - DEBUG - command rm /etc/init.d/sge* failed with status 1 PID: 18586 ssh.py:405 - DEBUG - rm: cannot remove `/etc/init.d/sge*': No such file or directory PID: 18586 ssh.py:405 - DEBUG - lx24-x86 PID: 18586 ssh.py:405 - DEBUG - [: 359: 11: unexpected operator PID: 18586 ssh.py:405 - DEBUG - [: 359: 11: unexpected operator PID: 18586 ssh.py:402 - DEBUG - command pkill -9 sge failed with status 1 PID: 18586 ssh.py:402 - DEBUG - command rm /etc/init.d/sge* failed with status 1 PID: 18586 ssh.py:405 - DEBUG - rm: cannot remove `/etc/init.d/sge*': No such file or directory PID: 18586 ssh.py:405 - DEBUG - lx24-x86 PID: 18586 ssh.py:405 - DEBUG - [: 359: 11: unexpected operator PID: 18586 ssh.py:405 - DEBUG - [: 359: 11: unexpected operator PID: 18586 clustersetup.py:360 - INFO - Installing Sun Grid Engine... PID: 18586 ssh.py:405 - DEBUG - Reading configuration from file ./ec2_sge.conf PID: 18586 ssh.py:405 - DEBUG - [H[2JInstall log can be found in: /opt/sge6/default/common/install_logs/qmaster_install_master_2011-08-04_15:40:36.log PID: 18586 ssh.py:405 - DEBUG - Install log can be found in: /opt/sge6/default/common/install_logs/execd_install_master_2011-08-04_15:40:44.log PID: 18586 ssh.py:405 - DEBUG - [: 359: 11: unexpected operator PID: 18586 ssh.py:405 - DEBUG - [: 359: 11: unexpected operator PID: 18586 ssh.py:405 - DEBUG - [: 359: 11: unexpected operator PID: 18586 ssh.py:405 - DEBUG - [: 359: 11: unexpected operator PID: 18586 ssh.py:405 - DEBUG - root@master modified "all.q" in cluster queue list PID: 18586 ssh.py:405 - DEBUG - adminhost "node001" already exists PID: 18586 ssh.py:405 - DEBUG - submithost "node001" already exists PID: 18586 threadpool.py:108 - DEBUG - unfinished_tasks = 1 PID: 18586 ssh.py:405 - DEBUG - lx24-x86 PID: 18586 ssh.py:405 - DEBUG - [: 359: 11: unexpected operator PID: 18586 ssh.py:405 - DEBUG - [: 359: 11: unexpected operator PID: 18586 ssh.py:405 - DEBUG - adminhost "node001" already exists PID: 18586 ssh.py:405 - DEBUG - submithost "node001" already exists PID: 18586 threadpool.py:108 - DEBUG - unfinished_tasks = 1 PID: 18586 threadpool.py:108 - DEBUG - unfinished_tasks = 1 PID: 18586 threadpool.py:108 - DEBUG - unfinished_tasks = 1 PID: 18586 threadpool.py:108 - DEBUG - unfinished_tasks = 1 PID: 18586 threadpool.py:108 - DEBUG - unfinished_tasks = 1 PID: 18586 threadpool.py:108 - DEBUG - unfinished_tasks = 1 PID: 18586 ssh.py:405 - DEBUG - Reading configuration from file ./ec2_sge.conf PID: 18586 ssh.py:405 - DEBUG - Install log can be found in: /opt/sge6/default/common/install_logs/execd_install_node001_2011-08-04_15:40:50.log PID: 18586 ssh.py:405 - DEBUG - [: 359: 11: unexpected operator PID: 18586 ssh.py:405 - DEBUG - [: 359: 11: unexpected operator PID: 18586 ssh.py:405 - DEBUG - [: 359: 11: unexpected operator PID: 18586 ssh.py:405 - DEBUG - [: 359: 11: unexpected operator PID: 18586 ssh.py:405 - DEBUG - 1 PID: 18586 ssh.py:405 - DEBUG - 1 PID: 18586 ssh.py:405 - DEBUG - root@master added "orte" to parallel environment list PID: 18586 ssh.py:405 - DEBUG - Unable to find "orte" in "pe_list" of "all.q" - Adding new element. PID: 18586 ssh.py:405 - DEBUG - PID: 18586 ssh.py:405 - DEBUG - root@master modified "all.q" in cluster queue list PID: 18586 threadpool.py:96 - INFO - Shutting down threads... PID: 18586 threadpool.py:108 - DEBUG - unfinished_tasks = 3 PID: 18586 cluster.py:1243 - INFO - The cluster has been started and configured. Login to the master node as root by running: $ starcluster sshmaster mycluster When you are finished using the cluster, run: $ starcluster stop mycluster to shutdown the cluster and stop paying for service. If this cluster uses EBS instances then the 'stop' command above will put all nodes into a 'stopped' state. The cluster may then be restarted at a later time, without losing data, by passing the -x option to the 'start' command. To completely terminate an EBS cluster: $ starcluster terminate mycluster NOTE: Terminating an EBS cluster will destroy all volumes backing the nodes. PID: 18586 utils.py:72 - INFO - Starting cluster took 5.292 mins PID: 18586 ssh.py:536 - DEBUG - __del__ called PID: 18586 ssh.py:536 - DEBUG - __del__ called PID: 18681 config.py:515 - DEBUG - Loading config PID: 18681 config.py:108 - DEBUG - Loading file: /home/jpk37/.starcluster/config PID: 18681 awsutils.py:54 - DEBUG - creating self._conn w/ connection_authenticator kwargs = {'path': '/', 'region': None, 'port': None, 'is_secure': True} PID: 18681 cluster.py:670 - DEBUG - existing nodes: {} PID: 18681 cluster.py:678 - DEBUG - adding node i-97f26bf6 to self._nodes list PID: 18681 cluster.py:678 - DEBUG - adding node i-9bf26bfa to self._nodes list PID: 18681 cluster.py:686 - DEBUG - returning self._nodes = [, ] PID: 18681 node.py:822 - DEBUG - using system's ssh client PID: 18681 node.py:825 - DEBUG - ssh_cmd: ssh -i /home/jpk37/Downloads/blastkey.pem root@ec2-107-20-63-150.compute-1.amazonaws.com PID: 18704 config.py:515 - DEBUG - Loading config PID: 18704 config.py:108 - DEBUG - Loading file: /home/jpk37/.starcluster/config PID: 18704 awsutils.py:54 - DEBUG - creating self._conn w/ connection_authenticator kwargs = {'path': '/', 'region': None, 'port': None, 'is_secure': True} PID: 18704 cluster.py:670 - DEBUG - existing nodes: {} PID: 18704 cluster.py:678 - DEBUG - adding node i-97f26bf6 to self._nodes list PID: 18704 cluster.py:678 - DEBUG - adding node i-9bf26bfa to self._nodes list PID: 18704 cluster.py:686 - DEBUG - returning self._nodes = [, ] PID: 18704 stop.py:51 - INFO - Aborting... PID: 18705 config.py:515 - DEBUG - Loading config PID: 18705 config.py:108 - DEBUG - Loading file: /home/jpk37/.starcluster/config PID: 18705 awsutils.py:54 - DEBUG - creating self._conn w/ connection_authenticator kwargs = {'path': '/', 'region': None, 'port': None, 'is_secure': True} PID: 18705 cluster.py:670 - DEBUG - existing nodes: {} PID: 18705 cluster.py:678 - DEBUG - adding node i-97f26bf6 to self._nodes list PID: 18705 cluster.py:678 - DEBUG - adding node i-9bf26bfa to self._nodes list PID: 18705 cluster.py:686 - DEBUG - returning self._nodes = [, ] PID: 18705 cluster.py:670 - DEBUG - existing nodes: {u'i-97f26bf6': , u'i-9bf26bfa': } PID: 18705 cluster.py:673 - DEBUG - updating existing node i-97f26bf6 in self._nodes PID: 18705 cluster.py:673 - DEBUG - updating existing node i-9bf26bfa in self._nodes PID: 18705 cluster.py:686 - DEBUG - returning self._nodes = [, ] PID: 18705 node.py:659 - INFO - Detaching volume vol-b2ff16d8 from master PID: 18705 cluster.py:670 - DEBUG - existing nodes: {u'i-97f26bf6': , u'i-9bf26bfa': } PID: 18705 cluster.py:673 - DEBUG - updating existing node i-97f26bf6 in self._nodes PID: 18705 cluster.py:673 - DEBUG - updating existing node i-9bf26bfa in self._nodes PID: 18705 cluster.py:686 - DEBUG - returning self._nodes = [, ] PID: 18705 node.py:745 - INFO - Terminating node: master (i-97f26bf6) PID: 18705 node.py:745 - INFO - Terminating node: node001 (i-9bf26bfa) PID: 18705 cluster.py:670 - DEBUG - existing nodes: {} PID: 18705 cluster.py:686 - DEBUG - returning self._nodes = [] PID: 18705 cluster.py:1178 - INFO - Removing @sc-mycluster security group PID: 18705 cli.py:157 - ERROR - InvalidGroup.InUse: There are active instances using security group '@sc-mycluster' PID: 18707 config.py:515 - DEBUG - Loading config PID: 18707 config.py:108 - DEBUG - Loading file: /home/jpk37/.starcluster/config PID: 18707 awsutils.py:54 - DEBUG - creating self._conn w/ connection_authenticator kwargs = {'path': '/', 'region': None, 'port': None, 'is_secure': True} PID: 18707 cluster.py:670 - DEBUG - existing nodes: {} PID: 18707 cluster.py:686 - DEBUG - returning self._nodes = [] PID: 18707 cluster.py:670 - DEBUG - existing nodes: {} PID: 18707 cluster.py:686 - DEBUG - returning self._nodes = [] PID: 18707 cluster.py:670 - DEBUG - existing nodes: {} PID: 18707 cluster.py:686 - DEBUG - returning self._nodes = [] PID: 18707 cluster.py:670 - DEBUG - existing nodes: {} PID: 18707 cluster.py:686 - DEBUG - returning self._nodes = [] PID: 18707 cluster.py:1178 - INFO - Removing @sc-mycluster security group PID: 18707 cli.py:157 - ERROR - InvalidGroup.InUse: There are active instances using security group '@sc-mycluster' PID: 18709 config.py:515 - DEBUG - Loading config PID: 18709 config.py:108 - DEBUG - Loading file: /home/jpk37/.starcluster/config PID: 18709 awsutils.py:54 - DEBUG - creating self._conn w/ connection_authenticator kwargs = {'path': '/', 'region': None, 'port': None, 'is_secure': True} PID: 18709 cluster.py:670 - DEBUG - existing nodes: {} PID: 18709 cluster.py:686 - DEBUG - returning self._nodes = [] PID: 18709 cluster.py:670 - DEBUG - existing nodes: {} PID: 18709 cluster.py:686 - DEBUG - returning self._nodes = [] PID: 18709 cluster.py:670 - DEBUG - existing nodes: {} PID: 18709 cluster.py:686 - DEBUG - returning self._nodes = [] PID: 18709 cluster.py:1196 - INFO - Removing @sc-mycluster security group PID: 18709 cli.py:157 - ERROR - InvalidGroup.InUse: There are active instances using security group '@sc-mycluster' PID: 25423 config.py:515 - DEBUG - Loading config PID: 25423 config.py:108 - DEBUG - Loading file: /home/jpk37/.starcluster/config PID: 25423 config.py:515 - DEBUG - Loading config PID: 25423 config.py:108 - DEBUG - Loading file: /home/jpk37/.starcluster/config PID: 25423 awsutils.py:54 - DEBUG - creating self._conn w/ connection_authenticator kwargs = {'path': '/', 'region': None, 'port': None, 'is_secure': True} PID: 25423 cluster.py:670 - DEBUG - existing nodes: {} PID: 25423 cluster.py:686 - DEBUG - returning self._nodes = [] PID: 25423 cluster.py:670 - DEBUG - existing nodes: {} PID: 25423 cluster.py:686 - DEBUG - returning self._nodes = [] PID: 25423 cli.py:170 - ERROR - Cluster with tag name mycluster already exists. If you want to reconfigure the existing instances use the 'restart' command: $ starcluster restart mycluster This will reboot all of the instances and configure the cluster starting from scratch. Otherwise either choose a different tag name, or terminate the existing cluster using: $ starcluster terminate mycluster PID: 25424 config.py:515 - DEBUG - Loading config PID: 25424 config.py:108 - DEBUG - Loading file: /home/jpk37/.starcluster/config PID: 25424 awsutils.py:54 - DEBUG - creating self._conn w/ connection_authenticator kwargs = {'path': '/', 'region': None, 'port': None, 'is_secure': True} PID: 25424 cluster.py:670 - DEBUG - existing nodes: {} PID: 25424 cluster.py:686 - DEBUG - returning self._nodes = [] PID: 25424 cli.py:170 - ERROR - No running nodes found PID: 25425 config.py:515 - DEBUG - Loading config PID: 25425 config.py:108 - DEBUG - Loading file: /home/jpk37/.starcluster/config PID: 25425 awsutils.py:54 - DEBUG - creating self._conn w/ connection_authenticator kwargs = {'path': '/', 'region': None, 'port': None, 'is_secure': True} PID: 25425 cluster.py:670 - DEBUG - existing nodes: {} PID: 25425 cluster.py:686 - DEBUG - returning self._nodes = [] PID: 25425 cluster.py:670 - DEBUG - existing nodes: {} PID: 25425 cluster.py:686 - DEBUG - returning self._nodes = [] PID: 25425 cluster.py:670 - DEBUG - existing nodes: {} PID: 25425 cluster.py:686 - DEBUG - returning self._nodes = [] PID: 25425 cluster.py:1196 - INFO - Removing @sc-mycluster security group PID: 25427 config.py:515 - DEBUG - Loading config PID: 25427 config.py:108 - DEBUG - Loading file: /home/jpk37/.starcluster/config PID: 25427 config.py:515 - DEBUG - Loading config PID: 25427 config.py:108 - DEBUG - Loading file: /home/jpk37/.starcluster/config PID: 25427 awsutils.py:54 - DEBUG - creating self._conn w/ connection_authenticator kwargs = {'path': '/', 'region': None, 'port': None, 'is_secure': True} PID: 25427 start.py:167 - INFO - Using default cluster template: smallcluster PID: 25427 cluster.py:1333 - INFO - Validating cluster template settings... PID: 25427 cluster.py:872 - DEBUG - Launch map: node001 (ami: ami-8cf913e5, type: m1.small)... PID: 25427 cluster.py:1355 - INFO - Cluster template settings are valid PID: 25427 cluster.py:1228 - INFO - Starting cluster... PID: 25427 cluster.py:891 - INFO - Launching a 2-node cluster... PID: 25427 cluster.py:908 - INFO - Launching a 2-node cluster... PID: 25427 cluster.py:872 - DEBUG - Launch map: node001 (ami: ami-8cf913e5, type: m1.small)... PID: 25427 cluster.py:919 - DEBUG - Launching master (ami: ami-8cf913e5, type: m1.small) PID: 25427 cluster.py:919 - DEBUG - Launching node001 (ami: ami-8cf913e5, type: m1.small) PID: 25427 awsutils.py:175 - INFO - Creating security group @sc-mycluster... PID: 25427 cluster.py:1048 - INFO - Waiting for cluster to come up... (updating every 30s) PID: 25427 cluster.py:670 - DEBUG - existing nodes: {} PID: 25427 cluster.py:678 - DEBUG - adding node i-d7dbbeb6 to self._nodes list PID: 25427 cluster.py:678 - DEBUG - adding node i-d9dbbeb8 to self._nodes list PID: 25427 cluster.py:686 - DEBUG - returning self._nodes = [, ] PID: 25427 cluster.py:1070 - INFO - Waiting for all nodes to be in a 'running' state... PID: 25427 cluster.py:670 - DEBUG - existing nodes: {u'i-d9dbbeb8': , u'i-d7dbbeb6': } PID: 25427 cluster.py:673 - DEBUG - updating existing node i-d7dbbeb6 in self._nodes PID: 25427 cluster.py:673 - DEBUG - updating existing node i-d9dbbeb8 in self._nodes PID: 25427 cluster.py:686 - DEBUG - returning self._nodes = [, ] PID: 25427 cluster.py:670 - DEBUG - existing nodes: {u'i-d9dbbeb8': , u'i-d7dbbeb6': } PID: 25427 cluster.py:673 - DEBUG - updating existing node i-d7dbbeb6 in self._nodes PID: 25427 cluster.py:673 - DEBUG - updating existing node i-d9dbbeb8 in self._nodes PID: 25427 cluster.py:686 - DEBUG - returning self._nodes = [, ] PID: 25427 cluster.py:670 - DEBUG - existing nodes: {u'i-d9dbbeb8': , u'i-d7dbbeb6': } PID: 25427 cluster.py:673 - DEBUG - updating existing node i-d7dbbeb6 in self._nodes PID: 25427 cluster.py:673 - DEBUG - updating existing node i-d9dbbeb8 in self._nodes PID: 25427 cluster.py:686 - DEBUG - returning self._nodes = [, ] PID: 25427 cluster.py:670 - DEBUG - existing nodes: {u'i-d9dbbeb8': , u'i-d7dbbeb6': } PID: 25427 cluster.py:673 - DEBUG - updating existing node i-d7dbbeb6 in self._nodes PID: 25427 cluster.py:673 - DEBUG - updating existing node i-d9dbbeb8 in self._nodes PID: 25427 cluster.py:686 - DEBUG - returning self._nodes = [, ] PID: 25427 cluster.py:670 - DEBUG - existing nodes: {u'i-d9dbbeb8': , u'i-d7dbbeb6': } PID: 25427 cluster.py:673 - DEBUG - updating existing node i-d7dbbeb6 in self._nodes PID: 25427 cluster.py:673 - DEBUG - updating existing node i-d9dbbeb8 in self._nodes PID: 25427 cluster.py:686 - DEBUG - returning self._nodes = [, ] PID: 25427 cluster.py:670 - DEBUG - existing nodes: {u'i-d9dbbeb8': , u'i-d7dbbeb6': } PID: 25427 cluster.py:673 - DEBUG - updating existing node i-d7dbbeb6 in self._nodes PID: 25427 cluster.py:673 - DEBUG - updating existing node i-d9dbbeb8 in self._nodes PID: 25427 cluster.py:686 - DEBUG - returning self._nodes = [, ] PID: 25427 cluster.py:670 - DEBUG - existing nodes: {u'i-d9dbbeb8': , u'i-d7dbbeb6': } PID: 25427 cluster.py:673 - DEBUG - updating existing node i-d7dbbeb6 in self._nodes PID: 25427 cluster.py:673 - DEBUG - updating existing node i-d9dbbeb8 in self._nodes PID: 25427 cluster.py:686 - DEBUG - returning self._nodes = [, ] PID: 25427 cluster.py:1081 - INFO - Waiting for SSH to come up on all nodes... PID: 25427 ssh.py:61 - DEBUG - loading private key /home/jpk37/Downloads/blastkey.pem PID: 25427 ssh.py:68 - DEBUG - specified key does not end in either rsa or dsa, trying both PID: 25427 ssh.py:145 - DEBUG - Using private key /home/jpk37/Downloads/blastkey.pem (rsa) PID: 25427 ssh.py:82 - DEBUG - connecting to host ec2-107-20-40-226.compute-1.amazonaws.com on port 22 as user root PID: 25427 ssh.py:61 - DEBUG - loading private key /home/jpk37/Downloads/blastkey.pem PID: 25427 ssh.py:68 - DEBUG - specified key does not end in either rsa or dsa, trying both PID: 25427 ssh.py:145 - DEBUG - Using private key /home/jpk37/Downloads/blastkey.pem (rsa) PID: 25427 ssh.py:82 - DEBUG - connecting to host ec2-107-20-44-237.compute-1.amazonaws.com on port 22 as user root PID: 25427 cluster.py:670 - DEBUG - existing nodes: {u'i-d9dbbeb8': , u'i-d7dbbeb6': } PID: 25427 cluster.py:673 - DEBUG - updating existing node i-d7dbbeb6 in self._nodes PID: 25427 cluster.py:673 - DEBUG - updating existing node i-d9dbbeb8 in self._nodes PID: 25427 cluster.py:686 - DEBUG - returning self._nodes = [, ] PID: 25427 ssh.py:82 - DEBUG - connecting to host ec2-107-20-44-237.compute-1.amazonaws.com on port 22 as user root PID: 25427 cluster.py:670 - DEBUG - existing nodes: {u'i-d9dbbeb8': , u'i-d7dbbeb6': } PID: 25427 cluster.py:673 - DEBUG - updating existing node i-d7dbbeb6 in self._nodes PID: 25427 cluster.py:673 - DEBUG - updating existing node i-d9dbbeb8 in self._nodes PID: 25427 cluster.py:686 - DEBUG - returning self._nodes = [, ] PID: 25427 ssh.py:82 - DEBUG - connecting to host ec2-107-20-44-237.compute-1.amazonaws.com on port 22 as user root PID: 25427 cluster.py:670 - DEBUG - existing nodes: {u'i-d9dbbeb8': , u'i-d7dbbeb6': } PID: 25427 cluster.py:673 - DEBUG - updating existing node i-d7dbbeb6 in self._nodes PID: 25427 cluster.py:673 - DEBUG - updating existing node i-d9dbbeb8 in self._nodes PID: 25427 cluster.py:686 - DEBUG - returning self._nodes = [, ] PID: 25427 ssh.py:82 - DEBUG - connecting to host ec2-107-20-44-237.compute-1.amazonaws.com on port 22 as user root PID: 25427 cluster.py:670 - DEBUG - existing nodes: {u'i-d9dbbeb8': , u'i-d7dbbeb6': } PID: 25427 cluster.py:673 - DEBUG - updating existing node i-d7dbbeb6 in self._nodes PID: 25427 cluster.py:673 - DEBUG - updating existing node i-d9dbbeb8 in self._nodes PID: 25427 cluster.py:686 - DEBUG - returning self._nodes = [, ] PID: 25427 ssh.py:82 - DEBUG - connecting to host ec2-107-20-44-237.compute-1.amazonaws.com on port 22 as user root PID: 25427 cluster.py:670 - DEBUG - existing nodes: {u'i-d9dbbeb8': , u'i-d7dbbeb6': } PID: 25427 cluster.py:673 - DEBUG - updating existing node i-d7dbbeb6 in self._nodes PID: 25427 cluster.py:673 - DEBUG - updating existing node i-d9dbbeb8 in self._nodes PID: 25427 cluster.py:686 - DEBUG - returning self._nodes = [, ] PID: 25427 ssh.py:82 - DEBUG - connecting to host ec2-107-20-44-237.compute-1.amazonaws.com on port 22 as user root PID: 25427 cluster.py:670 - DEBUG - existing nodes: {u'i-d9dbbeb8': , u'i-d7dbbeb6': } PID: 25427 cluster.py:673 - DEBUG - updating existing node i-d7dbbeb6 in self._nodes PID: 25427 cluster.py:673 - DEBUG - updating existing node i-d9dbbeb8 in self._nodes PID: 25427 cluster.py:686 - DEBUG - returning self._nodes = [, ] PID: 25427 ssh.py:82 - DEBUG - connecting to host ec2-107-20-44-237.compute-1.amazonaws.com on port 22 as user root PID: 25427 cluster.py:670 - DEBUG - existing nodes: {u'i-d9dbbeb8': , u'i-d7dbbeb6': } PID: 25427 cluster.py:673 - DEBUG - updating existing node i-d7dbbeb6 in self._nodes PID: 25427 cluster.py:673 - DEBUG - updating existing node i-d9dbbeb8 in self._nodes PID: 25427 cluster.py:686 - DEBUG - returning self._nodes = [, ] PID: 25427 ssh.py:82 - DEBUG - connecting to host ec2-107-20-44-237.compute-1.amazonaws.com on port 22 as user root PID: 25427 cluster.py:670 - DEBUG - existing nodes: {u'i-d9dbbeb8': , u'i-d7dbbeb6': } PID: 25427 cluster.py:673 - DEBUG - updating existing node i-d7dbbeb6 in self._nodes PID: 25427 cluster.py:673 - DEBUG - updating existing node i-d9dbbeb8 in self._nodes PID: 25427 cluster.py:686 - DEBUG - returning self._nodes = [, ] PID: 25427 ssh.py:82 - DEBUG - connecting to host ec2-107-20-44-237.compute-1.amazonaws.com on port 22 as user root PID: 25427 cluster.py:670 - DEBUG - existing nodes: {u'i-d9dbbeb8': , u'i-d7dbbeb6': } PID: 25427 cluster.py:673 - DEBUG - updating existing node i-d7dbbeb6 in self._nodes PID: 25427 cluster.py:673 - DEBUG - updating existing node i-d9dbbeb8 in self._nodes PID: 25427 cluster.py:686 - DEBUG - returning self._nodes = [, ] PID: 25427 ssh.py:82 - DEBUG - connecting to host ec2-107-20-44-237.compute-1.amazonaws.com on port 22 as user root PID: 25427 cluster.py:670 - DEBUG - existing nodes: {u'i-d9dbbeb8': , u'i-d7dbbeb6': } PID: 25427 cluster.py:673 - DEBUG - updating existing node i-d7dbbeb6 in self._nodes PID: 25427 cluster.py:673 - DEBUG - updating existing node i-d9dbbeb8 in self._nodes PID: 25427 cluster.py:686 - DEBUG - returning self._nodes = [, ] PID: 25427 ssh.py:82 - DEBUG - connecting to host ec2-107-20-44-237.compute-1.amazonaws.com on port 22 as user root PID: 25427 cluster.py:670 - DEBUG - existing nodes: {u'i-d9dbbeb8': , u'i-d7dbbeb6': } PID: 25427 cluster.py:673 - DEBUG - updating existing node i-d7dbbeb6 in self._nodes PID: 25427 cluster.py:673 - DEBUG - updating existing node i-d9dbbeb8 in self._nodes PID: 25427 cluster.py:686 - DEBUG - returning self._nodes = [, ] PID: 25427 ssh.py:82 - DEBUG - connecting to host ec2-107-20-44-237.compute-1.amazonaws.com on port 22 as user root PID: 25427 cluster.py:670 - DEBUG - existing nodes: {u'i-d9dbbeb8': , u'i-d7dbbeb6': } PID: 25427 cluster.py:673 - DEBUG - updating existing node i-d7dbbeb6 in self._nodes PID: 25427 cluster.py:673 - DEBUG - updating existing node i-d9dbbeb8 in self._nodes PID: 25427 cluster.py:686 - DEBUG - returning self._nodes = [, ] PID: 25427 ssh.py:82 - DEBUG - connecting to host ec2-107-20-44-237.compute-1.amazonaws.com on port 22 as user root PID: 25427 cluster.py:670 - DEBUG - existing nodes: {u'i-d9dbbeb8': , u'i-d7dbbeb6': } PID: 25427 cluster.py:673 - DEBUG - updating existing node i-d7dbbeb6 in self._nodes PID: 25427 cluster.py:673 - DEBUG - updating existing node i-d9dbbeb8 in self._nodes PID: 25427 cluster.py:686 - DEBUG - returning self._nodes = [, ] PID: 25427 ssh.py:82 - DEBUG - connecting to host ec2-107-20-44-237.compute-1.amazonaws.com on port 22 as user root PID: 25427 cluster.py:670 - DEBUG - existing nodes: {u'i-d9dbbeb8': , u'i-d7dbbeb6': } PID: 25427 cluster.py:673 - DEBUG - updating existing node i-d7dbbeb6 in self._nodes PID: 25427 cluster.py:673 - DEBUG - updating existing node i-d9dbbeb8 in self._nodes PID: 25427 cluster.py:686 - DEBUG - returning self._nodes = [, ] PID: 25427 ssh.py:82 - DEBUG - connecting to host ec2-107-20-44-237.compute-1.amazonaws.com on port 22 as user root PID: 25427 cluster.py:670 - DEBUG - existing nodes: {u'i-d9dbbeb8': , u'i-d7dbbeb6': } PID: 25427 cluster.py:673 - DEBUG - updating existing node i-d7dbbeb6 in self._nodes PID: 25427 cluster.py:673 - DEBUG - updating existing node i-d9dbbeb8 in self._nodes PID: 25427 cluster.py:686 - DEBUG - returning self._nodes = [, ] PID: 25427 ssh.py:82 - DEBUG - connecting to host ec2-107-20-44-237.compute-1.amazonaws.com on port 22 as user root PID: 25427 cluster.py:670 - DEBUG - existing nodes: {u'i-d9dbbeb8': , u'i-d7dbbeb6': } PID: 25427 cluster.py:673 - DEBUG - updating existing node i-d7dbbeb6 in self._nodes PID: 25427 cluster.py:673 - DEBUG - updating existing node i-d9dbbeb8 in self._nodes PID: 25427 cluster.py:686 - DEBUG - returning self._nodes = [, ] PID: 25427 ssh.py:82 - DEBUG - connecting to host ec2-107-20-44-237.compute-1.amazonaws.com on port 22 as user root PID: 25427 cluster.py:670 - DEBUG - existing nodes: {u'i-d9dbbeb8': , u'i-d7dbbeb6': } PID: 25427 cluster.py:673 - DEBUG - updating existing node i-d7dbbeb6 in self._nodes PID: 25427 cluster.py:673 - DEBUG - updating existing node i-d9dbbeb8 in self._nodes PID: 25427 cluster.py:686 - DEBUG - returning self._nodes = [, ] PID: 25427 ssh.py:82 - DEBUG - connecting to host ec2-107-20-44-237.compute-1.amazonaws.com on port 22 as user root PID: 25427 cluster.py:670 - DEBUG - existing nodes: {u'i-d9dbbeb8': , u'i-d7dbbeb6': } PID: 25427 cluster.py:673 - DEBUG - updating existing node i-d7dbbeb6 in self._nodes PID: 25427 cluster.py:673 - DEBUG - updating existing node i-d9dbbeb8 in self._nodes PID: 25427 cluster.py:686 - DEBUG - returning self._nodes = [, ] PID: 25427 ssh.py:82 - DEBUG - connecting to host ec2-107-20-44-237.compute-1.amazonaws.com on port 22 as user root PID: 25427 cluster.py:670 - DEBUG - existing nodes: {u'i-d9dbbeb8': , u'i-d7dbbeb6': } PID: 25427 cluster.py:673 - DEBUG - updating existing node i-d7dbbeb6 in self._nodes PID: 25427 cluster.py:673 - DEBUG - updating existing node i-d9dbbeb8 in self._nodes PID: 25427 cluster.py:686 - DEBUG - returning self._nodes = [, ] PID: 25427 ssh.py:82 - DEBUG - connecting to host ec2-107-20-44-237.compute-1.amazonaws.com on port 22 as user root PID: 25427 cluster.py:670 - DEBUG - existing nodes: {u'i-d9dbbeb8': , u'i-d7dbbeb6': } PID: 25427 cluster.py:673 - DEBUG - updating existing node i-d7dbbeb6 in self._nodes PID: 25427 cluster.py:673 - DEBUG - updating existing node i-d9dbbeb8 in self._nodes PID: 25427 cluster.py:686 - DEBUG - returning self._nodes = [, ] PID: 25427 ssh.py:82 - DEBUG - connecting to host ec2-107-20-44-237.compute-1.amazonaws.com on port 22 as user root PID: 25427 cluster.py:670 - DEBUG - existing nodes: {u'i-d9dbbeb8': , u'i-d7dbbeb6': } PID: 25427 cluster.py:673 - DEBUG - updating existing node i-d7dbbeb6 in self._nodes PID: 25427 cluster.py:673 - DEBUG - updating existing node i-d9dbbeb8 in self._nodes PID: 25427 cluster.py:686 - DEBUG - returning self._nodes = [, ] PID: 25427 ssh.py:82 - DEBUG - connecting to host ec2-107-20-44-237.compute-1.amazonaws.com on port 22 as user root PID: 25427 cluster.py:670 - DEBUG - existing nodes: {u'i-d9dbbeb8': , u'i-d7dbbeb6': } PID: 25427 cluster.py:673 - DEBUG - updating existing node i-d7dbbeb6 in self._nodes PID: 25427 cluster.py:673 - DEBUG - updating existing node i-d9dbbeb8 in self._nodes PID: 25427 cluster.py:686 - DEBUG - returning self._nodes = [, ] PID: 25427 ssh.py:82 - DEBUG - connecting to host ec2-107-20-44-237.compute-1.amazonaws.com on port 22 as user root PID: 25427 cluster.py:670 - DEBUG - existing nodes: {u'i-d9dbbeb8': , u'i-d7dbbeb6': } PID: 25427 cluster.py:673 - DEBUG - updating existing node i-d7dbbeb6 in self._nodes PID: 25427 cluster.py:673 - DEBUG - updating existing node i-d9dbbeb8 in self._nodes PID: 25427 cluster.py:686 - DEBUG - returning self._nodes = [, ] PID: 25427 ssh.py:82 - DEBUG - connecting to host ec2-107-20-44-237.compute-1.amazonaws.com on port 22 as user root PID: 25427 cluster.py:670 - DEBUG - existing nodes: {u'i-d9dbbeb8': , u'i-d7dbbeb6': } PID: 25427 cluster.py:673 - DEBUG - updating existing node i-d7dbbeb6 in self._nodes PID: 25427 cluster.py:673 - DEBUG - updating existing node i-d9dbbeb8 in self._nodes PID: 25427 cluster.py:686 - DEBUG - returning self._nodes = [, ] PID: 25427 ssh.py:82 - DEBUG - connecting to host ec2-107-20-44-237.compute-1.amazonaws.com on port 22 as user root PID: 25427 cluster.py:670 - DEBUG - existing nodes: {u'i-d9dbbeb8': , u'i-d7dbbeb6': } PID: 25427 cluster.py:673 - DEBUG - updating existing node i-d7dbbeb6 in self._nodes PID: 25427 cluster.py:673 - DEBUG - updating existing node i-d9dbbeb8 in self._nodes PID: 25427 cluster.py:686 - DEBUG - returning self._nodes = [, ] PID: 25427 ssh.py:82 - DEBUG - connecting to host ec2-107-20-44-237.compute-1.amazonaws.com on port 22 as user root PID: 25427 cluster.py:670 - DEBUG - existing nodes: {u'i-d9dbbeb8': , u'i-d7dbbeb6': } PID: 25427 cluster.py:673 - DEBUG - updating existing node i-d7dbbeb6 in self._nodes PID: 25427 cluster.py:673 - DEBUG - updating existing node i-d9dbbeb8 in self._nodes PID: 25427 cluster.py:686 - DEBUG - returning self._nodes = [, ] PID: 25427 ssh.py:82 - DEBUG - connecting to host ec2-107-20-44-237.compute-1.amazonaws.com on port 22 as user root PID: 25427 cluster.py:670 - DEBUG - existing nodes: {u'i-d9dbbeb8': , u'i-d7dbbeb6': } PID: 25427 cluster.py:673 - DEBUG - updating existing node i-d7dbbeb6 in self._nodes PID: 25427 cluster.py:673 - DEBUG - updating existing node i-d9dbbeb8 in self._nodes PID: 25427 cluster.py:686 - DEBUG - returning self._nodes = [, ] PID: 25427 ssh.py:82 - DEBUG - connecting to host ec2-107-20-44-237.compute-1.amazonaws.com on port 22 as user root PID: 25568 config.py:515 - DEBUG - Loading config PID: 25568 config.py:108 - DEBUG - Loading file: /home/jpk37/.starcluster/config PID: 25568 config.py:515 - DEBUG - Loading config PID: 25568 config.py:108 - DEBUG - Loading file: /home/jpk37/.starcluster/config PID: 25568 awsutils.py:54 - DEBUG - creating self._conn w/ connection_authenticator kwargs = {'path': '/', 'region': None, 'port': None, 'is_secure': True} PID: 25568 cluster.py:670 - DEBUG - existing nodes: {} PID: 25568 cluster.py:678 - DEBUG - adding node i-d7dbbeb6 to self._nodes list PID: 25568 cluster.py:678 - DEBUG - adding node i-d9dbbeb8 to self._nodes list PID: 25568 cluster.py:686 - DEBUG - returning self._nodes = [, ] PID: 25568 cluster.py:670 - DEBUG - existing nodes: {u'i-d9dbbeb8': , u'i-d7dbbeb6': } PID: 25568 cluster.py:673 - DEBUG - updating existing node i-d7dbbeb6 in self._nodes PID: 25568 cluster.py:673 - DEBUG - updating existing node i-d9dbbeb8 in self._nodes PID: 25568 cluster.py:686 - DEBUG - returning self._nodes = [, ] PID: 25568 cli.py:170 - ERROR - Cluster with tag name mycluster already exists. If you want to reconfigure the existing instances use the 'restart' command: $ starcluster restart mycluster This will reboot all of the instances and configure the cluster starting from scratch. Otherwise either choose a different tag name, or terminate the existing cluster using: $ starcluster terminate mycluster PID: 25569 config.py:515 - DEBUG - Loading config PID: 25569 config.py:108 - DEBUG - Loading file: /home/jpk37/.starcluster/config PID: 25569 awsutils.py:54 - DEBUG - creating self._conn w/ connection_authenticator kwargs = {'path': '/', 'region': None, 'port': None, 'is_secure': True} PID: 25569 cluster.py:670 - DEBUG - existing nodes: {} PID: 25569 cluster.py:678 - DEBUG - adding node i-d7dbbeb6 to self._nodes list PID: 25569 cluster.py:678 - DEBUG - adding node i-d9dbbeb8 to self._nodes list PID: 25569 cluster.py:686 - DEBUG - returning self._nodes = [, ] PID: 25569 cluster.py:1152 - INFO - Rebooting cluster... PID: 25569 cluster.py:1156 - INFO - Sleeping for 20 seconds... PID: 25569 cluster.py:1048 - INFO - Waiting for cluster to come up... (updating every 30s) PID: 25569 cluster.py:670 - DEBUG - existing nodes: {u'i-d9dbbeb8': , u'i-d7dbbeb6': } PID: 25569 cluster.py:673 - DEBUG - updating existing node i-d7dbbeb6 in self._nodes PID: 25569 cluster.py:673 - DEBUG - updating existing node i-d9dbbeb8 in self._nodes PID: 25569 cluster.py:686 - DEBUG - returning self._nodes = [, ] PID: 25569 cluster.py:1070 - INFO - Waiting for all nodes to be in a 'running' state... PID: 25569 cluster.py:1081 - INFO - Waiting for SSH to come up on all nodes... PID: 25569 ssh.py:61 - DEBUG - loading private key /home/jpk37/Downloads/blastkey.pem PID: 25569 ssh.py:68 - DEBUG - specified key does not end in either rsa or dsa, trying both PID: 25569 ssh.py:145 - DEBUG - Using private key /home/jpk37/Downloads/blastkey.pem (rsa) PID: 25569 ssh.py:82 - DEBUG - connecting to host ec2-107-20-40-226.compute-1.amazonaws.com on port 22 as user root PID: 25569 ssh.py:61 - DEBUG - loading private key /home/jpk37/Downloads/blastkey.pem PID: 25569 ssh.py:68 - DEBUG - specified key does not end in either rsa or dsa, trying both PID: 25569 ssh.py:145 - DEBUG - Using private key /home/jpk37/Downloads/blastkey.pem (rsa) PID: 25569 ssh.py:82 - DEBUG - connecting to host ec2-107-20-44-237.compute-1.amazonaws.com on port 22 as user root PID: 25569 cluster.py:670 - DEBUG - existing nodes: {u'i-d9dbbeb8': , u'i-d7dbbeb6': } PID: 25569 cluster.py:673 - DEBUG - updating existing node i-d7dbbeb6 in self._nodes PID: 25569 cluster.py:673 - DEBUG - updating existing node i-d9dbbeb8 in self._nodes PID: 25569 cluster.py:686 - DEBUG - returning self._nodes = [, ] PID: 25569 ssh.py:82 - DEBUG - connecting to host ec2-107-20-44-237.compute-1.amazonaws.com on port 22 as user root PID: 25569 cluster.py:1252 - INFO - The master node is ec2-107-20-40-226.compute-1.amazonaws.com PID: 25569 cluster.py:1253 - INFO - Setting up the cluster... PID: 25569 cluster.py:1127 - INFO - Attaching volume vol-b2ff16d8 to master node on /dev/sdz ... PID: 25569 cluster.py:1129 - DEBUG - resp = attaching PID: 25569 cluster.py:670 - DEBUG - existing nodes: {u'i-d9dbbeb8': , u'i-d7dbbeb6': } PID: 25569 cluster.py:673 - DEBUG - updating existing node i-d7dbbeb6 in self._nodes PID: 25569 cluster.py:673 - DEBUG - updating existing node i-d9dbbeb8 in self._nodes PID: 25569 cluster.py:686 - DEBUG - returning self._nodes = [, ] PID: 25569 clustersetup.py:96 - INFO - Configuring hostnames... PID: 25569 threadpool.py:108 - DEBUG - unfinished_tasks = 2 PID: 25569 ssh.py:164 - DEBUG - creating sftp connection PID: 25569 ssh.py:164 - DEBUG - creating sftp connection PID: 25569 clustersetup.py:262 - INFO - Mounting EBS volume vol-b2ff16d8 on /db... PID: 25569 ssh.py:405 - DEBUG - /dev/sda1 on / type ext3 (rw) PID: 25569 ssh.py:405 - DEBUG - proc on /proc type proc (rw,noexec,nosuid,nodev) PID: 25569 ssh.py:405 - DEBUG - none on /sys type sysfs (rw,noexec,nosuid,nodev) PID: 25569 ssh.py:405 - DEBUG - none on /sys/kernel/debug type debugfs (rw) PID: 25569 ssh.py:405 - DEBUG - none on /sys/kernel/security type securityfs (rw) PID: 25569 ssh.py:405 - DEBUG - devtmpfs on /dev type devtmpfs (rw,mode=0755) PID: 25569 ssh.py:405 - DEBUG - none on /dev/pts type devpts (rw,noexec,nosuid,gid=5,mode=0620) PID: 25569 ssh.py:405 - DEBUG - none on /dev/shm type tmpfs (rw,nosuid,nodev) PID: 25569 ssh.py:405 - DEBUG - none on /var/run type tmpfs (rw,nosuid,mode=0755) PID: 25569 ssh.py:405 - DEBUG - none on /var/lock type tmpfs (rw,noexec,nosuid,nodev) PID: 25569 ssh.py:405 - DEBUG - none on /lib/init/rw type tmpfs (rw,nosuid,mode=0755) PID: 25569 ssh.py:405 - DEBUG - /dev/sda2 on /mnt type ext3 (rw) PID: 25569 ssh.py:405 - DEBUG - nfsd on /proc/fs/nfsd type nfsd (rw) PID: 25569 ssh.py:254 - DEBUG - new /etc/fstab after removing regex ( /db ) matches: # /etc/fstab: static file system information. # proc /proc proc nodev,noexec,nosuid 0 0 /dev/sda1 / ext3 defaults 0 0 /dev/sda2 /mnt auto defaults,comment=cloudconfig 0 0 /dev/sda3 none swap sw,comment=cloudconfig 0 0 PID: 25569 clustersetup.py:144 - INFO - Creating cluster user: sgeadmin (uid: 1001, gid: 1001) PID: 25569 threadpool.py:108 - DEBUG - unfinished_tasks = 2 PID: 25569 clustersetup.py:161 - DEBUG - user sgeadmin does not exist, creating... PID: 25569 clustersetup.py:161 - DEBUG - user sgeadmin does not exist, creating... PID: 25569 clustersetup.py:186 - INFO - Configuring scratch space for user: sgeadmin PID: 25569 threadpool.py:108 - DEBUG - unfinished_tasks = 2 PID: 25569 clustersetup.py:195 - INFO - Configuring /etc/hosts on each node PID: 25569 threadpool.py:108 - DEBUG - unfinished_tasks = 2 PID: 25569 ssh.py:254 - DEBUG - new /etc/hosts after removing regex (master|node001) matches: 127.0.0.1 localhost # The following lines are desirable for IPv6 capable hosts ::1 ip6-localhost ip6-loopback fe00::0 ip6-localnet ff00::0 ip6-mcastprefix ff02::1 ip6-allnodes ff02::2 ip6-allrouters ff02::3 ip6-allhosts PID: 25569 ssh.py:254 - DEBUG - new /etc/hosts after removing regex (master|node001) matches: 127.0.0.1 localhost # The following lines are desirable for IPv6 capable hosts ::1 ip6-localhost ip6-loopback fe00::0 ip6-localnet ff00::0 ip6-mcastprefix ff02::1 ip6-allnodes ff02::2 ip6-allrouters ff02::3 ip6-allhosts PID: 25569 clustersetup.py:314 - INFO - Configuring NFS... PID: 25569 ssh.py:405 - DEBUG - Rather than invoking init scripts through /etc/init.d, use the service(8) PID: 25569 ssh.py:405 - DEBUG - utility, e.g. service portmap start PID: 25569 ssh.py:405 - DEBUG - PID: 25569 ssh.py:405 - DEBUG - Since the script you are attempting to invoke has been converted to an PID: 25569 ssh.py:405 - DEBUG - Upstart job, you may also use the start(8) utility, e.g. start portmap PID: 25569 ssh.py:405 - DEBUG - * Exporting directories for NFS kernel daemon... PID: 25569 ssh.py:405 - DEBUG - ...done. PID: 25569 ssh.py:405 - DEBUG - * Starting NFS kernel daemon PID: 25569 ssh.py:405 - DEBUG - ...done. PID: 25569 ssh.py:405 - DEBUG - /dev/sda1 on / type ext3 (rw) PID: 25569 ssh.py:405 - DEBUG - proc on /proc type proc (rw,noexec,nosuid,nodev) PID: 25569 ssh.py:405 - DEBUG - none on /sys type sysfs (rw,noexec,nosuid,nodev) PID: 25569 ssh.py:405 - DEBUG - none on /sys/kernel/debug type debugfs (rw) PID: 25569 ssh.py:405 - DEBUG - none on /sys/kernel/security type securityfs (rw) PID: 25569 ssh.py:405 - DEBUG - devtmpfs on /dev type devtmpfs (rw,mode=0755) PID: 25569 ssh.py:405 - DEBUG - none on /dev/pts type devpts (rw,noexec,nosuid,gid=5,mode=0620) PID: 25569 ssh.py:405 - DEBUG - none on /dev/shm type tmpfs (rw,nosuid,nodev) PID: 25569 ssh.py:405 - DEBUG - none on /var/run type tmpfs (rw,nosuid,mode=0755) PID: 25569 ssh.py:405 - DEBUG - none on /var/lock type tmpfs (rw,noexec,nosuid,nodev) PID: 25569 ssh.py:405 - DEBUG - none on /lib/init/rw type tmpfs (rw,nosuid,mode=0755) PID: 25569 ssh.py:405 - DEBUG - /dev/sda2 on /mnt type ext3 (rw) PID: 25569 ssh.py:405 - DEBUG - nfsd on /proc/fs/nfsd type nfsd (rw) PID: 25569 clustersetup.py:304 - INFO - Mounting shares for node node001 PID: 25569 threadpool.py:108 - DEBUG - unfinished_tasks = 1 PID: 25569 ssh.py:405 - DEBUG - Rather than invoking init scripts through /etc/init.d, use the service(8) PID: 25569 ssh.py:405 - DEBUG - utility, e.g. service portmap start PID: 25569 ssh.py:405 - DEBUG - PID: 25569 ssh.py:405 - DEBUG - Since the script you are attempting to invoke has been converted to an PID: 25569 ssh.py:405 - DEBUG - Upstart job, you may also use the start(8) utility, e.g. start portmap PID: 25569 ssh.py:402 - DEBUG - command mount -t devpts none /dev/pts failed with status 32 PID: 25569 ssh.py:405 - DEBUG - mount: none already mounted or /dev/pts busy PID: 25569 ssh.py:405 - DEBUG - mount: according to mtab, none is already mounted on /dev/pts PID: 25569 ssh.py:254 - DEBUG - new /etc/fstab after removing regex ( /home | /opt/sge6 | /db ) matches: # /etc/fstab: static file system information. # proc /proc proc nodev,noexec,nosuid 0 0 /dev/sda1 / ext3 defaults 0 0 /dev/sda2 /mnt auto defaults,comment=cloudconfig 0 0 /dev/sda3 none swap sw,comment=cloudconfig 0 0 PID: 25569 threadpool.py:108 - DEBUG - unfinished_tasks = 1 PID: 25569 utils.py:61 - INFO - _setup_nfs took 0.217 mins PID: 25569 clustersetup.py:207 - INFO - Configuring passwordless ssh for root PID: 25569 node.py:396 - DEBUG - adding auth_key_contents PID: 25569 clustersetup.py:215 - INFO - Configuring passwordless ssh for sgeadmin PID: 25569 node.py:396 - DEBUG - adding auth_key_contents PID: 25569 node.py:404 - DEBUG - adding conn_pubkey_contents PID: 25569 ssh.py:402 - DEBUG - command pkill -9 sge failed with status 1 PID: 25569 ssh.py:402 - DEBUG - command rm /etc/init.d/sge* failed with status 1 PID: 25569 ssh.py:405 - DEBUG - rm: cannot remove `/etc/init.d/sge*': No such file or directory PID: 25569 ssh.py:405 - DEBUG - lx24-x86 PID: 25569 ssh.py:405 - DEBUG - [: 359: 11: unexpected operator PID: 25569 ssh.py:405 - DEBUG - [: 359: 11: unexpected operator PID: 25569 ssh.py:402 - DEBUG - command pkill -9 sge failed with status 1 PID: 25569 ssh.py:402 - DEBUG - command rm /etc/init.d/sge* failed with status 1 PID: 25569 ssh.py:405 - DEBUG - rm: cannot remove `/etc/init.d/sge*': No such file or directory PID: 25569 ssh.py:405 - DEBUG - lx24-x86 PID: 25569 ssh.py:405 - DEBUG - [: 359: 11: unexpected operator PID: 25569 ssh.py:405 - DEBUG - [: 359: 11: unexpected operator PID: 25569 clustersetup.py:360 - INFO - Installing Sun Grid Engine... PID: 25569 ssh.py:405 - DEBUG - Reading configuration from file ./ec2_sge.conf PID: 25569 ssh.py:405 - DEBUG - [H[2JInstall log can be found in: /opt/sge6/default/common/install_logs/qmaster_install_master_2011-08-05_13:31:03.log PID: 25569 ssh.py:405 - DEBUG - Install log can be found in: /opt/sge6/default/common/install_logs/execd_install_master_2011-08-05_13:31:09.log PID: 25569 ssh.py:405 - DEBUG - [: 359: 11: unexpected operator PID: 25569 ssh.py:405 - DEBUG - [: 359: 11: unexpected operator PID: 25569 ssh.py:405 - DEBUG - [: 359: 11: unexpected operator PID: 25569 ssh.py:405 - DEBUG - [: 359: 11: unexpected operator PID: 25569 ssh.py:405 - DEBUG - root@master modified "all.q" in cluster queue list PID: 25569 ssh.py:405 - DEBUG - adminhost "node001" already exists PID: 25569 ssh.py:405 - DEBUG - submithost "node001" already exists PID: 25569 threadpool.py:108 - DEBUG - unfinished_tasks = 1 PID: 25569 ssh.py:405 - DEBUG - lx24-x86 PID: 25569 ssh.py:405 - DEBUG - [: 359: 11: unexpected operator PID: 25569 ssh.py:405 - DEBUG - [: 359: 11: unexpected operator PID: 25569 ssh.py:405 - DEBUG - adminhost "node001" already exists PID: 25569 ssh.py:405 - DEBUG - submithost "node001" already exists PID: 25569 threadpool.py:108 - DEBUG - unfinished_tasks = 1 PID: 25569 threadpool.py:108 - DEBUG - unfinished_tasks = 1 PID: 25569 threadpool.py:108 - DEBUG - unfinished_tasks = 1 PID: 25569 threadpool.py:108 - DEBUG - unfinished_tasks = 1 PID: 25569 threadpool.py:108 - DEBUG - unfinished_tasks = 1 PID: 25569 threadpool.py:108 - DEBUG - unfinished_tasks = 1 PID: 25569 ssh.py:405 - DEBUG - Reading configuration from file ./ec2_sge.conf PID: 25569 ssh.py:405 - DEBUG - Install log can be found in: /opt/sge6/default/common/install_logs/execd_install_node001_2011-08-05_13:31:16.log PID: 25569 ssh.py:405 - DEBUG - [: 359: 11: unexpected operator PID: 25569 ssh.py:405 - DEBUG - [: 359: 11: unexpected operator PID: 25569 ssh.py:405 - DEBUG - [: 359: 11: unexpected operator PID: 25569 ssh.py:405 - DEBUG - [: 359: 11: unexpected operator PID: 25569 ssh.py:405 - DEBUG - 1 PID: 25569 ssh.py:405 - DEBUG - 1 PID: 25569 ssh.py:405 - DEBUG - root@master added "orte" to parallel environment list PID: 25569 ssh.py:405 - DEBUG - Unable to find "orte" in "pe_list" of "all.q" - Adding new element. PID: 25569 ssh.py:405 - DEBUG - PID: 25569 ssh.py:405 - DEBUG - root@master modified "all.q" in cluster queue list PID: 25569 threadpool.py:96 - INFO - Shutting down threads... PID: 25569 threadpool.py:108 - DEBUG - unfinished_tasks = 2 PID: 25569 utils.py:72 - INFO - Restarting cluster took 1.925 mins PID: 25569 ssh.py:536 - DEBUG - __del__ called PID: 25569 ssh.py:536 - DEBUG - __del__ called PID: 25603 config.py:515 - DEBUG - Loading config PID: 25603 config.py:108 - DEBUG - Loading file: /home/jpk37/.starcluster/config PID: 25603 awsutils.py:54 - DEBUG - creating self._conn w/ connection_authenticator kwargs = {'path': '/', 'region': None, 'port': None, 'is_secure': True} PID: 25603 cluster.py:670 - DEBUG - existing nodes: {} PID: 25603 cluster.py:678 - DEBUG - adding node i-d7dbbeb6 to self._nodes list PID: 25603 cluster.py:678 - DEBUG - adding node i-d9dbbeb8 to self._nodes list PID: 25603 cluster.py:686 - DEBUG - returning self._nodes = [, ] PID: 25603 node.py:822 - DEBUG - using system's ssh client PID: 25603 node.py:825 - DEBUG - ssh_cmd: ssh -i /home/jpk37/Downloads/blastkey.pem root@ec2-107-20-40-226.compute-1.amazonaws.com PID: 25775 config.py:515 - DEBUG - Loading config PID: 25775 config.py:108 - DEBUG - Loading file: /home/jpk37/.starcluster/config PID: 25775 awsutils.py:54 - DEBUG - creating self._conn w/ connection_authenticator kwargs = {'path': '/', 'region': None, 'port': None, 'is_secure': True} PID: 25775 cluster.py:670 - DEBUG - existing nodes: {} PID: 25775 cluster.py:678 - DEBUG - adding node i-d7dbbeb6 to self._nodes list PID: 25775 cluster.py:678 - DEBUG - adding node i-d9dbbeb8 to self._nodes list PID: 25775 cluster.py:686 - DEBUG - returning self._nodes = [, ] PID: 25775 cluster.py:670 - DEBUG - existing nodes: {u'i-d9dbbeb8': , u'i-d7dbbeb6': } PID: 25775 cluster.py:673 - DEBUG - updating existing node i-d7dbbeb6 in self._nodes PID: 25775 cluster.py:673 - DEBUG - updating existing node i-d9dbbeb8 in self._nodes PID: 25775 cluster.py:686 - DEBUG - returning self._nodes = [, ] PID: 25775 node.py:659 - INFO - Detaching volume vol-b2ff16d8 from master PID: 25775 cluster.py:670 - DEBUG - existing nodes: {u'i-d9dbbeb8': , u'i-d7dbbeb6': } PID: 25775 cluster.py:673 - DEBUG - updating existing node i-d7dbbeb6 in self._nodes PID: 25775 cluster.py:673 - DEBUG - updating existing node i-d9dbbeb8 in self._nodes PID: 25775 cluster.py:686 - DEBUG - returning self._nodes = [, ] PID: 25775 node.py:745 - INFO - Terminating node: master (i-d7dbbeb6) PID: 25775 node.py:745 - INFO - Terminating node: node001 (i-d9dbbeb8) PID: 25775 cluster.py:1196 - INFO - Removing @sc-mycluster security group PID: 25775 cli.py:157 - ERROR - InvalidGroup.InUse: There are active instances using security group '@sc-mycluster' PID: 25776 config.py:515 - DEBUG - Loading config PID: 25776 config.py:108 - DEBUG - Loading file: /home/jpk37/.starcluster/config PID: 25776 awsutils.py:54 - DEBUG - creating self._conn w/ connection_authenticator kwargs = {'path': '/', 'region': None, 'port': None, 'is_secure': True} PID: 25776 cluster.py:670 - DEBUG - existing nodes: {} PID: 25776 cluster.py:686 - DEBUG - returning self._nodes = [] PID: 25776 cluster.py:670 - DEBUG - existing nodes: {} PID: 25776 cluster.py:686 - DEBUG - returning self._nodes = [] PID: 25776 cluster.py:670 - DEBUG - existing nodes: {} PID: 25776 cluster.py:686 - DEBUG - returning self._nodes = [] PID: 25776 cluster.py:670 - DEBUG - existing nodes: {} PID: 25776 cluster.py:686 - DEBUG - returning self._nodes = [] PID: 25776 cluster.py:1178 - INFO - Removing @sc-mycluster security group PID: 25776 cli.py:157 - ERROR - InvalidGroup.InUse: There are active instances using security group '@sc-mycluster' PID: 25779 config.py:515 - DEBUG - Loading config PID: 25779 config.py:108 - DEBUG - Loading file: /home/jpk37/.starcluster/config PID: 25779 awsutils.py:54 - DEBUG - creating self._conn w/ connection_authenticator kwargs = {'path': '/', 'region': None, 'port': None, 'is_secure': True} PID: 25779 cluster.py:670 - DEBUG - existing nodes: {} PID: 25779 cluster.py:686 - DEBUG - returning self._nodes = [] PID: 25779 cluster.py:670 - DEBUG - existing nodes: {} PID: 25779 cluster.py:686 - DEBUG - returning self._nodes = [] PID: 25779 cluster.py:670 - DEBUG - existing nodes: {} PID: 25779 cluster.py:686 - DEBUG - returning self._nodes = [] PID: 25779 cluster.py:1196 - INFO - Removing @sc-mycluster security group PID: 25938 config.py:515 - DEBUG - Loading config PID: 25938 config.py:108 - DEBUG - Loading file: /home/jpk37/.starcluster/config PID: 25938 config.py:515 - DEBUG - Loading config PID: 25938 config.py:108 - DEBUG - Loading file: /home/jpk37/.starcluster/config PID: 25938 awsutils.py:54 - DEBUG - creating self._conn w/ connection_authenticator kwargs = {'path': '/', 'region': None, 'port': None, 'is_secure': True} PID: 25938 start.py:167 - INFO - Using default cluster template: smallcluster PID: 25938 cluster.py:1333 - INFO - Validating cluster template settings... PID: 25938 cluster.py:670 - DEBUG - existing nodes: {} PID: 25938 cluster.py:686 - DEBUG - returning self._nodes = [] PID: 25938 start.py:180 - ERROR - settings for cluster template "smallcluster" are not valid: PID: 25938 cli.py:170 - ERROR - volume vol-b2ff16d8 is not available (status: in-use) PID: 25990 config.py:515 - DEBUG - Loading config PID: 25990 config.py:108 - DEBUG - Loading file: /home/jpk37/.starcluster/config PID: 25990 config.py:515 - DEBUG - Loading config PID: 25990 config.py:108 - DEBUG - Loading file: /home/jpk37/.starcluster/config PID: 25990 awsutils.py:54 - DEBUG - creating self._conn w/ connection_authenticator kwargs = {'path': '/', 'region': None, 'port': None, 'is_secure': True} PID: 25990 start.py:167 - INFO - Using default cluster template: smallcluster PID: 25990 cluster.py:1333 - INFO - Validating cluster template settings... PID: 25990 cluster.py:872 - DEBUG - Launch map: node001 (ami: ami-8cf913e5, type: m1.small)... PID: 25990 cluster.py:1355 - INFO - Cluster template settings are valid PID: 25990 cluster.py:1228 - INFO - Starting cluster... PID: 25990 cluster.py:891 - INFO - Launching a 2-node cluster... PID: 25990 cluster.py:908 - INFO - Launching a 2-node cluster... PID: 25990 cluster.py:872 - DEBUG - Launch map: node001 (ami: ami-8cf913e5, type: m1.small)... PID: 25990 cluster.py:919 - DEBUG - Launching master (ami: ami-8cf913e5, type: m1.small) PID: 25990 cluster.py:919 - DEBUG - Launching node001 (ami: ami-8cf913e5, type: m1.small) PID: 25990 awsutils.py:175 - INFO - Creating security group @sc-mycluster... PID: 25990 cluster.py:1048 - INFO - Waiting for cluster to come up... (updating every 30s) PID: 25990 cluster.py:670 - DEBUG - existing nodes: {} PID: 25990 cluster.py:678 - DEBUG - adding node i-15432774 to self._nodes list PID: 25990 cluster.py:678 - DEBUG - adding node i-17432776 to self._nodes list PID: 25990 cluster.py:686 - DEBUG - returning self._nodes = [, ] PID: 25990 cluster.py:1070 - INFO - Waiting for all nodes to be in a 'running' state... PID: 25990 cluster.py:670 - DEBUG - existing nodes: {u'i-15432774': , u'i-17432776': } PID: 25990 cluster.py:673 - DEBUG - updating existing node i-15432774 in self._nodes PID: 25990 cluster.py:673 - DEBUG - updating existing node i-17432776 in self._nodes PID: 25990 cluster.py:686 - DEBUG - returning self._nodes = [, ] PID: 25990 cluster.py:670 - DEBUG - existing nodes: {u'i-15432774': , u'i-17432776': } PID: 25990 cluster.py:673 - DEBUG - updating existing node i-15432774 in self._nodes PID: 25990 cluster.py:673 - DEBUG - updating existing node i-17432776 in self._nodes PID: 25990 cluster.py:686 - DEBUG - returning self._nodes = [, ] PID: 25990 cluster.py:670 - DEBUG - existing nodes: {u'i-15432774': , u'i-17432776': } PID: 25990 cluster.py:673 - DEBUG - updating existing node i-15432774 in self._nodes PID: 25990 cluster.py:673 - DEBUG - updating existing node i-17432776 in self._nodes PID: 25990 cluster.py:686 - DEBUG - returning self._nodes = [, ] PID: 25990 cluster.py:670 - DEBUG - existing nodes: {u'i-15432774': , u'i-17432776': } PID: 25990 cluster.py:673 - DEBUG - updating existing node i-15432774 in self._nodes PID: 25990 cluster.py:673 - DEBUG - updating existing node i-17432776 in self._nodes PID: 25990 cluster.py:686 - DEBUG - returning self._nodes = [, ] PID: 25990 cluster.py:670 - DEBUG - existing nodes: {u'i-15432774': , u'i-17432776': } PID: 25990 cluster.py:673 - DEBUG - updating existing node i-15432774 in self._nodes PID: 25990 cluster.py:673 - DEBUG - updating existing node i-17432776 in self._nodes PID: 25990 cluster.py:686 - DEBUG - returning self._nodes = [, ] PID: 25990 cluster.py:670 - DEBUG - existing nodes: {u'i-15432774': , u'i-17432776': } PID: 25990 cluster.py:673 - DEBUG - updating existing node i-15432774 in self._nodes PID: 25990 cluster.py:673 - DEBUG - updating existing node i-17432776 in self._nodes PID: 25990 cluster.py:686 - DEBUG - returning self._nodes = [, ] PID: 25990 cluster.py:670 - DEBUG - existing nodes: {u'i-15432774': , u'i-17432776': } PID: 25990 cluster.py:673 - DEBUG - updating existing node i-15432774 in self._nodes PID: 25990 cluster.py:673 - DEBUG - updating existing node i-17432776 in self._nodes PID: 25990 cluster.py:686 - DEBUG - returning self._nodes = [, ] PID: 25990 cluster.py:670 - DEBUG - existing nodes: {u'i-15432774': , u'i-17432776': } PID: 25990 cluster.py:673 - DEBUG - updating existing node i-15432774 in self._nodes PID: 25990 cluster.py:673 - DEBUG - updating existing node i-17432776 in self._nodes PID: 25990 cluster.py:686 - DEBUG - returning self._nodes = [, ] PID: 25990 cluster.py:1081 - INFO - Waiting for SSH to come up on all nodes... PID: 25990 ssh.py:61 - DEBUG - loading private key /home/jpk37/Downloads/blastkey.pem PID: 25990 ssh.py:68 - DEBUG - specified key does not end in either rsa or dsa, trying both PID: 25990 ssh.py:145 - DEBUG - Using private key /home/jpk37/Downloads/blastkey.pem (rsa) PID: 25990 ssh.py:82 - DEBUG - connecting to host ec2-184-72-75-86.compute-1.amazonaws.com on port 22 as user root PID: 25990 ssh.py:61 - DEBUG - loading private key /home/jpk37/Downloads/blastkey.pem PID: 25990 ssh.py:68 - DEBUG - specified key does not end in either rsa or dsa, trying both PID: 25990 ssh.py:145 - DEBUG - Using private key /home/jpk37/Downloads/blastkey.pem (rsa) PID: 25990 ssh.py:82 - DEBUG - connecting to host ec2-75-101-231-5.compute-1.amazonaws.com on port 22 as user root PID: 25990 cluster.py:1252 - INFO - The master node is ec2-184-72-75-86.compute-1.amazonaws.com PID: 25990 cluster.py:1253 - INFO - Setting up the cluster... PID: 25990 cluster.py:1127 - INFO - Attaching volume vol-b2ff16d8 to master node on /dev/sdz ... PID: 25990 cluster.py:1129 - DEBUG - resp = attaching PID: 25990 cluster.py:670 - DEBUG - existing nodes: {u'i-15432774': , u'i-17432776': } PID: 25990 cluster.py:673 - DEBUG - updating existing node i-15432774 in self._nodes PID: 25990 cluster.py:673 - DEBUG - updating existing node i-17432776 in self._nodes PID: 25990 cluster.py:686 - DEBUG - returning self._nodes = [, ] PID: 25990 clustersetup.py:96 - INFO - Configuring hostnames... PID: 25990 threadpool.py:108 - DEBUG - unfinished_tasks = 2 PID: 25990 ssh.py:164 - DEBUG - creating sftp connection PID: 25990 ssh.py:164 - DEBUG - creating sftp connection PID: 25990 clustersetup.py:262 - INFO - Mounting EBS volume vol-b2ff16d8 on /db... PID: 25990 ssh.py:405 - DEBUG - /dev/sda1 on / type ext3 (rw) PID: 25990 ssh.py:405 - DEBUG - proc on /proc type proc (rw,noexec,nosuid,nodev) PID: 25990 ssh.py:405 - DEBUG - none on /sys type sysfs (rw,noexec,nosuid,nodev) PID: 25990 ssh.py:405 - DEBUG - none on /sys/kernel/debug type debugfs (rw) PID: 25990 ssh.py:405 - DEBUG - none on /sys/kernel/security type securityfs (rw) PID: 25990 ssh.py:405 - DEBUG - devtmpfs on /dev type devtmpfs (rw,mode=0755) PID: 25990 ssh.py:405 - DEBUG - none on /dev/pts type devpts (rw,noexec,nosuid,gid=5,mode=0620) PID: 25990 ssh.py:405 - DEBUG - none on /dev/shm type tmpfs (rw,nosuid,nodev) PID: 25990 ssh.py:405 - DEBUG - none on /var/run type tmpfs (rw,nosuid,mode=0755) PID: 25990 ssh.py:405 - DEBUG - none on /var/lock type tmpfs (rw,noexec,nosuid,nodev) PID: 25990 ssh.py:405 - DEBUG - none on /lib/init/rw type tmpfs (rw,nosuid,mode=0755) PID: 25990 ssh.py:405 - DEBUG - /dev/sda2 on /mnt type ext3 (rw) PID: 25990 ssh.py:405 - DEBUG - nfsd on /proc/fs/nfsd type nfsd (rw) PID: 25990 ssh.py:254 - DEBUG - new /etc/fstab after removing regex ( /db ) matches: # /etc/fstab: static file system information. # proc /proc proc nodev,noexec,nosuid 0 0 /dev/sda1 / ext3 defaults 0 0 /dev/sda2 /mnt auto defaults,comment=cloudconfig 0 0 /dev/sda3 none swap sw,comment=cloudconfig 0 0 PID: 25990 clustersetup.py:144 - INFO - Creating cluster user: sgeadmin (uid: 1001, gid: 1001) PID: 25990 threadpool.py:108 - DEBUG - unfinished_tasks = 2 PID: 25990 clustersetup.py:161 - DEBUG - user sgeadmin does not exist, creating... PID: 25990 clustersetup.py:161 - DEBUG - user sgeadmin does not exist, creating... PID: 25990 clustersetup.py:186 - INFO - Configuring scratch space for user: sgeadmin PID: 25990 threadpool.py:108 - DEBUG - unfinished_tasks = 2 PID: 25990 clustersetup.py:195 - INFO - Configuring /etc/hosts on each node PID: 25990 threadpool.py:108 - DEBUG - unfinished_tasks = 2 PID: 25990 ssh.py:254 - DEBUG - new /etc/hosts after removing regex (master|node001) matches: 127.0.0.1 localhost # The following lines are desirable for IPv6 capable hosts ::1 ip6-localhost ip6-loopback fe00::0 ip6-localnet ff00::0 ip6-mcastprefix ff02::1 ip6-allnodes ff02::2 ip6-allrouters ff02::3 ip6-allhosts PID: 25990 ssh.py:254 - DEBUG - new /etc/hosts after removing regex (master|node001) matches: 127.0.0.1 localhost # The following lines are desirable for IPv6 capable hosts ::1 ip6-localhost ip6-loopback fe00::0 ip6-localnet ff00::0 ip6-mcastprefix ff02::1 ip6-allnodes ff02::2 ip6-allrouters ff02::3 ip6-allhosts PID: 25990 clustersetup.py:314 - INFO - Configuring NFS... PID: 25990 ssh.py:405 - DEBUG - Rather than invoking init scripts through /etc/init.d, use the service(8) PID: 25990 ssh.py:405 - DEBUG - utility, e.g. service portmap start PID: 25990 ssh.py:405 - DEBUG - PID: 25990 ssh.py:405 - DEBUG - Since the script you are attempting to invoke has been converted to an PID: 25990 ssh.py:405 - DEBUG - Upstart job, you may also use the start(8) utility, e.g. start portmap PID: 25990 ssh.py:405 - DEBUG - * Exporting directories for NFS kernel daemon... PID: 25990 ssh.py:405 - DEBUG - ...done. PID: 25990 ssh.py:405 - DEBUG - * Starting NFS kernel daemon PID: 25990 ssh.py:405 - DEBUG - ...done. PID: 25990 ssh.py:405 - DEBUG - /dev/sda1 on / type ext3 (rw) PID: 25990 ssh.py:405 - DEBUG - proc on /proc type proc (rw,noexec,nosuid,nodev) PID: 25990 ssh.py:405 - DEBUG - none on /sys type sysfs (rw,noexec,nosuid,nodev) PID: 25990 ssh.py:405 - DEBUG - none on /sys/kernel/debug type debugfs (rw) PID: 25990 ssh.py:405 - DEBUG - none on /sys/kernel/security type securityfs (rw) PID: 25990 ssh.py:405 - DEBUG - devtmpfs on /dev type devtmpfs (rw,mode=0755) PID: 25990 ssh.py:405 - DEBUG - none on /dev/pts type devpts (rw,noexec,nosuid,gid=5,mode=0620) PID: 25990 ssh.py:405 - DEBUG - none on /dev/shm type tmpfs (rw,nosuid,nodev) PID: 25990 ssh.py:405 - DEBUG - none on /var/run type tmpfs (rw,nosuid,mode=0755) PID: 25990 ssh.py:405 - DEBUG - none on /var/lock type tmpfs (rw,noexec,nosuid,nodev) PID: 25990 ssh.py:405 - DEBUG - none on /lib/init/rw type tmpfs (rw,nosuid,mode=0755) PID: 25990 ssh.py:405 - DEBUG - /dev/sda2 on /mnt type ext3 (rw) PID: 25990 ssh.py:405 - DEBUG - nfsd on /proc/fs/nfsd type nfsd (rw) PID: 25990 clustersetup.py:304 - INFO - Mounting shares for node node001 PID: 25990 threadpool.py:108 - DEBUG - unfinished_tasks = 1 PID: 25990 ssh.py:405 - DEBUG - Rather than invoking init scripts through /etc/init.d, use the service(8) PID: 25990 ssh.py:405 - DEBUG - utility, e.g. service portmap start PID: 25990 ssh.py:405 - DEBUG - PID: 25990 ssh.py:405 - DEBUG - Since the script you are attempting to invoke has been converted to an PID: 25990 ssh.py:405 - DEBUG - Upstart job, you may also use the start(8) utility, e.g. start portmap PID: 25990 ssh.py:402 - DEBUG - command mount -t devpts none /dev/pts failed with status 32 PID: 25990 ssh.py:405 - DEBUG - mount: none already mounted or /dev/pts busy PID: 25990 ssh.py:405 - DEBUG - mount: according to mtab, none is already mounted on /dev/pts PID: 25990 ssh.py:254 - DEBUG - new /etc/fstab after removing regex ( /home | /opt/sge6 | /db ) matches: # /etc/fstab: static file system information. # proc /proc proc nodev,noexec,nosuid 0 0 /dev/sda1 / ext3 defaults 0 0 /dev/sda2 /mnt auto defaults,comment=cloudconfig 0 0 /dev/sda3 none swap sw,comment=cloudconfig 0 0 PID: 25990 threadpool.py:108 - DEBUG - unfinished_tasks = 1 PID: 25990 utils.py:61 - INFO - _setup_nfs took 0.171 mins PID: 25990 clustersetup.py:207 - INFO - Configuring passwordless ssh for root PID: 25990 node.py:396 - DEBUG - adding auth_key_contents PID: 25990 clustersetup.py:215 - INFO - Configuring passwordless ssh for sgeadmin PID: 25990 node.py:396 - DEBUG - adding auth_key_contents PID: 25990 node.py:404 - DEBUG - adding conn_pubkey_contents PID: 25990 ssh.py:402 - DEBUG - command pkill -9 sge failed with status 1 PID: 25990 ssh.py:402 - DEBUG - command rm /etc/init.d/sge* failed with status 1 PID: 25990 ssh.py:405 - DEBUG - rm: cannot remove `/etc/init.d/sge*': No such file or directory PID: 25990 ssh.py:405 - DEBUG - lx24-x86 PID: 25990 ssh.py:405 - DEBUG - [: 359: 11: unexpected operator PID: 25990 ssh.py:405 - DEBUG - [: 359: 11: unexpected operator PID: 25990 ssh.py:402 - DEBUG - command pkill -9 sge failed with status 1 PID: 25990 ssh.py:402 - DEBUG - command rm /etc/init.d/sge* failed with status 1 PID: 25990 ssh.py:405 - DEBUG - rm: cannot remove `/etc/init.d/sge*': No such file or directory PID: 25990 ssh.py:405 - DEBUG - lx24-x86 PID: 25990 ssh.py:405 - DEBUG - [: 359: 11: unexpected operator PID: 25990 ssh.py:405 - DEBUG - [: 359: 11: unexpected operator PID: 25990 clustersetup.py:360 - INFO - Installing Sun Grid Engine... PID: 25990 ssh.py:405 - DEBUG - Reading configuration from file ./ec2_sge.conf PID: 25990 ssh.py:405 - DEBUG - [H[2JInstall log can be found in: /opt/sge6/default/common/install_logs/qmaster_install_master_2011-08-05_14:45:07.log PID: 25990 ssh.py:405 - DEBUG - Install log can be found in: /opt/sge6/default/common/install_logs/execd_install_master_2011-08-05_14:45:14.log PID: 25990 ssh.py:405 - DEBUG - [: 359: 11: unexpected operator PID: 25990 ssh.py:405 - DEBUG - [: 359: 11: unexpected operator PID: 25990 ssh.py:405 - DEBUG - [: 359: 11: unexpected operator PID: 25990 ssh.py:405 - DEBUG - [: 359: 11: unexpected operator PID: 25990 ssh.py:405 - DEBUG - root@master modified "all.q" in cluster queue list PID: 25990 ssh.py:405 - DEBUG - adminhost "node001" already exists PID: 25990 ssh.py:405 - DEBUG - submithost "node001" already exists PID: 25990 threadpool.py:108 - DEBUG - unfinished_tasks = 1 PID: 25990 ssh.py:405 - DEBUG - lx24-x86 PID: 25990 ssh.py:405 - DEBUG - [: 359: 11: unexpected operator PID: 25990 ssh.py:405 - DEBUG - [: 359: 11: unexpected operator PID: 25990 ssh.py:405 - DEBUG - adminhost "node001" already exists PID: 25990 ssh.py:405 - DEBUG - submithost "node001" already exists PID: 25990 threadpool.py:108 - DEBUG - unfinished_tasks = 1 PID: 25990 threadpool.py:108 - DEBUG - unfinished_tasks = 1 PID: 25990 threadpool.py:108 - DEBUG - unfinished_tasks = 1 PID: 25990 threadpool.py:108 - DEBUG - unfinished_tasks = 1 PID: 25990 threadpool.py:108 - DEBUG - unfinished_tasks = 1 PID: 25990 threadpool.py:108 - DEBUG - unfinished_tasks = 1 PID: 25990 ssh.py:405 - DEBUG - Reading configuration from file ./ec2_sge.conf PID: 25990 ssh.py:405 - DEBUG - Install log can be found in: /opt/sge6/default/common/install_logs/execd_install_node001_2011-08-05_14:45:20.log PID: 25990 ssh.py:405 - DEBUG - [: 359: 11: unexpected operator PID: 25990 ssh.py:405 - DEBUG - [: 359: 11: unexpected operator PID: 25990 ssh.py:405 - DEBUG - [: 359: 11: unexpected operator PID: 25990 ssh.py:405 - DEBUG - [: 359: 11: unexpected operator PID: 25990 ssh.py:405 - DEBUG - 1 PID: 25990 ssh.py:405 - DEBUG - 1 PID: 25990 ssh.py:405 - DEBUG - root@master added "orte" to parallel environment list PID: 25990 ssh.py:405 - DEBUG - Unable to find "orte" in "pe_list" of "all.q" - Adding new element. PID: 25990 ssh.py:405 - DEBUG - PID: 25990 ssh.py:405 - DEBUG - root@master modified "all.q" in cluster queue list PID: 25990 threadpool.py:96 - INFO - Shutting down threads... PID: 25990 threadpool.py:108 - DEBUG - unfinished_tasks = 3 PID: 25990 cluster.py:1243 - INFO - The cluster has been started and configured. Login to the master node as root by running: $ starcluster sshmaster mycluster When you are finished using the cluster, run: $ starcluster stop mycluster to shutdown the cluster and stop paying for service. If this cluster uses EBS instances then the 'stop' command above will put all nodes into a 'stopped' state. The cluster may then be restarted at a later time, without losing data, by passing the -x option to the 'start' command. To completely terminate an EBS cluster: $ starcluster terminate mycluster NOTE: Terminating an EBS cluster will destroy all volumes backing the nodes. PID: 25990 utils.py:72 - INFO - Starting cluster took 5.300 mins PID: 25990 ssh.py:536 - DEBUG - __del__ called PID: 25990 ssh.py:536 - DEBUG - __del__ called PID: 26025 config.py:515 - DEBUG - Loading config PID: 26025 config.py:108 - DEBUG - Loading file: /home/jpk37/.starcluster/config PID: 26025 awsutils.py:54 - DEBUG - creating self._conn w/ connection_authenticator kwargs = {'path': '/', 'region': None, 'port': None, 'is_secure': True} PID: 26025 cluster.py:670 - DEBUG - existing nodes: {} PID: 26025 cluster.py:678 - DEBUG - adding node i-15432774 to self._nodes list PID: 26025 cluster.py:678 - DEBUG - adding node i-17432776 to self._nodes list PID: 26025 cluster.py:686 - DEBUG - returning self._nodes = [, ] PID: 26025 node.py:822 - DEBUG - using system's ssh client PID: 26025 node.py:825 - DEBUG - ssh_cmd: ssh -i /home/jpk37/Downloads/blastkey.pem root@ec2-184-72-75-86.compute-1.amazonaws.com PID: 26161 config.py:515 - DEBUG - Loading config PID: 26161 config.py:108 - DEBUG - Loading file: /home/jpk37/.starcluster/config PID: 26161 awsutils.py:54 - DEBUG - creating self._conn w/ connection_authenticator kwargs = {'path': '/', 'region': None, 'port': None, 'is_secure': True} PID: 26161 cluster.py:670 - DEBUG - existing nodes: {} PID: 26161 cluster.py:678 - DEBUG - adding node i-15432774 to self._nodes list PID: 26161 cluster.py:678 - DEBUG - adding node i-17432776 to self._nodes list PID: 26161 cluster.py:686 - DEBUG - returning self._nodes = [, ] PID: 26161 cluster.py:670 - DEBUG - existing nodes: {u'i-15432774': , u'i-17432776': } PID: 26161 cluster.py:673 - DEBUG - updating existing node i-15432774 in self._nodes PID: 26161 cluster.py:673 - DEBUG - updating existing node i-17432776 in self._nodes PID: 26161 cluster.py:686 - DEBUG - returning self._nodes = [, ] PID: 26161 node.py:659 - INFO - Detaching volume vol-b2ff16d8 from master PID: 26161 cluster.py:670 - DEBUG - existing nodes: {u'i-15432774': , u'i-17432776': } PID: 26161 cluster.py:673 - DEBUG - updating existing node i-15432774 in self._nodes PID: 26161 cluster.py:673 - DEBUG - updating existing node i-17432776 in self._nodes PID: 26161 cluster.py:686 - DEBUG - returning self._nodes = [, ] PID: 26161 node.py:745 - INFO - Terminating node: master (i-15432774) PID: 26161 node.py:745 - INFO - Terminating node: node001 (i-17432776) PID: 26161 cluster.py:1196 - INFO - Removing @sc-mycluster security group PID: 26161 cli.py:157 - ERROR - InvalidGroup.InUse: There are active instances using security group '@sc-mycluster' PID: 26162 config.py:515 - DEBUG - Loading config PID: 26162 config.py:108 - DEBUG - Loading file: /home/jpk37/.starcluster/config PID: 26162 awsutils.py:54 - DEBUG - creating self._conn w/ connection_authenticator kwargs = {'path': '/', 'region': None, 'port': None, 'is_secure': True} PID: 26162 cluster.py:670 - DEBUG - existing nodes: {} PID: 26162 cluster.py:686 - DEBUG - returning self._nodes = [] PID: 26162 cluster.py:670 - DEBUG - existing nodes: {} PID: 26162 cluster.py:686 - DEBUG - returning self._nodes = [] PID: 26162 cluster.py:670 - DEBUG - existing nodes: {} PID: 26162 cluster.py:686 - DEBUG - returning self._nodes = [] PID: 26162 cluster.py:670 - DEBUG - existing nodes: {} PID: 26162 cluster.py:686 - DEBUG - returning self._nodes = [] PID: 26162 cluster.py:1178 - INFO - Removing @sc-mycluster security group PID: 26162 cli.py:157 - ERROR - InvalidGroup.InUse: There are active instances using security group '@sc-mycluster' PID: 26171 config.py:515 - DEBUG - Loading config PID: 26171 config.py:108 - DEBUG - Loading file: /home/jpk37/.starcluster/config PID: 26171 awsutils.py:54 - DEBUG - creating self._conn w/ connection_authenticator kwargs = {'path': '/', 'region': None, 'port': None, 'is_secure': True} PID: 26171 cluster.py:670 - DEBUG - existing nodes: {} PID: 26171 cluster.py:686 - DEBUG - returning self._nodes = [] PID: 26171 cluster.py:670 - DEBUG - existing nodes: {} PID: 26171 cluster.py:686 - DEBUG - returning self._nodes = [] PID: 26171 cluster.py:670 - DEBUG - existing nodes: {} PID: 26171 cluster.py:686 - DEBUG - returning self._nodes = [] PID: 26171 cluster.py:1196 - INFO - Removing @sc-mycluster security group PID: 26177 config.py:515 - DEBUG - Loading config PID: 26177 config.py:108 - DEBUG - Loading file: /home/jpk37/.starcluster/config PID: 26177 config.py:515 - DEBUG - Loading config PID: 26177 config.py:108 - DEBUG - Loading file: /home/jpk37/.starcluster/config PID: 26177 awsutils.py:54 - DEBUG - creating self._conn w/ connection_authenticator kwargs = {'path': '/', 'region': None, 'port': None, 'is_secure': True} PID: 26177 start.py:167 - INFO - Using default cluster template: smallcluster PID: 26177 cluster.py:1333 - INFO - Validating cluster template settings... PID: 26177 cluster.py:872 - DEBUG - Launch map: node001 (ami: ami-0af31963, type: m1.xlarge)... PID: 26177 cluster.py:1355 - INFO - Cluster template settings are valid PID: 26177 cluster.py:1228 - INFO - Starting cluster... PID: 26177 cluster.py:891 - INFO - Launching a 2-node cluster... PID: 26177 cluster.py:908 - INFO - Launching a 2-node cluster... PID: 26177 cluster.py:872 - DEBUG - Launch map: node001 (ami: ami-0af31963, type: m1.xlarge)... PID: 26177 cluster.py:919 - DEBUG - Launching master (ami: ami-0af31963, type: m1.xlarge) PID: 26177 cluster.py:919 - DEBUG - Launching node001 (ami: ami-0af31963, type: m1.xlarge) PID: 26177 awsutils.py:175 - INFO - Creating security group @sc-mycluster... PID: 26177 cluster.py:1048 - INFO - Waiting for cluster to come up... (updating every 30s) PID: 26177 cluster.py:670 - DEBUG - existing nodes: {} PID: 26177 cluster.py:678 - DEBUG - adding node i-edb0d48c to self._nodes list PID: 26177 cluster.py:678 - DEBUG - adding node i-efb0d48e to self._nodes list PID: 26177 cluster.py:686 - DEBUG - returning self._nodes = [, ] PID: 26177 cluster.py:1070 - INFO - Waiting for all nodes to be in a 'running' state... PID: 26177 cluster.py:670 - DEBUG - existing nodes: {u'i-edb0d48c': , u'i-efb0d48e': } PID: 26177 cluster.py:673 - DEBUG - updating existing node i-edb0d48c in self._nodes PID: 26177 cluster.py:673 - DEBUG - updating existing node i-efb0d48e in self._nodes PID: 26177 cluster.py:686 - DEBUG - returning self._nodes = [, ] PID: 26177 cluster.py:670 - DEBUG - existing nodes: {u'i-edb0d48c': , u'i-efb0d48e': } PID: 26177 cluster.py:673 - DEBUG - updating existing node i-edb0d48c in self._nodes PID: 26177 cluster.py:673 - DEBUG - updating existing node i-efb0d48e in self._nodes PID: 26177 cluster.py:686 - DEBUG - returning self._nodes = [, ] PID: 26177 cluster.py:670 - DEBUG - existing nodes: {u'i-edb0d48c': , u'i-efb0d48e': } PID: 26177 cluster.py:673 - DEBUG - updating existing node i-edb0d48c in self._nodes PID: 26177 cluster.py:673 - DEBUG - updating existing node i-efb0d48e in self._nodes PID: 26177 cluster.py:686 - DEBUG - returning self._nodes = [, ] PID: 26177 cluster.py:670 - DEBUG - existing nodes: {u'i-edb0d48c': , u'i-efb0d48e': } PID: 26177 cluster.py:673 - DEBUG - updating existing node i-edb0d48c in self._nodes PID: 26177 cluster.py:673 - DEBUG - updating existing node i-efb0d48e in self._nodes PID: 26177 cluster.py:686 - DEBUG - returning self._nodes = [, ] PID: 26177 cluster.py:670 - DEBUG - existing nodes: {u'i-edb0d48c': , u'i-efb0d48e': } PID: 26177 cluster.py:673 - DEBUG - updating existing node i-edb0d48c in self._nodes PID: 26177 cluster.py:673 - DEBUG - updating existing node i-efb0d48e in self._nodes PID: 26177 cluster.py:686 - DEBUG - returning self._nodes = [, ] PID: 26177 cluster.py:1081 - INFO - Waiting for SSH to come up on all nodes... PID: 26177 ssh.py:61 - DEBUG - loading private key /home/jpk37/Downloads/blastkey.pem PID: 26177 ssh.py:68 - DEBUG - specified key does not end in either rsa or dsa, trying both PID: 26177 ssh.py:145 - DEBUG - Using private key /home/jpk37/Downloads/blastkey.pem (rsa) PID: 26177 ssh.py:82 - DEBUG - connecting to host ec2-50-17-154-227.compute-1.amazonaws.com on port 22 as user root PID: 26177 ssh.py:61 - DEBUG - loading private key /home/jpk37/Downloads/blastkey.pem PID: 26177 ssh.py:68 - DEBUG - specified key does not end in either rsa or dsa, trying both PID: 26177 ssh.py:145 - DEBUG - Using private key /home/jpk37/Downloads/blastkey.pem (rsa) PID: 26177 ssh.py:82 - DEBUG - connecting to host ec2-50-17-68-75.compute-1.amazonaws.com on port 22 as user root PID: 26177 cluster.py:1252 - INFO - The master node is ec2-50-17-154-227.compute-1.amazonaws.com PID: 26177 cluster.py:1253 - INFO - Setting up the cluster... PID: 26177 cluster.py:1127 - INFO - Attaching volume vol-b2ff16d8 to master node on /dev/sdz ... PID: 26177 cluster.py:1129 - DEBUG - resp = attaching PID: 26177 cluster.py:670 - DEBUG - existing nodes: {u'i-edb0d48c': , u'i-efb0d48e': } PID: 26177 cluster.py:673 - DEBUG - updating existing node i-edb0d48c in self._nodes PID: 26177 cluster.py:673 - DEBUG - updating existing node i-efb0d48e in self._nodes PID: 26177 cluster.py:686 - DEBUG - returning self._nodes = [, ] PID: 26177 clustersetup.py:96 - INFO - Configuring hostnames... PID: 26177 ssh.py:164 - DEBUG - creating sftp connection PID: 26177 threadpool.py:108 - DEBUG - unfinished_tasks = 2 PID: 26177 ssh.py:164 - DEBUG - creating sftp connection PID: 26177 clustersetup.py:262 - INFO - Mounting EBS volume vol-b2ff16d8 on /db... PID: 26177 ssh.py:405 - DEBUG - /dev/sda1 on / type ext3 (rw) PID: 26177 ssh.py:405 - DEBUG - proc on /proc type proc (rw,noexec,nosuid,nodev) PID: 26177 ssh.py:405 - DEBUG - none on /sys type sysfs (rw,noexec,nosuid,nodev) PID: 26177 ssh.py:405 - DEBUG - none on /sys/kernel/debug type debugfs (rw) PID: 26177 ssh.py:405 - DEBUG - none on /sys/kernel/security type securityfs (rw) PID: 26177 ssh.py:405 - DEBUG - devtmpfs on /dev type devtmpfs (rw,mode=0755) PID: 26177 ssh.py:405 - DEBUG - none on /dev/pts type devpts (rw,noexec,nosuid,gid=5,mode=0620) PID: 26177 ssh.py:405 - DEBUG - none on /dev/shm type tmpfs (rw,nosuid,nodev) PID: 26177 ssh.py:405 - DEBUG - none on /var/run type tmpfs (rw,nosuid,mode=0755) PID: 26177 ssh.py:405 - DEBUG - none on /var/lock type tmpfs (rw,noexec,nosuid,nodev) PID: 26177 ssh.py:405 - DEBUG - none on /lib/init/rw type tmpfs (rw,nosuid,mode=0755) PID: 26177 ssh.py:405 - DEBUG - /dev/sdb on /mnt type ext3 (rw) PID: 26177 ssh.py:405 - DEBUG - nfsd on /proc/fs/nfsd type nfsd (rw) PID: 26177 ssh.py:254 - DEBUG - new /etc/fstab after removing regex ( /db ) matches: # /etc/fstab: static file system information. # proc /proc proc nodev,noexec,nosuid 0 0 /dev/sda1 / ext3 defaults 0 0 /dev/sdb /mnt auto defaults,comment=cloudconfig 0 0 PID: 26177 clustersetup.py:144 - INFO - Creating cluster user: sgeadmin (uid: 1001, gid: 1001) PID: 26177 threadpool.py:108 - DEBUG - unfinished_tasks = 2 PID: 26177 clustersetup.py:161 - DEBUG - user sgeadmin does not exist, creating... PID: 26177 clustersetup.py:161 - DEBUG - user sgeadmin does not exist, creating... PID: 26177 clustersetup.py:186 - INFO - Configuring scratch space for user: sgeadmin PID: 26177 threadpool.py:108 - DEBUG - unfinished_tasks = 2 PID: 26177 clustersetup.py:195 - INFO - Configuring /etc/hosts on each node PID: 26177 threadpool.py:108 - DEBUG - unfinished_tasks = 2 PID: 26177 ssh.py:254 - DEBUG - new /etc/hosts after removing regex (master|node001) matches: 127.0.0.1 localhost # The following lines are desirable for IPv6 capable hosts ::1 ip6-localhost ip6-loopback fe00::0 ip6-localnet ff00::0 ip6-mcastprefix ff02::1 ip6-allnodes ff02::2 ip6-allrouters ff02::3 ip6-allhosts PID: 26177 ssh.py:254 - DEBUG - new /etc/hosts after removing regex (master|node001) matches: 127.0.0.1 localhost # The following lines are desirable for IPv6 capable hosts ::1 ip6-localhost ip6-loopback fe00::0 ip6-localnet ff00::0 ip6-mcastprefix ff02::1 ip6-allnodes ff02::2 ip6-allrouters ff02::3 ip6-allhosts PID: 26177 clustersetup.py:314 - INFO - Configuring NFS... PID: 26177 ssh.py:405 - DEBUG - Rather than invoking init scripts through /etc/init.d, use the service(8) PID: 26177 ssh.py:405 - DEBUG - utility, e.g. service portmap start PID: 26177 ssh.py:405 - DEBUG - PID: 26177 ssh.py:405 - DEBUG - Since the script you are attempting to invoke has been converted to an PID: 26177 ssh.py:405 - DEBUG - Upstart job, you may also use the start(8) utility, e.g. start portmap PID: 26177 ssh.py:405 - DEBUG - * Exporting directories for NFS kernel daemon... PID: 26177 ssh.py:405 - DEBUG - ...done. PID: 26177 ssh.py:405 - DEBUG - * Starting NFS kernel daemon PID: 26177 ssh.py:405 - DEBUG - ...done. PID: 26177 ssh.py:405 - DEBUG - /dev/sda1 on / type ext3 (rw) PID: 26177 ssh.py:405 - DEBUG - proc on /proc type proc (rw,noexec,nosuid,nodev) PID: 26177 ssh.py:405 - DEBUG - none on /sys type sysfs (rw,noexec,nosuid,nodev) PID: 26177 ssh.py:405 - DEBUG - none on /sys/kernel/debug type debugfs (rw) PID: 26177 ssh.py:405 - DEBUG - none on /sys/kernel/security type securityfs (rw) PID: 26177 ssh.py:405 - DEBUG - devtmpfs on /dev type devtmpfs (rw,mode=0755) PID: 26177 ssh.py:405 - DEBUG - none on /dev/pts type devpts (rw,noexec,nosuid,gid=5,mode=0620) PID: 26177 ssh.py:405 - DEBUG - none on /dev/shm type tmpfs (rw,nosuid,nodev) PID: 26177 ssh.py:405 - DEBUG - none on /var/run type tmpfs (rw,nosuid,mode=0755) PID: 26177 ssh.py:405 - DEBUG - none on /var/lock type tmpfs (rw,noexec,nosuid,nodev) PID: 26177 ssh.py:405 - DEBUG - none on /lib/init/rw type tmpfs (rw,nosuid,mode=0755) PID: 26177 ssh.py:405 - DEBUG - /dev/sdb on /mnt type ext3 (rw) PID: 26177 ssh.py:405 - DEBUG - nfsd on /proc/fs/nfsd type nfsd (rw) PID: 26177 clustersetup.py:304 - INFO - Mounting shares for node node001 PID: 26177 threadpool.py:108 - DEBUG - unfinished_tasks = 1 PID: 26177 ssh.py:405 - DEBUG - Rather than invoking init scripts through /etc/init.d, use the service(8) PID: 26177 ssh.py:405 - DEBUG - utility, e.g. service portmap start PID: 26177 ssh.py:405 - DEBUG - PID: 26177 ssh.py:405 - DEBUG - Since the script you are attempting to invoke has been converted to an PID: 26177 ssh.py:405 - DEBUG - Upstart job, you may also use the start(8) utility, e.g. start portmap PID: 26177 ssh.py:402 - DEBUG - command mount -t devpts none /dev/pts failed with status 32 PID: 26177 ssh.py:405 - DEBUG - mount: none already mounted or /dev/pts busy PID: 26177 ssh.py:405 - DEBUG - mount: according to mtab, none is already mounted on /dev/pts PID: 26177 ssh.py:254 - DEBUG - new /etc/fstab after removing regex ( /home | /opt/sge6 | /db ) matches: # /etc/fstab: static file system information. # proc /proc proc nodev,noexec,nosuid 0 0 /dev/sda1 / ext3 defaults 0 0 /dev/sdb /mnt auto defaults,comment=cloudconfig 0 0 PID: 26177 threadpool.py:108 - DEBUG - unfinished_tasks = 1 PID: 26177 threadpool.py:108 - DEBUG - unfinished_tasks = 1 PID: 26177 threadpool.py:108 - DEBUG - unfinished_tasks = 1 PID: 26177 utils.py:61 - INFO - _setup_nfs took 0.149 mins PID: 26177 clustersetup.py:207 - INFO - Configuring passwordless ssh for root PID: 26177 node.py:396 - DEBUG - adding auth_key_contents PID: 26177 clustersetup.py:215 - INFO - Configuring passwordless ssh for sgeadmin PID: 26177 node.py:396 - DEBUG - adding auth_key_contents PID: 26177 node.py:404 - DEBUG - adding conn_pubkey_contents PID: 26177 ssh.py:402 - DEBUG - command pkill -9 sge failed with status 1 PID: 26177 ssh.py:402 - DEBUG - command rm /etc/init.d/sge* failed with status 1 PID: 26177 ssh.py:405 - DEBUG - rm: cannot remove `/etc/init.d/sge*': No such file or directory PID: 26177 ssh.py:405 - DEBUG - lx24-amd64 PID: 26177 ssh.py:405 - DEBUG - [: 359: 11: unexpected operator PID: 26177 ssh.py:405 - DEBUG - [: 359: 11: unexpected operator PID: 26177 ssh.py:402 - DEBUG - command pkill -9 sge failed with status 1 PID: 26177 ssh.py:402 - DEBUG - command rm /etc/init.d/sge* failed with status 1 PID: 26177 ssh.py:405 - DEBUG - rm: cannot remove `/etc/init.d/sge*': No such file or directory PID: 26177 ssh.py:405 - DEBUG - lx24-amd64 PID: 26177 ssh.py:405 - DEBUG - [: 359: 11: unexpected operator PID: 26177 ssh.py:405 - DEBUG - [: 359: 11: unexpected operator PID: 26177 clustersetup.py:360 - INFO - Installing Sun Grid Engine... PID: 26177 ssh.py:405 - DEBUG - Reading configuration from file ./ec2_sge.conf PID: 26177 ssh.py:405 - DEBUG - [H[2JInstall log can be found in: /opt/sge6/default/common/install_logs/qmaster_install_master_2011-08-05_15:27:34.log PID: 26177 ssh.py:405 - DEBUG - Install log can be found in: /opt/sge6/default/common/install_logs/execd_install_master_2011-08-05_15:27:38.log PID: 26177 ssh.py:405 - DEBUG - [: 359: 11: unexpected operator PID: 26177 ssh.py:405 - DEBUG - [: 359: 11: unexpected operator PID: 26177 ssh.py:405 - DEBUG - [: 359: 11: unexpected operator PID: 26177 ssh.py:405 - DEBUG - [: 359: 11: unexpected operator PID: 26177 ssh.py:405 - DEBUG - root@master modified "all.q" in cluster queue list PID: 26177 ssh.py:405 - DEBUG - adminhost "node001" already exists PID: 26177 ssh.py:405 - DEBUG - submithost "node001" already exists PID: 26177 threadpool.py:108 - DEBUG - unfinished_tasks = 1 PID: 26177 ssh.py:405 - DEBUG - lx24-amd64 PID: 26177 ssh.py:405 - DEBUG - [: 359: 11: unexpected operator PID: 26177 ssh.py:405 - DEBUG - [: 359: 11: unexpected operator PID: 26177 ssh.py:405 - DEBUG - adminhost "node001" already exists PID: 26177 ssh.py:405 - DEBUG - submithost "node001" already exists PID: 26177 threadpool.py:108 - DEBUG - unfinished_tasks = 1 PID: 26177 threadpool.py:108 - DEBUG - unfinished_tasks = 1 PID: 26177 threadpool.py:108 - DEBUG - unfinished_tasks = 1 PID: 26177 threadpool.py:108 - DEBUG - unfinished_tasks = 1 PID: 26177 ssh.py:405 - DEBUG - Reading configuration from file ./ec2_sge.conf PID: 26177 ssh.py:405 - DEBUG - Install log can be found in: /opt/sge6/default/common/install_logs/execd_install_node001_2011-08-05_15:27:43.log PID: 26177 ssh.py:405 - DEBUG - [: 359: 11: unexpected operator PID: 26177 ssh.py:405 - DEBUG - [: 359: 11: unexpected operator PID: 26177 ssh.py:405 - DEBUG - [: 359: 11: unexpected operator PID: 26177 ssh.py:405 - DEBUG - [: 359: 11: unexpected operator PID: 26177 ssh.py:405 - DEBUG - 4 PID: 26177 ssh.py:405 - DEBUG - 4 PID: 26177 ssh.py:405 - DEBUG - root@master added "orte" to parallel environment list PID: 26177 ssh.py:405 - DEBUG - Unable to find "orte" in "pe_list" of "all.q" - Adding new element. PID: 26177 ssh.py:405 - DEBUG - PID: 26177 ssh.py:405 - DEBUG - root@master modified "all.q" in cluster queue list PID: 26177 threadpool.py:96 - INFO - Shutting down threads... PID: 26177 threadpool.py:108 - DEBUG - unfinished_tasks = 2 PID: 26177 cluster.py:1243 - INFO - The cluster has been started and configured. Login to the master node as root by running: $ starcluster sshmaster mycluster When you are finished using the cluster, run: $ starcluster stop mycluster to shutdown the cluster and stop paying for service. If this cluster uses EBS instances then the 'stop' command above will put all nodes into a 'stopped' state. The cluster may then be restarted at a later time, without losing data, by passing the -x option to the 'start' command. To completely terminate an EBS cluster: $ starcluster terminate mycluster NOTE: Terminating an EBS cluster will destroy all volumes backing the nodes. PID: 26177 utils.py:72 - INFO - Starting cluster took 3.395 mins PID: 26177 ssh.py:536 - DEBUG - __del__ called PID: 26177 ssh.py:536 - DEBUG - __del__ called PID: 26217 config.py:515 - DEBUG - Loading config PID: 26217 config.py:108 - DEBUG - Loading file: /home/jpk37/.starcluster/config PID: 26217 awsutils.py:54 - DEBUG - creating self._conn w/ connection_authenticator kwargs = {'path': '/', 'region': None, 'port': None, 'is_secure': True} PID: 26217 cluster.py:670 - DEBUG - existing nodes: {} PID: 26217 cluster.py:678 - DEBUG - adding node i-edb0d48c to self._nodes list PID: 26217 cluster.py:678 - DEBUG - adding node i-efb0d48e to self._nodes list PID: 26217 cluster.py:686 - DEBUG - returning self._nodes = [, ] PID: 26217 node.py:822 - DEBUG - using system's ssh client PID: 26217 node.py:825 - DEBUG - ssh_cmd: ssh -i /home/jpk37/Downloads/blastkey.pem root@ec2-50-17-154-227.compute-1.amazonaws.com PID: 26314 config.py:515 - DEBUG - Loading config PID: 26314 config.py:108 - DEBUG - Loading file: /home/jpk37/.starcluster/config PID: 26314 awsutils.py:54 - DEBUG - creating self._conn w/ connection_authenticator kwargs = {'path': '/', 'region': None, 'port': None, 'is_secure': True} PID: 26314 cluster.py:670 - DEBUG - existing nodes: {} PID: 26314 cluster.py:678 - DEBUG - adding node i-edb0d48c to self._nodes list PID: 26314 cluster.py:678 - DEBUG - adding node i-efb0d48e to self._nodes list PID: 26314 cluster.py:686 - DEBUG - returning self._nodes = [, ] PID: 26314 cluster.py:670 - DEBUG - existing nodes: {u'i-edb0d48c': , u'i-efb0d48e': } PID: 26314 cluster.py:673 - DEBUG - updating existing node i-edb0d48c in self._nodes PID: 26314 cluster.py:673 - DEBUG - updating existing node i-efb0d48e in self._nodes PID: 26314 cluster.py:686 - DEBUG - returning self._nodes = [, ] PID: 26314 node.py:659 - INFO - Detaching volume vol-b2ff16d8 from master PID: 26314 cluster.py:670 - DEBUG - existing nodes: {u'i-edb0d48c': , u'i-efb0d48e': } PID: 26314 cluster.py:673 - DEBUG - updating existing node i-edb0d48c in self._nodes PID: 26314 cluster.py:673 - DEBUG - updating existing node i-efb0d48e in self._nodes PID: 26314 cluster.py:686 - DEBUG - returning self._nodes = [, ] PID: 26314 node.py:745 - INFO - Terminating node: master (i-edb0d48c) PID: 26314 node.py:745 - INFO - Terminating node: node001 (i-efb0d48e) PID: 26314 cluster.py:1196 - INFO - Removing @sc-mycluster security group PID: 26314 cli.py:157 - ERROR - InvalidGroup.InUse: There are active instances using security group '@sc-mycluster' PID: 26319 config.py:515 - DEBUG - Loading config PID: 26319 config.py:108 - DEBUG - Loading file: /home/jpk37/.starcluster/config PID: 26319 awsutils.py:54 - DEBUG - creating self._conn w/ connection_authenticator kwargs = {'path': '/', 'region': None, 'port': None, 'is_secure': True} PID: 26319 cluster.py:670 - DEBUG - existing nodes: {} PID: 26319 cluster.py:686 - DEBUG - returning self._nodes = [] PID: 26319 cluster.py:670 - DEBUG - existing nodes: {} PID: 26319 cluster.py:686 - DEBUG - returning self._nodes = [] PID: 26319 cluster.py:670 - DEBUG - existing nodes: {} PID: 26319 cluster.py:686 - DEBUG - returning self._nodes = [] PID: 26319 cluster.py:1196 - INFO - Removing @sc-mycluster security group PID: 26324 config.py:515 - DEBUG - Loading config PID: 26324 config.py:108 - DEBUG - Loading file: /home/jpk37/.starcluster/config PID: 26324 config.py:515 - DEBUG - Loading config PID: 26324 config.py:108 - DEBUG - Loading file: /home/jpk37/.starcluster/config PID: 26324 awsutils.py:54 - DEBUG - creating self._conn w/ connection_authenticator kwargs = {'path': '/', 'region': None, 'port': None, 'is_secure': True} PID: 26324 start.py:167 - INFO - Using default cluster template: smallcluster PID: 26324 cluster.py:1333 - INFO - Validating cluster template settings... PID: 26324 cluster.py:872 - DEBUG - Launch map: node001 (ami: ami-0af31963, type: m1.xlarge)... PID: 26324 cluster.py:872 - DEBUG - Launch map: node002 (ami: ami-0af31963, type: m1.xlarge)... PID: 26324 cluster.py:872 - DEBUG - Launch map: node003 (ami: ami-0af31963, type: m1.xlarge)... PID: 26324 cluster.py:872 - DEBUG - Launch map: node004 (ami: ami-0af31963, type: m1.xlarge)... PID: 26324 cluster.py:872 - DEBUG - Launch map: node005 (ami: ami-0af31963, type: m1.xlarge)... PID: 26324 cluster.py:872 - DEBUG - Launch map: node006 (ami: ami-0af31963, type: m1.xlarge)... PID: 26324 cluster.py:872 - DEBUG - Launch map: node007 (ami: ami-0af31963, type: m1.xlarge)... PID: 26324 cluster.py:872 - DEBUG - Launch map: node008 (ami: ami-0af31963, type: m1.xlarge)... PID: 26324 cluster.py:872 - DEBUG - Launch map: node009 (ami: ami-0af31963, type: m1.xlarge)... PID: 26324 cluster.py:1355 - INFO - Cluster template settings are valid PID: 26324 cluster.py:1228 - INFO - Starting cluster... PID: 26324 cluster.py:891 - INFO - Launching a 10-node cluster... PID: 26324 cluster.py:908 - INFO - Launching a 10-node cluster... PID: 26324 cluster.py:872 - DEBUG - Launch map: node001 (ami: ami-0af31963, type: m1.xlarge)... PID: 26324 cluster.py:872 - DEBUG - Launch map: node002 (ami: ami-0af31963, type: m1.xlarge)... PID: 26324 cluster.py:872 - DEBUG - Launch map: node003 (ami: ami-0af31963, type: m1.xlarge)... PID: 26324 cluster.py:872 - DEBUG - Launch map: node004 (ami: ami-0af31963, type: m1.xlarge)... PID: 26324 cluster.py:872 - DEBUG - Launch map: node005 (ami: ami-0af31963, type: m1.xlarge)... PID: 26324 cluster.py:872 - DEBUG - Launch map: node006 (ami: ami-0af31963, type: m1.xlarge)... PID: 26324 cluster.py:872 - DEBUG - Launch map: node007 (ami: ami-0af31963, type: m1.xlarge)... PID: 26324 cluster.py:872 - DEBUG - Launch map: node008 (ami: ami-0af31963, type: m1.xlarge)... PID: 26324 cluster.py:872 - DEBUG - Launch map: node009 (ami: ami-0af31963, type: m1.xlarge)... PID: 26324 cluster.py:919 - DEBUG - Launching master (ami: ami-0af31963, type: m1.xlarge) PID: 26324 cluster.py:919 - DEBUG - Launching node001 (ami: ami-0af31963, type: m1.xlarge) PID: 26324 cluster.py:919 - DEBUG - Launching node002 (ami: ami-0af31963, type: m1.xlarge) PID: 26324 cluster.py:919 - DEBUG - Launching node003 (ami: ami-0af31963, type: m1.xlarge) PID: 26324 cluster.py:919 - DEBUG - Launching node004 (ami: ami-0af31963, type: m1.xlarge) PID: 26324 cluster.py:919 - DEBUG - Launching node005 (ami: ami-0af31963, type: m1.xlarge) PID: 26324 cluster.py:919 - DEBUG - Launching node006 (ami: ami-0af31963, type: m1.xlarge) PID: 26324 cluster.py:919 - DEBUG - Launching node007 (ami: ami-0af31963, type: m1.xlarge) PID: 26324 cluster.py:919 - DEBUG - Launching node008 (ami: ami-0af31963, type: m1.xlarge) PID: 26324 cluster.py:919 - DEBUG - Launching node009 (ami: ami-0af31963, type: m1.xlarge) PID: 26324 awsutils.py:175 - INFO - Creating security group @sc-mycluster... PID: 26324 cluster.py:1048 - INFO - Waiting for cluster to come up... (updating every 30s) PID: 26324 cluster.py:670 - DEBUG - existing nodes: {} PID: 26324 cluster.py:678 - DEBUG - adding node i-ff8bef9e to self._nodes list PID: 26324 cluster.py:678 - DEBUG - adding node i-c18befa0 to self._nodes list PID: 26324 cluster.py:678 - DEBUG - adding node i-c38befa2 to self._nodes list PID: 26324 cluster.py:678 - DEBUG - adding node i-c58befa4 to self._nodes list PID: 26324 cluster.py:678 - DEBUG - adding node i-c78befa6 to self._nodes list PID: 26324 cluster.py:678 - DEBUG - adding node i-c98befa8 to self._nodes list PID: 26324 cluster.py:678 - DEBUG - adding node i-cb8befaa to self._nodes list PID: 26324 cluster.py:678 - DEBUG - adding node i-cd8befac to self._nodes list PID: 26324 cluster.py:678 - DEBUG - adding node i-cf8befae to self._nodes list PID: 26324 cluster.py:678 - DEBUG - adding node i-d18befb0 to self._nodes list PID: 26324 cluster.py:686 - DEBUG - returning self._nodes = [, , , , , , , , , ] PID: 26324 cluster.py:1070 - INFO - Waiting for all nodes to be in a 'running' state... PID: 26324 cluster.py:670 - DEBUG - existing nodes: {u'i-c38befa2': , u'i-cf8befae': , u'i-c78befa6': , u'i-c18befa0': , u'i-cb8befaa': , u'i-d18befb0': , u'i-cd8befac': , u'i-c58befa4': , u'i-ff8bef9e': , u'i-c98befa8': } PID: 26324 cluster.py:673 - DEBUG - updating existing node i-ff8bef9e in self._nodes PID: 26324 cluster.py:673 - DEBUG - updating existing node i-c18befa0 in self._nodes PID: 26324 cluster.py:673 - DEBUG - updating existing node i-c38befa2 in self._nodes PID: 26324 cluster.py:673 - DEBUG - updating existing node i-c58befa4 in self._nodes PID: 26324 cluster.py:673 - DEBUG - updating existing node i-c78befa6 in self._nodes PID: 26324 cluster.py:673 - DEBUG - updating existing node i-c98befa8 in self._nodes PID: 26324 cluster.py:673 - DEBUG - updating existing node i-cb8befaa in self._nodes PID: 26324 cluster.py:673 - DEBUG - updating existing node i-cd8befac in self._nodes PID: 26324 cluster.py:673 - DEBUG - updating existing node i-cf8befae in self._nodes PID: 26324 cluster.py:673 - DEBUG - updating existing node i-d18befb0 in self._nodes PID: 26324 cluster.py:686 - DEBUG - returning self._nodes = [, , , , , , , , , ] PID: 26324 cluster.py:670 - DEBUG - existing nodes: {u'i-c38befa2': , u'i-cf8befae': , u'i-c78befa6': , u'i-c18befa0': , u'i-cb8befaa': , u'i-d18befb0': , u'i-cd8befac': , u'i-c58befa4': , u'i-ff8bef9e': , u'i-c98befa8': } PID: 26324 cluster.py:673 - DEBUG - updating existing node i-ff8bef9e in self._nodes PID: 26324 cluster.py:673 - DEBUG - updating existing node i-c18befa0 in self._nodes PID: 26324 cluster.py:673 - DEBUG - updating existing node i-c38befa2 in self._nodes PID: 26324 cluster.py:673 - DEBUG - updating existing node i-c58befa4 in self._nodes PID: 26324 cluster.py:673 - DEBUG - updating existing node i-c78befa6 in self._nodes PID: 26324 cluster.py:673 - DEBUG - updating existing node i-c98befa8 in self._nodes PID: 26324 cluster.py:673 - DEBUG - updating existing node i-cb8befaa in self._nodes PID: 26324 cluster.py:673 - DEBUG - updating existing node i-cd8befac in self._nodes PID: 26324 cluster.py:673 - DEBUG - updating existing node i-cf8befae in self._nodes PID: 26324 cluster.py:673 - DEBUG - updating existing node i-d18befb0 in self._nodes PID: 26324 cluster.py:686 - DEBUG - returning self._nodes = [, , , , , , , , , ] PID: 26324 cluster.py:670 - DEBUG - existing nodes: {u'i-c38befa2': , u'i-cf8befae': , u'i-c78befa6': , u'i-c18befa0': , u'i-cb8befaa': , u'i-d18befb0': , u'i-cd8befac': , u'i-c58befa4': , u'i-ff8bef9e': , u'i-c98befa8': } PID: 26324 cluster.py:673 - DEBUG - updating existing node i-ff8bef9e in self._nodes PID: 26324 cluster.py:673 - DEBUG - updating existing node i-c18befa0 in self._nodes PID: 26324 cluster.py:673 - DEBUG - updating existing node i-c38befa2 in self._nodes PID: 26324 cluster.py:673 - DEBUG - updating existing node i-c58befa4 in self._nodes PID: 26324 cluster.py:673 - DEBUG - updating existing node i-c78befa6 in self._nodes PID: 26324 cluster.py:673 - DEBUG - updating existing node i-c98befa8 in self._nodes PID: 26324 cluster.py:673 - DEBUG - updating existing node i-cb8befaa in self._nodes PID: 26324 cluster.py:673 - DEBUG - updating existing node i-cd8befac in self._nodes PID: 26324 cluster.py:673 - DEBUG - updating existing node i-cf8befae in self._nodes PID: 26324 cluster.py:673 - DEBUG - updating existing node i-d18befb0 in self._nodes PID: 26324 cluster.py:686 - DEBUG - returning self._nodes = [, , , , , , , , , ] PID: 26324 cluster.py:670 - DEBUG - existing nodes: {u'i-c38befa2': , u'i-cf8befae': , u'i-c78befa6': , u'i-c18befa0': , u'i-cb8befaa': , u'i-d18befb0': , u'i-cd8befac': , u'i-c58befa4': , u'i-ff8bef9e': , u'i-c98befa8': } PID: 26324 cluster.py:673 - DEBUG - updating existing node i-ff8bef9e in self._nodes PID: 26324 cluster.py:673 - DEBUG - updating existing node i-c18befa0 in self._nodes PID: 26324 cluster.py:673 - DEBUG - updating existing node i-c38befa2 in self._nodes PID: 26324 cluster.py:673 - DEBUG - updating existing node i-c58befa4 in self._nodes PID: 26324 cluster.py:673 - DEBUG - updating existing node i-c78befa6 in self._nodes PID: 26324 cluster.py:673 - DEBUG - updating existing node i-c98befa8 in self._nodes PID: 26324 cluster.py:673 - DEBUG - updating existing node i-cb8befaa in self._nodes PID: 26324 cluster.py:673 - DEBUG - updating existing node i-cd8befac in self._nodes PID: 26324 cluster.py:673 - DEBUG - updating existing node i-cf8befae in self._nodes PID: 26324 cluster.py:673 - DEBUG - updating existing node i-d18befb0 in self._nodes PID: 26324 cluster.py:686 - DEBUG - returning self._nodes = [, , , , , , , , , ] PID: 26324 cluster.py:670 - DEBUG - existing nodes: {u'i-c38befa2': , u'i-cf8befae': , u'i-c78befa6': , u'i-c18befa0': , u'i-cb8befaa': , u'i-d18befb0': , u'i-cd8befac': , u'i-c58befa4': , u'i-ff8bef9e': , u'i-c98befa8': } PID: 26324 cluster.py:673 - DEBUG - updating existing node i-ff8bef9e in self._nodes PID: 26324 cluster.py:673 - DEBUG - updating existing node i-c18befa0 in self._nodes PID: 26324 cluster.py:673 - DEBUG - updating existing node i-c38befa2 in self._nodes PID: 26324 cluster.py:673 - DEBUG - updating existing node i-c58befa4 in self._nodes PID: 26324 cluster.py:673 - DEBUG - updating existing node i-c78befa6 in self._nodes PID: 26324 cluster.py:673 - DEBUG - updating existing node i-c98befa8 in self._nodes PID: 26324 cluster.py:673 - DEBUG - updating existing node i-cb8befaa in self._nodes PID: 26324 cluster.py:673 - DEBUG - updating existing node i-cd8befac in self._nodes PID: 26324 cluster.py:673 - DEBUG - updating existing node i-cf8befae in self._nodes PID: 26324 cluster.py:673 - DEBUG - updating existing node i-d18befb0 in self._nodes PID: 26324 cluster.py:686 - DEBUG - returning self._nodes = [, , , , , , , , , ] PID: 26324 cluster.py:670 - DEBUG - existing nodes: {u'i-c38befa2': , u'i-cf8befae': , u'i-c78befa6': , u'i-c18befa0': , u'i-cb8befaa': , u'i-d18befb0': , u'i-cd8befac': , u'i-c58befa4': , u'i-ff8bef9e': , u'i-c98befa8': } PID: 26324 cluster.py:673 - DEBUG - updating existing node i-ff8bef9e in self._nodes PID: 26324 cluster.py:673 - DEBUG - updating existing node i-c18befa0 in self._nodes PID: 26324 cluster.py:673 - DEBUG - updating existing node i-c38befa2 in self._nodes PID: 26324 cluster.py:673 - DEBUG - updating existing node i-c58befa4 in self._nodes PID: 26324 cluster.py:673 - DEBUG - updating existing node i-c78befa6 in self._nodes PID: 26324 cluster.py:673 - DEBUG - updating existing node i-c98befa8 in self._nodes PID: 26324 cluster.py:673 - DEBUG - updating existing node i-cb8befaa in self._nodes PID: 26324 cluster.py:673 - DEBUG - updating existing node i-cd8befac in self._nodes PID: 26324 cluster.py:673 - DEBUG - updating existing node i-cf8befae in self._nodes PID: 26324 cluster.py:673 - DEBUG - updating existing node i-d18befb0 in self._nodes PID: 26324 cluster.py:686 - DEBUG - returning self._nodes = [, , , , , , , , , ] PID: 26324 cluster.py:1081 - INFO - Waiting for SSH to come up on all nodes... PID: 26324 ssh.py:61 - DEBUG - loading private key /home/jpk37/Downloads/blastkey.pem PID: 26324 ssh.py:68 - DEBUG - specified key does not end in either rsa or dsa, trying both PID: 26324 ssh.py:145 - DEBUG - Using private key /home/jpk37/Downloads/blastkey.pem (rsa) PID: 26324 ssh.py:82 - DEBUG - connecting to host ec2-50-19-70-119.compute-1.amazonaws.com on port 22 as user root PID: 26324 ssh.py:61 - DEBUG - loading private key /home/jpk37/Downloads/blastkey.pem PID: 26324 ssh.py:68 - DEBUG - specified key does not end in either rsa or dsa, trying both PID: 26324 ssh.py:145 - DEBUG - Using private key /home/jpk37/Downloads/blastkey.pem (rsa) PID: 26324 ssh.py:82 - DEBUG - connecting to host ec2-50-19-133-45.compute-1.amazonaws.com on port 22 as user root PID: 26324 ssh.py:61 - DEBUG - loading private key /home/jpk37/Downloads/blastkey.pem PID: 26324 ssh.py:68 - DEBUG - specified key does not end in either rsa or dsa, trying both PID: 26324 ssh.py:145 - DEBUG - Using private key /home/jpk37/Downloads/blastkey.pem (rsa) PID: 26324 ssh.py:82 - DEBUG - connecting to host ec2-50-19-43-255.compute-1.amazonaws.com on port 22 as user root PID: 26324 ssh.py:61 - DEBUG - loading private key /home/jpk37/Downloads/blastkey.pem PID: 26324 ssh.py:68 - DEBUG - specified key does not end in either rsa or dsa, trying both PID: 26324 ssh.py:145 - DEBUG - Using private key /home/jpk37/Downloads/blastkey.pem (rsa) PID: 26324 ssh.py:82 - DEBUG - connecting to host ec2-107-20-48-152.compute-1.amazonaws.com on port 22 as user root PID: 26324 ssh.py:61 - DEBUG - loading private key /home/jpk37/Downloads/blastkey.pem PID: 26324 ssh.py:68 - DEBUG - specified key does not end in either rsa or dsa, trying both PID: 26324 ssh.py:145 - DEBUG - Using private key /home/jpk37/Downloads/blastkey.pem (rsa) PID: 26324 ssh.py:82 - DEBUG - connecting to host ec2-50-19-150-73.compute-1.amazonaws.com on port 22 as user root PID: 26324 ssh.py:61 - DEBUG - loading private key /home/jpk37/Downloads/blastkey.pem PID: 26324 ssh.py:68 - DEBUG - specified key does not end in either rsa or dsa, trying both PID: 26324 ssh.py:145 - DEBUG - Using private key /home/jpk37/Downloads/blastkey.pem (rsa) PID: 26324 ssh.py:82 - DEBUG - connecting to host ec2-50-17-154-227.compute-1.amazonaws.com on port 22 as user root PID: 26324 ssh.py:61 - DEBUG - loading private key /home/jpk37/Downloads/blastkey.pem PID: 26324 ssh.py:68 - DEBUG - specified key does not end in either rsa or dsa, trying both PID: 26324 ssh.py:145 - DEBUG - Using private key /home/jpk37/Downloads/blastkey.pem (rsa) PID: 26324 ssh.py:82 - DEBUG - connecting to host ec2-184-72-80-38.compute-1.amazonaws.com on port 22 as user root PID: 26324 ssh.py:61 - DEBUG - loading private key /home/jpk37/Downloads/blastkey.pem PID: 26324 ssh.py:68 - DEBUG - specified key does not end in either rsa or dsa, trying both PID: 26324 ssh.py:145 - DEBUG - Using private key /home/jpk37/Downloads/blastkey.pem (rsa) PID: 26324 ssh.py:82 - DEBUG - connecting to host ec2-174-129-97-72.compute-1.amazonaws.com on port 22 as user root PID: 26324 ssh.py:61 - DEBUG - loading private key /home/jpk37/Downloads/blastkey.pem PID: 26324 ssh.py:68 - DEBUG - specified key does not end in either rsa or dsa, trying both PID: 26324 ssh.py:145 - DEBUG - Using private key /home/jpk37/Downloads/blastkey.pem (rsa) PID: 26324 ssh.py:82 - DEBUG - connecting to host ec2-184-73-18-69.compute-1.amazonaws.com on port 22 as user root PID: 26324 ssh.py:61 - DEBUG - loading private key /home/jpk37/Downloads/blastkey.pem PID: 26324 ssh.py:68 - DEBUG - specified key does not end in either rsa or dsa, trying both PID: 26324 ssh.py:145 - DEBUG - Using private key /home/jpk37/Downloads/blastkey.pem (rsa) PID: 26324 ssh.py:82 - DEBUG - connecting to host ec2-50-17-175-68.compute-1.amazonaws.com on port 22 as user root PID: 26324 cluster.py:1252 - INFO - The master node is ec2-50-19-70-119.compute-1.amazonaws.com PID: 26324 cluster.py:1253 - INFO - Setting up the cluster... PID: 26324 cluster.py:1127 - INFO - Attaching volume vol-b2ff16d8 to master node on /dev/sdz ... PID: 26324 cluster.py:1129 - DEBUG - resp = attaching PID: 26324 cluster.py:670 - DEBUG - existing nodes: {u'i-c38befa2': , u'i-cf8befae': , u'i-c78befa6': , u'i-c18befa0': , u'i-cb8befaa': , u'i-d18befb0': , u'i-cd8befac': , u'i-c58befa4': , u'i-ff8bef9e': , u'i-c98befa8': } PID: 26324 cluster.py:673 - DEBUG - updating existing node i-ff8bef9e in self._nodes PID: 26324 cluster.py:673 - DEBUG - updating existing node i-c18befa0 in self._nodes PID: 26324 cluster.py:673 - DEBUG - updating existing node i-c38befa2 in self._nodes PID: 26324 cluster.py:673 - DEBUG - updating existing node i-c58befa4 in self._nodes PID: 26324 cluster.py:673 - DEBUG - updating existing node i-c78befa6 in self._nodes PID: 26324 cluster.py:673 - DEBUG - updating existing node i-c98befa8 in self._nodes PID: 26324 cluster.py:673 - DEBUG - updating existing node i-cb8befaa in self._nodes PID: 26324 cluster.py:673 - DEBUG - updating existing node i-cd8befac in self._nodes PID: 26324 cluster.py:673 - DEBUG - updating existing node i-cf8befae in self._nodes PID: 26324 cluster.py:673 - DEBUG - updating existing node i-d18befb0 in self._nodes PID: 26324 cluster.py:686 - DEBUG - returning self._nodes = [, , , , , , , , , ] PID: 26324 clustersetup.py:96 - INFO - Configuring hostnames... PID: 26324 ssh.py:164 - DEBUG - creating sftp connection PID: 26324 ssh.py:164 - DEBUG - creating sftp connection PID: 26324 ssh.py:164 - DEBUG - creating sftp connection PID: 26324 ssh.py:164 - DEBUG - creating sftp connection PID: 26324 ssh.py:164 - DEBUG - creating sftp connection PID: 26324 ssh.py:164 - DEBUG - creating sftp connection PID: 26324 ssh.py:164 - DEBUG - creating sftp connection PID: 26324 ssh.py:164 - DEBUG - creating sftp connection PID: 26324 threadpool.py:108 - DEBUG - unfinished_tasks = 10 PID: 26324 ssh.py:164 - DEBUG - creating sftp connection PID: 26324 ssh.py:164 - DEBUG - creating sftp connection PID: 26324 clustersetup.py:262 - INFO - Mounting EBS volume vol-b2ff16d8 on /db... PID: 26324 ssh.py:405 - DEBUG - /dev/sda1 on / type ext3 (rw) PID: 26324 ssh.py:405 - DEBUG - proc on /proc type proc (rw,noexec,nosuid,nodev) PID: 26324 ssh.py:405 - DEBUG - none on /sys type sysfs (rw,noexec,nosuid,nodev) PID: 26324 ssh.py:405 - DEBUG - none on /sys/kernel/debug type debugfs (rw) PID: 26324 ssh.py:405 - DEBUG - none on /sys/kernel/security type securityfs (rw) PID: 26324 ssh.py:405 - DEBUG - devtmpfs on /dev type devtmpfs (rw,mode=0755) PID: 26324 ssh.py:405 - DEBUG - none on /dev/pts type devpts (rw,noexec,nosuid,gid=5,mode=0620) PID: 26324 ssh.py:405 - DEBUG - none on /dev/shm type tmpfs (rw,nosuid,nodev) PID: 26324 ssh.py:405 - DEBUG - none on /var/run type tmpfs (rw,nosuid,mode=0755) PID: 26324 ssh.py:405 - DEBUG - none on /var/lock type tmpfs (rw,noexec,nosuid,nodev) PID: 26324 ssh.py:405 - DEBUG - none on /lib/init/rw type tmpfs (rw,nosuid,mode=0755) PID: 26324 ssh.py:405 - DEBUG - /dev/sdb on /mnt type ext3 (rw) PID: 26324 ssh.py:405 - DEBUG - nfsd on /proc/fs/nfsd type nfsd (rw) PID: 26324 ssh.py:254 - DEBUG - new /etc/fstab after removing regex ( /db ) matches: # /etc/fstab: static file system information. # proc /proc proc nodev,noexec,nosuid 0 0 /dev/sda1 / ext3 defaults 0 0 /dev/sdb /mnt auto defaults,comment=cloudconfig 0 0 PID: 26324 clustersetup.py:144 - INFO - Creating cluster user: sgeadmin (uid: 1001, gid: 1001) PID: 26324 threadpool.py:108 - DEBUG - unfinished_tasks = 10 PID: 26324 clustersetup.py:161 - DEBUG - user sgeadmin does not exist, creating... PID: 26324 clustersetup.py:161 - DEBUG - user sgeadmin does not exist, creating... PID: 26324 clustersetup.py:161 - DEBUG - user sgeadmin does not exist, creating... PID: 26324 clustersetup.py:161 - DEBUG - user sgeadmin does not exist, creating... PID: 26324 clustersetup.py:161 - DEBUG - user sgeadmin does not exist, creating... PID: 26324 clustersetup.py:161 - DEBUG - user sgeadmin does not exist, creating... PID: 26324 clustersetup.py:161 - DEBUG - user sgeadmin does not exist, creating... PID: 26324 clustersetup.py:161 - DEBUG - user sgeadmin does not exist, creating... PID: 26324 clustersetup.py:161 - DEBUG - user sgeadmin does not exist, creating... PID: 26324 clustersetup.py:161 - DEBUG - user sgeadmin does not exist, creating... PID: 26324 clustersetup.py:186 - INFO - Configuring scratch space for user: sgeadmin PID: 26324 threadpool.py:108 - DEBUG - unfinished_tasks = 10 PID: 26324 clustersetup.py:195 - INFO - Configuring /etc/hosts on each node PID: 26324 threadpool.py:108 - DEBUG - unfinished_tasks = 10 PID: 26324 ssh.py:254 - DEBUG - new /etc/hosts after removing regex (master|node001|node002|node003|node004|node005|node006|node007|node008|node009) matches: 127.0.0.1 localhost # The following lines are desirable for IPv6 capable hosts ::1 ip6-localhost ip6-loopback fe00::0 ip6-localnet ff00::0 ip6-mcastprefix ff02::1 ip6-allnodes ff02::2 ip6-allrouters ff02::3 ip6-allhosts PID: 26324 ssh.py:254 - DEBUG - new /etc/hosts after removing regex (master|node001|node002|node003|node004|node005|node006|node007|node008|node009) matches: 127.0.0.1 localhost # The following lines are desirable for IPv6 capable hosts ::1 ip6-localhost ip6-loopback fe00::0 ip6-localnet ff00::0 ip6-mcastprefix ff02::1 ip6-allnodes ff02::2 ip6-allrouters ff02::3 ip6-allhosts PID: 26324 ssh.py:254 - DEBUG - new /etc/hosts after removing regex (master|node001|node002|node003|node004|node005|node006|node007|node008|node009) matches: 127.0.0.1 localhost # The following lines are desirable for IPv6 capable hosts ::1 ip6-localhost ip6-loopback fe00::0 ip6-localnet ff00::0 ip6-mcastprefix ff02::1 ip6-allnodes ff02::2 ip6-allrouters ff02::3 ip6-allhosts PID: 26324 ssh.py:254 - DEBUG - new /etc/hosts after removing regex (master|node001|node002|node003|node004|node005|node006|node007|node008|node009) matches: 127.0.0.1 localhost # The following lines are desirable for IPv6 capable hosts ::1 ip6-localhost ip6-loopback fe00::0 ip6-localnet ff00::0 ip6-mcastprefix ff02::1 ip6-allnodes ff02::2 ip6-allrouters ff02::3 ip6-allhosts PID: 26324 ssh.py:254 - DEBUG - new /etc/hosts after removing regex (master|node001|node002|node003|node004|node005|node006|node007|node008|node009) matches: 127.0.0.1 localhost # The following lines are desirable for IPv6 capable hosts ::1 ip6-localhost ip6-loopback fe00::0 ip6-localnet ff00::0 ip6-mcastprefix ff02::1 ip6-allnodes ff02::2 ip6-allrouters ff02::3 ip6-allhosts PID: 26324 ssh.py:254 - DEBUG - new /etc/hosts after removing regex (master|node001|node002|node003|node004|node005|node006|node007|node008|node009) matches: 127.0.0.1 localhost # The following lines are desirable for IPv6 capable hosts ::1 ip6-localhost ip6-loopback fe00::0 ip6-localnet ff00::0 ip6-mcastprefix ff02::1 ip6-allnodes ff02::2 ip6-allrouters ff02::3 ip6-allhosts PID: 26324 ssh.py:254 - DEBUG - new /etc/hosts after removing regex (master|node001|node002|node003|node004|node005|node006|node007|node008|node009) matches: 127.0.0.1 localhost # The following lines are desirable for IPv6 capable hosts ::1 ip6-localhost ip6-loopback fe00::0 ip6-localnet ff00::0 ip6-mcastprefix ff02::1 ip6-allnodes ff02::2 ip6-allrouters ff02::3 ip6-allhosts PID: 26324 ssh.py:254 - DEBUG - new /etc/hosts after removing regex (master|node001|node002|node003|node004|node005|node006|node007|node008|node009) matches: 127.0.0.1 localhost # The following lines are desirable for IPv6 capable hosts ::1 ip6-localhost ip6-loopback fe00::0 ip6-localnet ff00::0 ip6-mcastprefix ff02::1 ip6-allnodes ff02::2 ip6-allrouters ff02::3 ip6-allhosts PID: 26324 ssh.py:254 - DEBUG - new /etc/hosts after removing regex (master|node001|node002|node003|node004|node005|node006|node007|node008|node009) matches: 127.0.0.1 localhost # The following lines are desirable for IPv6 capable hosts ::1 ip6-localhost ip6-loopback fe00::0 ip6-localnet ff00::0 ip6-mcastprefix ff02::1 ip6-allnodes ff02::2 ip6-allrouters ff02::3 ip6-allhosts PID: 26324 ssh.py:254 - DEBUG - new /etc/hosts after removing regex (master|node001|node002|node003|node004|node005|node006|node007|node008|node009) matches: 127.0.0.1 localhost # The following lines are desirable for IPv6 capable hosts ::1 ip6-localhost ip6-loopback fe00::0 ip6-localnet ff00::0 ip6-mcastprefix ff02::1 ip6-allnodes ff02::2 ip6-allrouters ff02::3 ip6-allhosts PID: 26324 clustersetup.py:314 - INFO - Configuring NFS... PID: 26324 ssh.py:405 - DEBUG - Rather than invoking init scripts through /etc/init.d, use the service(8) PID: 26324 ssh.py:405 - DEBUG - utility, e.g. service portmap start PID: 26324 ssh.py:405 - DEBUG - PID: 26324 ssh.py:405 - DEBUG - Since the script you are attempting to invoke has been converted to an PID: 26324 ssh.py:405 - DEBUG - Upstart job, you may also use the start(8) utility, e.g. start portmap PID: 26324 ssh.py:405 - DEBUG - * Exporting directories for NFS kernel daemon... PID: 26324 ssh.py:405 - DEBUG - ...done. PID: 26324 ssh.py:405 - DEBUG - * Starting NFS kernel daemon PID: 26324 ssh.py:405 - DEBUG - ...done. PID: 26324 ssh.py:405 - DEBUG - /dev/sda1 on / type ext3 (rw) PID: 26324 ssh.py:405 - DEBUG - proc on /proc type proc (rw,noexec,nosuid,nodev) PID: 26324 ssh.py:405 - DEBUG - none on /sys type sysfs (rw,noexec,nosuid,nodev) PID: 26324 ssh.py:405 - DEBUG - none on /sys/kernel/debug type debugfs (rw) PID: 26324 ssh.py:405 - DEBUG - none on /sys/kernel/security type securityfs (rw) PID: 26324 ssh.py:405 - DEBUG - devtmpfs on /dev type devtmpfs (rw,mode=0755) PID: 26324 ssh.py:405 - DEBUG - none on /dev/pts type devpts (rw,noexec,nosuid,gid=5,mode=0620) PID: 26324 ssh.py:405 - DEBUG - none on /dev/shm type tmpfs (rw,nosuid,nodev) PID: 26324 ssh.py:405 - DEBUG - none on /var/run type tmpfs (rw,nosuid,mode=0755) PID: 26324 ssh.py:405 - DEBUG - none on /var/lock type tmpfs (rw,noexec,nosuid,nodev) PID: 26324 ssh.py:405 - DEBUG - none on /lib/init/rw type tmpfs (rw,nosuid,mode=0755) PID: 26324 ssh.py:405 - DEBUG - /dev/sdb on /mnt type ext3 (rw) PID: 26324 ssh.py:405 - DEBUG - nfsd on /proc/fs/nfsd type nfsd (rw) PID: 26324 clustersetup.py:304 - INFO - Mounting shares for node node001 PID: 26324 ssh.py:405 - DEBUG - /dev/sda1 on / type ext3 (rw) PID: 26324 ssh.py:405 - DEBUG - proc on /proc type proc (rw,noexec,nosuid,nodev) PID: 26324 ssh.py:405 - DEBUG - none on /sys type sysfs (rw,noexec,nosuid,nodev) PID: 26324 ssh.py:405 - DEBUG - none on /sys/kernel/debug type debugfs (rw) PID: 26324 ssh.py:405 - DEBUG - none on /sys/kernel/security type securityfs (rw) PID: 26324 ssh.py:405 - DEBUG - devtmpfs on /dev type devtmpfs (rw,mode=0755) PID: 26324 ssh.py:405 - DEBUG - none on /dev/pts type devpts (rw,noexec,nosuid,gid=5,mode=0620) PID: 26324 ssh.py:405 - DEBUG - none on /dev/shm type tmpfs (rw,nosuid,nodev) PID: 26324 ssh.py:405 - DEBUG - none on /var/run type tmpfs (rw,nosuid,mode=0755) PID: 26324 ssh.py:405 - DEBUG - none on /var/lock type tmpfs (rw,noexec,nosuid,nodev) PID: 26324 ssh.py:405 - DEBUG - none on /lib/init/rw type tmpfs (rw,nosuid,mode=0755) PID: 26324 ssh.py:405 - DEBUG - /dev/sdb on /mnt type ext3 (rw) PID: 26324 ssh.py:405 - DEBUG - nfsd on /proc/fs/nfsd type nfsd (rw) PID: 26324 clustersetup.py:304 - INFO - Mounting shares for node node002 PID: 26324 ssh.py:405 - DEBUG - /dev/sda1 on / type ext3 (rw) PID: 26324 ssh.py:405 - DEBUG - proc on /proc type proc (rw,noexec,nosuid,nodev) PID: 26324 ssh.py:405 - DEBUG - none on /sys type sysfs (rw,noexec,nosuid,nodev) PID: 26324 ssh.py:405 - DEBUG - none on /sys/kernel/debug type debugfs (rw) PID: 26324 ssh.py:405 - DEBUG - none on /sys/kernel/security type securityfs (rw) PID: 26324 ssh.py:405 - DEBUG - devtmpfs on /dev type devtmpfs (rw,mode=0755) PID: 26324 ssh.py:405 - DEBUG - none on /dev/pts type devpts (rw,noexec,nosuid,gid=5,mode=0620) PID: 26324 ssh.py:405 - DEBUG - none on /dev/shm type tmpfs (rw,nosuid,nodev) PID: 26324 ssh.py:405 - DEBUG - none on /var/run type tmpfs (rw,nosuid,mode=0755) PID: 26324 ssh.py:405 - DEBUG - none on /var/lock type tmpfs (rw,noexec,nosuid,nodev) PID: 26324 ssh.py:405 - DEBUG - none on /lib/init/rw type tmpfs (rw,nosuid,mode=0755) PID: 26324 ssh.py:405 - DEBUG - /dev/sdb on /mnt type ext3 (rw) PID: 26324 ssh.py:405 - DEBUG - nfsd on /proc/fs/nfsd type nfsd (rw) PID: 26324 clustersetup.py:304 - INFO - Mounting shares for node node003 PID: 26324 ssh.py:405 - DEBUG - /dev/sda1 on / type ext3 (rw) PID: 26324 ssh.py:405 - DEBUG - proc on /proc type proc (rw,noexec,nosuid,nodev) PID: 26324 ssh.py:405 - DEBUG - none on /sys type sysfs (rw,noexec,nosuid,nodev) PID: 26324 ssh.py:405 - DEBUG - none on /sys/kernel/debug type debugfs (rw) PID: 26324 ssh.py:405 - DEBUG - none on /sys/kernel/security type securityfs (rw) PID: 26324 ssh.py:405 - DEBUG - devtmpfs on /dev type devtmpfs (rw,mode=0755) PID: 26324 ssh.py:405 - DEBUG - none on /dev/pts type devpts (rw,noexec,nosuid,gid=5,mode=0620) PID: 26324 ssh.py:405 - DEBUG - none on /dev/shm type tmpfs (rw,nosuid,nodev) PID: 26324 ssh.py:405 - DEBUG - none on /var/run type tmpfs (rw,nosuid,mode=0755) PID: 26324 ssh.py:405 - DEBUG - none on /var/lock type tmpfs (rw,noexec,nosuid,nodev) PID: 26324 ssh.py:405 - DEBUG - none on /lib/init/rw type tmpfs (rw,nosuid,mode=0755) PID: 26324 ssh.py:405 - DEBUG - /dev/sdb on /mnt type ext3 (rw) PID: 26324 ssh.py:405 - DEBUG - nfsd on /proc/fs/nfsd type nfsd (rw) PID: 26324 clustersetup.py:304 - INFO - Mounting shares for node node004 PID: 26324 ssh.py:405 - DEBUG - Rather than invoking init scripts through /etc/init.d, use the service(8) PID: 26324 ssh.py:405 - DEBUG - utility, e.g. service portmap start PID: 26324 ssh.py:405 - DEBUG - PID: 26324 ssh.py:405 - DEBUG - Since the script you are attempting to invoke has been converted to an PID: 26324 ssh.py:405 - DEBUG - Upstart job, you may also use the start(8) utility, e.g. start portmap PID: 26324 ssh.py:405 - DEBUG - /dev/sda1 on / type ext3 (rw) PID: 26324 ssh.py:405 - DEBUG - proc on /proc type proc (rw,noexec,nosuid,nodev) PID: 26324 ssh.py:405 - DEBUG - none on /sys type sysfs (rw,noexec,nosuid,nodev) PID: 26324 ssh.py:405 - DEBUG - none on /sys/kernel/debug type debugfs (rw) PID: 26324 ssh.py:405 - DEBUG - none on /sys/kernel/security type securityfs (rw) PID: 26324 ssh.py:405 - DEBUG - devtmpfs on /dev type devtmpfs (rw,mode=0755) PID: 26324 ssh.py:405 - DEBUG - none on /dev/pts type devpts (rw,noexec,nosuid,gid=5,mode=0620) PID: 26324 ssh.py:405 - DEBUG - none on /dev/shm type tmpfs (rw,nosuid,nodev) PID: 26324 ssh.py:405 - DEBUG - none on /var/run type tmpfs (rw,nosuid,mode=0755) PID: 26324 ssh.py:405 - DEBUG - none on /var/lock type tmpfs (rw,noexec,nosuid,nodev) PID: 26324 ssh.py:405 - DEBUG - none on /lib/init/rw type tmpfs (rw,nosuid,mode=0755) PID: 26324 ssh.py:405 - DEBUG - /dev/sdb on /mnt type ext3 (rw) PID: 26324 ssh.py:405 - DEBUG - nfsd on /proc/fs/nfsd type nfsd (rw) PID: 26324 clustersetup.py:304 - INFO - Mounting shares for node node005 PID: 26324 ssh.py:405 - DEBUG - Rather than invoking init scripts through /etc/init.d, use the service(8) PID: 26324 ssh.py:405 - DEBUG - utility, e.g. service portmap start PID: 26324 ssh.py:405 - DEBUG - PID: 26324 ssh.py:405 - DEBUG - Since the script you are attempting to invoke has been converted to an PID: 26324 ssh.py:405 - DEBUG - Upstart job, you may also use the start(8) utility, e.g. start portmap PID: 26324 ssh.py:405 - DEBUG - /dev/sda1 on / type ext3 (rw) PID: 26324 ssh.py:405 - DEBUG - proc on /proc type proc (rw,noexec,nosuid,nodev) PID: 26324 ssh.py:405 - DEBUG - none on /sys type sysfs (rw,noexec,nosuid,nodev) PID: 26324 ssh.py:405 - DEBUG - none on /sys/kernel/debug type debugfs (rw) PID: 26324 ssh.py:405 - DEBUG - none on /sys/kernel/security type securityfs (rw) PID: 26324 ssh.py:405 - DEBUG - devtmpfs on /dev type devtmpfs (rw,mode=0755) PID: 26324 ssh.py:405 - DEBUG - none on /dev/pts type devpts (rw,noexec,nosuid,gid=5,mode=0620) PID: 26324 ssh.py:405 - DEBUG - none on /dev/shm type tmpfs (rw,nosuid,nodev) PID: 26324 ssh.py:405 - DEBUG - none on /var/run type tmpfs (rw,nosuid,mode=0755) PID: 26324 ssh.py:405 - DEBUG - none on /var/lock type tmpfs (rw,noexec,nosuid,nodev) PID: 26324 ssh.py:405 - DEBUG - none on /lib/init/rw type tmpfs (rw,nosuid,mode=0755) PID: 26324 ssh.py:405 - DEBUG - /dev/sdb on /mnt type ext3 (rw) PID: 26324 ssh.py:405 - DEBUG - nfsd on /proc/fs/nfsd type nfsd (rw) PID: 26324 clustersetup.py:304 - INFO - Mounting shares for node node006 PID: 26324 ssh.py:405 - DEBUG - Rather than invoking init scripts through /etc/init.d, use the service(8) PID: 26324 ssh.py:405 - DEBUG - utility, e.g. service portmap start PID: 26324 ssh.py:405 - DEBUG - PID: 26324 ssh.py:405 - DEBUG - Since the script you are attempting to invoke has been converted to an PID: 26324 ssh.py:405 - DEBUG - Upstart job, you may also use the start(8) utility, e.g. start portmap PID: 26324 ssh.py:402 - DEBUG - command mount -t devpts none /dev/pts failed with status 32 PID: 26324 ssh.py:405 - DEBUG - mount: none already mounted or /dev/pts busy PID: 26324 ssh.py:405 - DEBUG - mount: according to mtab, none is already mounted on /dev/pts PID: 26324 ssh.py:405 - DEBUG - /dev/sda1 on / type ext3 (rw) PID: 26324 ssh.py:405 - DEBUG - proc on /proc type proc (rw,noexec,nosuid,nodev) PID: 26324 ssh.py:405 - DEBUG - none on /sys type sysfs (rw,noexec,nosuid,nodev) PID: 26324 ssh.py:405 - DEBUG - none on /sys/kernel/debug type debugfs (rw) PID: 26324 ssh.py:405 - DEBUG - none on /sys/kernel/security type securityfs (rw) PID: 26324 ssh.py:405 - DEBUG - devtmpfs on /dev type devtmpfs (rw,mode=0755) PID: 26324 ssh.py:405 - DEBUG - none on /dev/pts type devpts (rw,noexec,nosuid,gid=5,mode=0620) PID: 26324 ssh.py:405 - DEBUG - none on /dev/shm type tmpfs (rw,nosuid,nodev) PID: 26324 ssh.py:405 - DEBUG - none on /var/run type tmpfs (rw,nosuid,mode=0755) PID: 26324 ssh.py:405 - DEBUG - none on /var/lock type tmpfs (rw,noexec,nosuid,nodev) PID: 26324 ssh.py:405 - DEBUG - none on /lib/init/rw type tmpfs (rw,nosuid,mode=0755) PID: 26324 ssh.py:405 - DEBUG - /dev/sdb on /mnt type ext3 (rw) PID: 26324 ssh.py:405 - DEBUG - nfsd on /proc/fs/nfsd type nfsd (rw) PID: 26324 clustersetup.py:304 - INFO - Mounting shares for node node007 PID: 26324 ssh.py:402 - DEBUG - command mount -t devpts none /dev/pts failed with status 32 PID: 26324 ssh.py:405 - DEBUG - mount: none already mounted or /dev/pts busy PID: 26324 ssh.py:405 - DEBUG - mount: according to mtab, none is already mounted on /dev/pts PID: 26324 ssh.py:405 - DEBUG - Rather than invoking init scripts through /etc/init.d, use the service(8) PID: 26324 ssh.py:405 - DEBUG - utility, e.g. service portmap start PID: 26324 ssh.py:405 - DEBUG - PID: 26324 ssh.py:405 - DEBUG - Since the script you are attempting to invoke has been converted to an PID: 26324 ssh.py:405 - DEBUG - Upstart job, you may also use the start(8) utility, e.g. start portmap PID: 26324 ssh.py:405 - DEBUG - /dev/sda1 on / type ext3 (rw) PID: 26324 ssh.py:405 - DEBUG - proc on /proc type proc (rw,noexec,nosuid,nodev) PID: 26324 ssh.py:405 - DEBUG - none on /sys type sysfs (rw,noexec,nosuid,nodev) PID: 26324 ssh.py:405 - DEBUG - none on /sys/kernel/debug type debugfs (rw) PID: 26324 ssh.py:405 - DEBUG - none on /sys/kernel/security type securityfs (rw) PID: 26324 ssh.py:405 - DEBUG - devtmpfs on /dev type devtmpfs (rw,mode=0755) PID: 26324 ssh.py:405 - DEBUG - none on /dev/pts type devpts (rw,noexec,nosuid,gid=5,mode=0620) PID: 26324 ssh.py:405 - DEBUG - none on /dev/shm type tmpfs (rw,nosuid,nodev) PID: 26324 ssh.py:405 - DEBUG - none on /var/run type tmpfs (rw,nosuid,mode=0755) PID: 26324 ssh.py:405 - DEBUG - none on /var/lock type tmpfs (rw,noexec,nosuid,nodev) PID: 26324 ssh.py:405 - DEBUG - none on /lib/init/rw type tmpfs (rw,nosuid,mode=0755) PID: 26324 ssh.py:405 - DEBUG - /dev/sdb on /mnt type ext3 (rw) PID: 26324 ssh.py:405 - DEBUG - nfsd on /proc/fs/nfsd type nfsd (rw) PID: 26324 clustersetup.py:304 - INFO - Mounting shares for node node008 PID: 26324 ssh.py:402 - DEBUG - command mount -t devpts none /dev/pts failed with status 32 PID: 26324 ssh.py:405 - DEBUG - mount: none already mounted or /dev/pts busy PID: 26324 ssh.py:405 - DEBUG - mount: according to mtab, none is already mounted on /dev/pts PID: 26324 ssh.py:405 - DEBUG - Rather than invoking init scripts through /etc/init.d, use the service(8) PID: 26324 ssh.py:405 - DEBUG - utility, e.g. service portmap start PID: 26324 ssh.py:405 - DEBUG - PID: 26324 ssh.py:405 - DEBUG - Since the script you are attempting to invoke has been converted to an PID: 26324 ssh.py:405 - DEBUG - Upstart job, you may also use the start(8) utility, e.g. start portmap PID: 26324 ssh.py:254 - DEBUG - new /etc/fstab after removing regex ( /home | /opt/sge6 | /db ) matches: # /etc/fstab: static file system information. # proc /proc proc nodev,noexec,nosuid 0 0 /dev/sda1 / ext3 defaults 0 0 /dev/sdb /mnt auto defaults,comment=cloudconfig 0 0 PID: 26324 ssh.py:405 - DEBUG - /dev/sda1 on / type ext3 (rw) PID: 26324 ssh.py:405 - DEBUG - proc on /proc type proc (rw,noexec,nosuid,nodev) PID: 26324 ssh.py:405 - DEBUG - none on /sys type sysfs (rw,noexec,nosuid,nodev) PID: 26324 ssh.py:405 - DEBUG - none on /sys/kernel/debug type debugfs (rw) PID: 26324 ssh.py:405 - DEBUG - none on /sys/kernel/security type securityfs (rw) PID: 26324 ssh.py:405 - DEBUG - devtmpfs on /dev type devtmpfs (rw,mode=0755) PID: 26324 ssh.py:405 - DEBUG - none on /dev/pts type devpts (rw,noexec,nosuid,gid=5,mode=0620) PID: 26324 ssh.py:405 - DEBUG - none on /dev/shm type tmpfs (rw,nosuid,nodev) PID: 26324 ssh.py:405 - DEBUG - none on /var/run type tmpfs (rw,nosuid,mode=0755) PID: 26324 ssh.py:405 - DEBUG - none on /var/lock type tmpfs (rw,noexec,nosuid,nodev) PID: 26324 ssh.py:405 - DEBUG - none on /lib/init/rw type tmpfs (rw,nosuid,mode=0755) PID: 26324 ssh.py:405 - DEBUG - /dev/sdb on /mnt type ext3 (rw) PID: 26324 ssh.py:405 - DEBUG - nfsd on /proc/fs/nfsd type nfsd (rw) PID: 26324 clustersetup.py:304 - INFO - Mounting shares for node node009 PID: 26324 threadpool.py:108 - DEBUG - unfinished_tasks = 9 PID: 26324 ssh.py:405 - DEBUG - Rather than invoking init scripts through /etc/init.d, use the service(8) PID: 26324 ssh.py:405 - DEBUG - utility, e.g. service portmap start PID: 26324 ssh.py:405 - DEBUG - PID: 26324 ssh.py:405 - DEBUG - Since the script you are attempting to invoke has been converted to an PID: 26324 ssh.py:405 - DEBUG - Upstart job, you may also use the start(8) utility, e.g. start portmap PID: 26324 ssh.py:254 - DEBUG - new /etc/fstab after removing regex ( /home | /opt/sge6 | /db ) matches: # /etc/fstab: static file system information. # proc /proc proc nodev,noexec,nosuid 0 0 /dev/sda1 / ext3 defaults 0 0 /dev/sdb /mnt auto defaults,comment=cloudconfig 0 0 PID: 26324 ssh.py:402 - DEBUG - command mount -t devpts none /dev/pts failed with status 32 PID: 26324 ssh.py:405 - DEBUG - mount: none already mounted or /dev/pts busy PID: 26324 ssh.py:405 - DEBUG - mount: according to mtab, none is already mounted on /dev/pts PID: 26324 ssh.py:405 - DEBUG - Rather than invoking init scripts through /etc/init.d, use the service(8) PID: 26324 ssh.py:405 - DEBUG - utility, e.g. service portmap start PID: 26324 ssh.py:405 - DEBUG - PID: 26324 ssh.py:405 - DEBUG - Since the script you are attempting to invoke has been converted to an PID: 26324 ssh.py:405 - DEBUG - Upstart job, you may also use the start(8) utility, e.g. start portmap PID: 26324 ssh.py:402 - DEBUG - command mount -t devpts none /dev/pts failed with status 32 PID: 26324 ssh.py:405 - DEBUG - mount: none already mounted or /dev/pts busy PID: 26324 ssh.py:405 - DEBUG - mount: according to mtab, none is already mounted on /dev/pts PID: 26324 ssh.py:402 - DEBUG - command mount -t devpts none /dev/pts failed with status 32 PID: 26324 ssh.py:405 - DEBUG - mount: none already mounted or /dev/pts busy PID: 26324 ssh.py:405 - DEBUG - mount: according to mtab, none is already mounted on /dev/pts PID: 26324 ssh.py:405 - DEBUG - Rather than invoking init scripts through /etc/init.d, use the service(8) PID: 26324 ssh.py:405 - DEBUG - utility, e.g. service portmap start PID: 26324 ssh.py:405 - DEBUG - PID: 26324 ssh.py:405 - DEBUG - Since the script you are attempting to invoke has been converted to an PID: 26324 ssh.py:405 - DEBUG - Upstart job, you may also use the start(8) utility, e.g. start portmap PID: 26324 ssh.py:405 - DEBUG - Rather than invoking init scripts through /etc/init.d, use the service(8) PID: 26324 ssh.py:405 - DEBUG - utility, e.g. service portmap start PID: 26324 ssh.py:405 - DEBUG - PID: 26324 ssh.py:405 - DEBUG - Since the script you are attempting to invoke has been converted to an PID: 26324 ssh.py:405 - DEBUG - Upstart job, you may also use the start(8) utility, e.g. start portmap PID: 26324 ssh.py:254 - DEBUG - new /etc/fstab after removing regex ( /home | /opt/sge6 | /db ) matches: # /etc/fstab: static file system information. # proc /proc proc nodev,noexec,nosuid 0 0 /dev/sda1 / ext3 defaults 0 0 /dev/sdb /mnt auto defaults,comment=cloudconfig 0 0 PID: 26324 ssh.py:402 - DEBUG - command mount -t devpts none /dev/pts failed with status 32 PID: 26324 ssh.py:405 - DEBUG - mount: none already mounted or /dev/pts busy PID: 26324 ssh.py:405 - DEBUG - mount: according to mtab, none is already mounted on /dev/pts PID: 26324 ssh.py:402 - DEBUG - command mount -t devpts none /dev/pts failed with status 32 PID: 26324 ssh.py:405 - DEBUG - mount: none already mounted or /dev/pts busy PID: 26324 ssh.py:405 - DEBUG - mount: according to mtab, none is already mounted on /dev/pts PID: 26324 ssh.py:402 - DEBUG - command mount -t devpts none /dev/pts failed with status 32 PID: 26324 ssh.py:405 - DEBUG - mount: none already mounted or /dev/pts busy PID: 26324 ssh.py:405 - DEBUG - mount: according to mtab, none is already mounted on /dev/pts PID: 26324 ssh.py:254 - DEBUG - new /etc/fstab after removing regex ( /home | /opt/sge6 | /db ) matches: # /etc/fstab: static file system information. # proc /proc proc nodev,noexec,nosuid 0 0 /dev/sda1 / ext3 defaults 0 0 /dev/sdb /mnt auto defaults,comment=cloudconfig 0 0 PID: 26324 ssh.py:254 - DEBUG - new /etc/fstab after removing regex ( /home | /opt/sge6 | /db ) matches: # /etc/fstab: static file system information. # proc /proc proc nodev,noexec,nosuid 0 0 /dev/sda1 / ext3 defaults 0 0 /dev/sdb /mnt auto defaults,comment=cloudconfig 0 0 PID: 26324 ssh.py:254 - DEBUG - new /etc/fstab after removing regex ( /home | /opt/sge6 | /db ) matches: # /etc/fstab: static file system information. # proc /proc proc nodev,noexec,nosuid 0 0 /dev/sda1 / ext3 defaults 0 0 /dev/sdb /mnt auto defaults,comment=cloudconfig 0 0 PID: 26324 ssh.py:254 - DEBUG - new /etc/fstab after removing regex ( /home | /opt/sge6 | /db ) matches: # /etc/fstab: static file system information. # proc /proc proc nodev,noexec,nosuid 0 0 /dev/sda1 / ext3 defaults 0 0 /dev/sdb /mnt auto defaults,comment=cloudconfig 0 0 PID: 26324 ssh.py:254 - DEBUG - new /etc/fstab after removing regex ( /home | /opt/sge6 | /db ) matches: # /etc/fstab: static file system information. # proc /proc proc nodev,noexec,nosuid 0 0 /dev/sda1 / ext3 defaults 0 0 /dev/sdb /mnt auto defaults,comment=cloudconfig 0 0 PID: 26324 ssh.py:254 - DEBUG - new /etc/fstab after removing regex ( /home | /opt/sge6 | /db ) matches: # /etc/fstab: static file system information. # proc /proc proc nodev,noexec,nosuid 0 0 /dev/sda1 / ext3 defaults 0 0 /dev/sdb /mnt auto defaults,comment=cloudconfig 0 0 PID: 26324 threadpool.py:108 - DEBUG - unfinished_tasks = 7 PID: 26324 utils.py:61 - INFO - _setup_nfs took 0.125 mins PID: 26324 clustersetup.py:207 - INFO - Configuring passwordless ssh for root PID: 26324 node.py:396 - DEBUG - adding auth_key_contents PID: 26324 clustersetup.py:215 - INFO - Configuring passwordless ssh for sgeadmin PID: 26324 node.py:396 - DEBUG - adding auth_key_contents PID: 26324 node.py:404 - DEBUG - adding conn_pubkey_contents PID: 26324 ssh.py:402 - DEBUG - command pkill -9 sge failed with status 1 PID: 26324 ssh.py:402 - DEBUG - command rm /etc/init.d/sge* failed with status 1 PID: 26324 ssh.py:405 - DEBUG - rm: cannot remove `/etc/init.d/sge*': No such file or directory PID: 26324 ssh.py:405 - DEBUG - lx24-amd64 PID: 26324 ssh.py:405 - DEBUG - [: 359: 11: unexpected operator PID: 26324 ssh.py:405 - DEBUG - [: 359: 11: unexpected operator PID: 26324 ssh.py:402 - DEBUG - command pkill -9 sge failed with status 1 PID: 26324 ssh.py:402 - DEBUG - command rm /etc/init.d/sge* failed with status 1 PID: 26324 ssh.py:405 - DEBUG - rm: cannot remove `/etc/init.d/sge*': No such file or directory PID: 26324 ssh.py:405 - DEBUG - lx24-amd64 PID: 26324 ssh.py:405 - DEBUG - [: 359: 11: unexpected operator PID: 26324 ssh.py:405 - DEBUG - [: 359: 11: unexpected operator PID: 26324 ssh.py:402 - DEBUG - command pkill -9 sge failed with status 1 PID: 26324 ssh.py:402 - DEBUG - command rm /etc/init.d/sge* failed with status 1 PID: 26324 ssh.py:405 - DEBUG - rm: cannot remove `/etc/init.d/sge*': No such file or directory PID: 26324 ssh.py:405 - DEBUG - lx24-amd64 PID: 26324 ssh.py:405 - DEBUG - [: 359: 11: unexpected operator PID: 26324 ssh.py:405 - DEBUG - [: 359: 11: unexpected operator PID: 26324 ssh.py:402 - DEBUG - command pkill -9 sge failed with status 1 PID: 26324 ssh.py:402 - DEBUG - command rm /etc/init.d/sge* failed with status 1 PID: 26324 ssh.py:405 - DEBUG - rm: cannot remove `/etc/init.d/sge*': No such file or directory PID: 26324 ssh.py:405 - DEBUG - lx24-amd64 PID: 26324 ssh.py:405 - DEBUG - [: 359: 11: unexpected operator PID: 26324 ssh.py:405 - DEBUG - [: 359: 11: unexpected operator PID: 26324 ssh.py:402 - DEBUG - command pkill -9 sge failed with status 1 PID: 26324 ssh.py:402 - DEBUG - command rm /etc/init.d/sge* failed with status 1 PID: 26324 ssh.py:405 - DEBUG - rm: cannot remove `/etc/init.d/sge*': No such file or directory PID: 26324 ssh.py:405 - DEBUG - lx24-amd64 PID: 26324 ssh.py:405 - DEBUG - [: 359: 11: unexpected operator PID: 26324 ssh.py:405 - DEBUG - [: 359: 11: unexpected operator PID: 26324 ssh.py:402 - DEBUG - command pkill -9 sge failed with status 1 PID: 26324 ssh.py:402 - DEBUG - command rm /etc/init.d/sge* failed with status 1 PID: 26324 ssh.py:405 - DEBUG - rm: cannot remove `/etc/init.d/sge*': No such file or directory PID: 26324 ssh.py:405 - DEBUG - lx24-amd64 PID: 26324 ssh.py:405 - DEBUG - [: 359: 11: unexpected operator PID: 26324 ssh.py:405 - DEBUG - [: 359: 11: unexpected operator PID: 26324 ssh.py:402 - DEBUG - command pkill -9 sge failed with status 1 PID: 26324 ssh.py:402 - DEBUG - command rm /etc/init.d/sge* failed with status 1 PID: 26324 ssh.py:405 - DEBUG - rm: cannot remove `/etc/init.d/sge*': No such file or directory PID: 26324 ssh.py:405 - DEBUG - lx24-amd64 PID: 26324 ssh.py:405 - DEBUG - [: 359: 11: unexpected operator PID: 26324 ssh.py:405 - DEBUG - [: 359: 11: unexpected operator PID: 26324 ssh.py:402 - DEBUG - command pkill -9 sge failed with status 1 PID: 26324 ssh.py:402 - DEBUG - command rm /etc/init.d/sge* failed with status 1 PID: 26324 ssh.py:405 - DEBUG - rm: cannot remove `/etc/init.d/sge*': No such file or directory PID: 26324 ssh.py:405 - DEBUG - lx24-amd64 PID: 26324 ssh.py:405 - DEBUG - [: 359: 11: unexpected operator PID: 26324 ssh.py:405 - DEBUG - [: 359: 11: unexpected operator PID: 26324 ssh.py:402 - DEBUG - command pkill -9 sge failed with status 1 PID: 26324 ssh.py:402 - DEBUG - command rm /etc/init.d/sge* failed with status 1 PID: 26324 ssh.py:405 - DEBUG - rm: cannot remove `/etc/init.d/sge*': No such file or directory PID: 26324 ssh.py:405 - DEBUG - lx24-amd64 PID: 26324 ssh.py:405 - DEBUG - [: 359: 11: unexpected operator PID: 26324 ssh.py:405 - DEBUG - [: 359: 11: unexpected operator PID: 26324 ssh.py:402 - DEBUG - command pkill -9 sge failed with status 1 PID: 26324 ssh.py:402 - DEBUG - command rm /etc/init.d/sge* failed with status 1 PID: 26324 ssh.py:405 - DEBUG - rm: cannot remove `/etc/init.d/sge*': No such file or directory PID: 26324 ssh.py:405 - DEBUG - lx24-amd64 PID: 26324 ssh.py:405 - DEBUG - [: 359: 11: unexpected operator PID: 26324 ssh.py:405 - DEBUG - [: 359: 11: unexpected operator PID: 26324 clustersetup.py:360 - INFO - Installing Sun Grid Engine... PID: 26324 ssh.py:405 - DEBUG - Reading configuration from file ./ec2_sge.conf PID: 26324 ssh.py:405 - DEBUG - [H[2JInstall log can be found in: /opt/sge6/default/common/install_logs/qmaster_install_master_2011-08-05_15:58:28.log PID: 26324 ssh.py:405 - DEBUG - Install log can be found in: /opt/sge6/default/common/install_logs/execd_install_master_2011-08-05_15:58:33.log PID: 26324 ssh.py:405 - DEBUG - [: 359: 11: unexpected operator PID: 26324 ssh.py:405 - DEBUG - [: 359: 11: unexpected operator PID: 26324 ssh.py:405 - DEBUG - [: 359: 11: unexpected operator PID: 26324 ssh.py:405 - DEBUG - [: 359: 11: unexpected operator PID: 26324 ssh.py:405 - DEBUG - root@master modified "all.q" in cluster queue list PID: 26324 ssh.py:405 - DEBUG - adminhost "node001" already exists PID: 26324 ssh.py:405 - DEBUG - submithost "node001" already exists PID: 26324 ssh.py:405 - DEBUG - adminhost "node002" already exists PID: 26324 ssh.py:405 - DEBUG - lx24-amd64 PID: 26324 ssh.py:405 - DEBUG - [: 359: 11: unexpected operator PID: 26324 ssh.py:405 - DEBUG - [: 359: 11: unexpected operator PID: 26324 ssh.py:405 - DEBUG - submithost "node002" already exists PID: 26324 ssh.py:405 - DEBUG - adminhost "node001" already exists PID: 26324 ssh.py:405 - DEBUG - adminhost "node003" already exists PID: 26324 ssh.py:405 - DEBUG - lx24-amd64 PID: 26324 ssh.py:405 - DEBUG - [: 359: 11: unexpected operator PID: 26324 ssh.py:405 - DEBUG - [: 359: 11: unexpected operator PID: 26324 ssh.py:405 - DEBUG - submithost "node003" already exists PID: 26324 ssh.py:405 - DEBUG - submithost "node001" already exists PID: 26324 ssh.py:405 - DEBUG - adminhost "node002" already exists PID: 26324 ssh.py:405 - DEBUG - adminhost "node004" already exists PID: 26324 ssh.py:405 - DEBUG - lx24-amd64 PID: 26324 ssh.py:405 - DEBUG - [: 359: 11: unexpected operator PID: 26324 ssh.py:405 - DEBUG - [: 359: 11: unexpected operator PID: 26324 ssh.py:405 - DEBUG - submithost "node004" already exists PID: 26324 ssh.py:405 - DEBUG - submithost "node002" already exists PID: 26324 ssh.py:405 - DEBUG - adminhost "node003" already exists PID: 26324 ssh.py:405 - DEBUG - lx24-amd64 PID: 26324 ssh.py:405 - DEBUG - [: 359: 11: unexpected operator PID: 26324 ssh.py:405 - DEBUG - [: 359: 11: unexpected operator PID: 26324 ssh.py:405 - DEBUG - adminhost "node005" already exists PID: 26324 ssh.py:405 - DEBUG - submithost "node003" already exists PID: 26324 ssh.py:405 - DEBUG - submithost "node005" already exists PID: 26324 ssh.py:405 - DEBUG - adminhost "node004" already exists PID: 26324 ssh.py:405 - DEBUG - lx24-amd64 PID: 26324 ssh.py:405 - DEBUG - [: 359: 11: unexpected operator PID: 26324 ssh.py:405 - DEBUG - [: 359: 11: unexpected operator PID: 26324 ssh.py:405 - DEBUG - adminhost "node006" already exists PID: 26324 ssh.py:405 - DEBUG - submithost "node004" already exists PID: 26324 ssh.py:405 - DEBUG - submithost "node006" already exists PID: 26324 ssh.py:405 - DEBUG - adminhost "node005" already exists PID: 26324 ssh.py:405 - DEBUG - adminhost "node007" already exists PID: 26324 ssh.py:405 - DEBUG - submithost "node005" already exists PID: 26324 ssh.py:405 - DEBUG - submithost "node007" already exists PID: 26324 ssh.py:405 - DEBUG - adminhost "node008" already exists PID: 26324 ssh.py:405 - DEBUG - lx24-amd64 PID: 26324 ssh.py:405 - DEBUG - [: 359: 11: unexpected operator PID: 26324 ssh.py:405 - DEBUG - [: 359: 11: unexpected operator PID: 26324 ssh.py:405 - DEBUG - lx24-amd64 PID: 26324 ssh.py:405 - DEBUG - [: 359: 11: unexpected operator PID: 26324 ssh.py:405 - DEBUG - [: 359: 11: unexpected operator PID: 26324 ssh.py:405 - DEBUG - submithost "node008" already exists PID: 26324 ssh.py:405 - DEBUG - adminhost "node007" already exists PID: 26324 ssh.py:405 - DEBUG - lx24-amd64 PID: 26324 ssh.py:405 - DEBUG - [: 359: 11: unexpected operator PID: 26324 ssh.py:405 - DEBUG - [: 359: 11: unexpected operator PID: 26324 ssh.py:405 - DEBUG - adminhost "node009" already exists PID: 26324 ssh.py:405 - DEBUG - adminhost "node006" already exists PID: 26324 ssh.py:405 - DEBUG - submithost "node007" already exists PID: 26324 ssh.py:405 - DEBUG - submithost "node009" already exists PID: 26324 threadpool.py:108 - DEBUG - unfinished_tasks = 9 PID: 26324 ssh.py:405 - DEBUG - submithost "node006" already exists PID: 26324 ssh.py:405 - DEBUG - adminhost "node008" already exists PID: 26324 ssh.py:405 - DEBUG - lx24-amd64 PID: 26324 ssh.py:405 - DEBUG - [: 359: 11: unexpected operator PID: 26324 ssh.py:405 - DEBUG - [: 359: 11: unexpected operator PID: 26324 ssh.py:405 - DEBUG - submithost "node008" already exists PID: 26324 ssh.py:405 - DEBUG - adminhost "node009" already exists PID: 26324 ssh.py:405 - DEBUG - submithost "node009" already exists PID: 26324 threadpool.py:108 - DEBUG - unfinished_tasks = 9 PID: 26324 threadpool.py:108 - DEBUG - unfinished_tasks = 9 PID: 26324 threadpool.py:108 - DEBUG - unfinished_tasks = 9 PID: 26324 ssh.py:405 - DEBUG - Reading configuration from file ./ec2_sge.conf PID: 26324 ssh.py:405 - DEBUG - Install log can be found in: /opt/sge6/default/common/install_logs/execd_install_node001_2011-08-05_15:58:38.log PID: 26324 ssh.py:405 - DEBUG - [: 359: 11: unexpected operator PID: 26324 ssh.py:405 - DEBUG - [: 359: 11: unexpected operator PID: 26324 ssh.py:405 - DEBUG - [: 359: 11: unexpected operator PID: 26324 ssh.py:405 - DEBUG - [: 359: 11: unexpected operator PID: 26324 threadpool.py:108 - DEBUG - unfinished_tasks = 8 PID: 26324 ssh.py:405 - DEBUG - Reading configuration from file ./ec2_sge.conf PID: 26324 ssh.py:405 - DEBUG - Install log can be found in: /opt/sge6/default/common/install_logs/execd_install_node002_2011-08-05_15:58:39.log PID: 26324 ssh.py:405 - DEBUG - [: 359: 11: unexpected operator PID: 26324 ssh.py:405 - DEBUG - [: 359: 11: unexpected operator PID: 26324 ssh.py:405 - DEBUG - [: 359: 11: unexpected operator PID: 26324 ssh.py:405 - DEBUG - [: 359: 11: unexpected operator PID: 26324 ssh.py:405 - DEBUG - Reading configuration from file ./ec2_sge.conf PID: 26324 ssh.py:405 - DEBUG - Install log can be found in: /opt/sge6/default/common/install_logs/execd_install_node003_2011-08-05_15:58:39.log PID: 26324 ssh.py:405 - DEBUG - [: 359: 11: unexpected operator PID: 26324 ssh.py:405 - DEBUG - [: 359: 11: unexpected operator PID: 26324 ssh.py:405 - DEBUG - [: 359: 11: unexpected operator PID: 26324 ssh.py:405 - DEBUG - [: 359: 11: unexpected operator PID: 26324 ssh.py:405 - DEBUG - Reading configuration from file ./ec2_sge.conf PID: 26324 ssh.py:405 - DEBUG - Install log can be found in: /opt/sge6/default/common/install_logs/execd_install_node004_2011-08-05_15:58:39.log PID: 26324 ssh.py:405 - DEBUG - [: 359: 11: unexpected operator PID: 26324 ssh.py:405 - DEBUG - [: 359: 11: unexpected operator PID: 26324 ssh.py:405 - DEBUG - [: 359: 11: unexpected operator PID: 26324 ssh.py:405 - DEBUG - [: 359: 11: unexpected operator PID: 26324 ssh.py:405 - DEBUG - Reading configuration from file ./ec2_sge.conf PID: 26324 ssh.py:405 - DEBUG - Install log can be found in: /opt/sge6/default/common/install_logs/execd_install_node005_2011-08-05_15:58:39.log PID: 26324 ssh.py:405 - DEBUG - [: 359: 11: unexpected operator PID: 26324 ssh.py:405 - DEBUG - [: 359: 11: unexpected operator PID: 26324 ssh.py:405 - DEBUG - [: 359: 11: unexpected operator PID: 26324 ssh.py:405 - DEBUG - [: 359: 11: unexpected operator PID: 26324 ssh.py:405 - DEBUG - Reading configuration from file ./ec2_sge.conf PID: 26324 ssh.py:405 - DEBUG - Install log can be found in: /opt/sge6/default/common/install_logs/execd_install_node006_2011-08-05_15:58:39.log PID: 26324 ssh.py:405 - DEBUG - [: 359: 11: unexpected operator PID: 26324 ssh.py:405 - DEBUG - [: 359: 11: unexpected operator PID: 26324 ssh.py:405 - DEBUG - [: 359: 11: unexpected operator PID: 26324 ssh.py:405 - DEBUG - [: 359: 11: unexpected operator PID: 26324 threadpool.py:108 - DEBUG - unfinished_tasks = 3 PID: 26324 ssh.py:405 - DEBUG - Reading configuration from file ./ec2_sge.conf PID: 26324 ssh.py:405 - DEBUG - Install log can be found in: /opt/sge6/default/common/install_logs/execd_install_node007_2011-08-05_15:58:39.log PID: 26324 ssh.py:405 - DEBUG - [: 359: 11: unexpected operator PID: 26324 ssh.py:405 - DEBUG - [: 359: 11: unexpected operator PID: 26324 ssh.py:405 - DEBUG - [: 359: 11: unexpected operator PID: 26324 ssh.py:405 - DEBUG - [: 359: 11: unexpected operator PID: 26324 ssh.py:405 - DEBUG - Reading configuration from file ./ec2_sge.conf PID: 26324 ssh.py:405 - DEBUG - Install log can be found in: /opt/sge6/default/common/install_logs/execd_install_node008_2011-08-05_15:58:40.log PID: 26324 ssh.py:405 - DEBUG - [: 359: 11: unexpected operator PID: 26324 ssh.py:405 - DEBUG - [: 359: 11: unexpected operator PID: 26324 ssh.py:405 - DEBUG - [: 359: 11: unexpected operator PID: 26324 ssh.py:405 - DEBUG - [: 359: 11: unexpected operator PID: 26324 ssh.py:405 - DEBUG - Reading configuration from file ./ec2_sge.conf PID: 26324 ssh.py:405 - DEBUG - Install log can be found in: /opt/sge6/default/common/install_logs/execd_install_node009_2011-08-05_15:58:40.log PID: 26324 ssh.py:405 - DEBUG - [: 359: 11: unexpected operator PID: 26324 ssh.py:405 - DEBUG - [: 359: 11: unexpected operator PID: 26324 ssh.py:405 - DEBUG - [: 359: 11: unexpected operator PID: 26324 ssh.py:405 - DEBUG - [: 359: 11: unexpected operator PID: 26324 ssh.py:405 - DEBUG - 4 PID: 26324 ssh.py:405 - DEBUG - 4 PID: 26324 ssh.py:405 - DEBUG - 4 PID: 26324 ssh.py:405 - DEBUG - 4 PID: 26324 ssh.py:405 - DEBUG - 4 PID: 26324 ssh.py:405 - DEBUG - 4 PID: 26324 ssh.py:405 - DEBUG - 4 PID: 26324 ssh.py:405 - DEBUG - 4 PID: 26324 ssh.py:405 - DEBUG - 4 PID: 26324 ssh.py:405 - DEBUG - 4 PID: 26324 ssh.py:405 - DEBUG - root@master added "orte" to parallel environment list PID: 26324 ssh.py:405 - DEBUG - Unable to find "orte" in "pe_list" of "all.q" - Adding new element. PID: 26324 ssh.py:405 - DEBUG - PID: 26324 ssh.py:405 - DEBUG - root@master modified "all.q" in cluster queue list PID: 26324 threadpool.py:96 - INFO - Shutting down threads... PID: 26324 threadpool.py:108 - DEBUG - unfinished_tasks = 1 PID: 26324 cluster.py:1243 - INFO - The cluster has been started and configured. Login to the master node as root by running: $ starcluster sshmaster mycluster When you are finished using the cluster, run: $ starcluster stop mycluster to shutdown the cluster and stop paying for service. If this cluster uses EBS instances then the 'stop' command above will put all nodes into a 'stopped' state. The cluster may then be restarted at a later time, without losing data, by passing the -x option to the 'start' command. To completely terminate an EBS cluster: $ starcluster terminate mycluster NOTE: Terminating an EBS cluster will destroy all volumes backing the nodes. PID: 26324 utils.py:72 - INFO - Starting cluster took 4.467 mins PID: 26324 ssh.py:536 - DEBUG - __del__ called PID: 26324 ssh.py:536 - DEBUG - __del__ called PID: 26324 ssh.py:536 - DEBUG - __del__ called PID: 26324 ssh.py:536 - DEBUG - __del__ called PID: 26324 ssh.py:536 - DEBUG - __del__ called PID: 26324 ssh.py:536 - DEBUG - __del__ called PID: 26324 ssh.py:536 - DEBUG - __del__ called PID: 26324 ssh.py:536 - DEBUG - __del__ called PID: 26324 ssh.py:536 - DEBUG - __del__ called PID: 26324 ssh.py:536 - DEBUG - __del__ called PID: 26393 config.py:515 - DEBUG - Loading config PID: 26393 config.py:108 - DEBUG - Loading file: /home/jpk37/.starcluster/config PID: 26393 awsutils.py:54 - DEBUG - creating self._conn w/ connection_authenticator kwargs = {'path': '/', 'region': None, 'port': None, 'is_secure': True} PID: 26393 cluster.py:670 - DEBUG - existing nodes: {} PID: 26393 cluster.py:678 - DEBUG - adding node i-ff8bef9e to self._nodes list PID: 26393 cluster.py:678 - DEBUG - adding node i-c18befa0 to self._nodes list PID: 26393 cluster.py:678 - DEBUG - adding node i-c38befa2 to self._nodes list PID: 26393 cluster.py:678 - DEBUG - adding node i-c58befa4 to self._nodes list PID: 26393 cluster.py:678 - DEBUG - adding node i-c78befa6 to self._nodes list PID: 26393 cluster.py:678 - DEBUG - adding node i-c98befa8 to self._nodes list PID: 26393 cluster.py:678 - DEBUG - adding node i-cb8befaa to self._nodes list PID: 26393 cluster.py:678 - DEBUG - adding node i-cd8befac to self._nodes list PID: 26393 cluster.py:678 - DEBUG - adding node i-cf8befae to self._nodes list PID: 26393 cluster.py:678 - DEBUG - adding node i-d18befb0 to self._nodes list PID: 26393 cluster.py:686 - DEBUG - returning self._nodes = [, , , , , , , , , ] PID: 26393 node.py:822 - DEBUG - using system's ssh client PID: 26393 node.py:825 - DEBUG - ssh_cmd: ssh -i /home/jpk37/Downloads/blastkey.pem root@ec2-50-19-70-119.compute-1.amazonaws.com PID: 26632 config.py:515 - DEBUG - Loading config PID: 26632 config.py:108 - DEBUG - Loading file: /home/jpk37/.starcluster/config PID: 26632 awsutils.py:54 - DEBUG - creating self._conn w/ connection_authenticator kwargs = {'path': '/', 'region': None, 'port': None, 'is_secure': True} PID: 26632 cluster.py:670 - DEBUG - existing nodes: {} PID: 26632 cluster.py:678 - DEBUG - adding node i-ff8bef9e to self._nodes list PID: 26632 cluster.py:678 - DEBUG - adding node i-c18befa0 to self._nodes list PID: 26632 cluster.py:678 - DEBUG - adding node i-c38befa2 to self._nodes list PID: 26632 cluster.py:678 - DEBUG - adding node i-c58befa4 to self._nodes list PID: 26632 cluster.py:678 - DEBUG - adding node i-c78befa6 to self._nodes list PID: 26632 cluster.py:678 - DEBUG - adding node i-c98befa8 to self._nodes list PID: 26632 cluster.py:678 - DEBUG - adding node i-cb8befaa to self._nodes list PID: 26632 cluster.py:678 - DEBUG - adding node i-cd8befac to self._nodes list PID: 26632 cluster.py:678 - DEBUG - adding node i-cf8befae to self._nodes list PID: 26632 cluster.py:678 - DEBUG - adding node i-d18befb0 to self._nodes list PID: 26632 cluster.py:686 - DEBUG - returning self._nodes = [, , , , , , , , , ] PID: 26632 cluster.py:670 - DEBUG - existing nodes: {u'i-c38befa2': , u'i-cf8befae': , u'i-c78befa6': , u'i-c18befa0': , u'i-cb8befaa': , u'i-d18befb0': , u'i-cd8befac': , u'i-c58befa4': , u'i-ff8bef9e': , u'i-c98befa8': } PID: 26632 cluster.py:673 - DEBUG - updating existing node i-ff8bef9e in self._nodes PID: 26632 cluster.py:673 - DEBUG - updating existing node i-c18befa0 in self._nodes PID: 26632 cluster.py:673 - DEBUG - updating existing node i-c38befa2 in self._nodes PID: 26632 cluster.py:673 - DEBUG - updating existing node i-c58befa4 in self._nodes PID: 26632 cluster.py:673 - DEBUG - updating existing node i-c78befa6 in self._nodes PID: 26632 cluster.py:673 - DEBUG - updating existing node i-c98befa8 in self._nodes PID: 26632 cluster.py:673 - DEBUG - updating existing node i-cb8befaa in self._nodes PID: 26632 cluster.py:673 - DEBUG - updating existing node i-cd8befac in self._nodes PID: 26632 cluster.py:673 - DEBUG - updating existing node i-cf8befae in self._nodes PID: 26632 cluster.py:673 - DEBUG - updating existing node i-d18befb0 in self._nodes PID: 26632 cluster.py:686 - DEBUG - returning self._nodes = [, , , , , , , , , ] PID: 26632 node.py:659 - INFO - Detaching volume vol-b2ff16d8 from master PID: 26632 cluster.py:670 - DEBUG - existing nodes: {u'i-c38befa2': , u'i-cf8befae': , u'i-c78befa6': , u'i-c18befa0': , u'i-cb8befaa': , u'i-d18befb0': , u'i-cd8befac': , u'i-c58befa4': , u'i-ff8bef9e': , u'i-c98befa8': } PID: 26632 cluster.py:673 - DEBUG - updating existing node i-ff8bef9e in self._nodes PID: 26632 cluster.py:673 - DEBUG - updating existing node i-c18befa0 in self._nodes PID: 26632 cluster.py:673 - DEBUG - updating existing node i-c38befa2 in self._nodes PID: 26632 cluster.py:673 - DEBUG - updating existing node i-c58befa4 in self._nodes PID: 26632 cluster.py:673 - DEBUG - updating existing node i-c78befa6 in self._nodes PID: 26632 cluster.py:673 - DEBUG - updating existing node i-c98befa8 in self._nodes PID: 26632 cluster.py:673 - DEBUG - updating existing node i-cb8befaa in self._nodes PID: 26632 cluster.py:673 - DEBUG - updating existing node i-cd8befac in self._nodes PID: 26632 cluster.py:673 - DEBUG - updating existing node i-cf8befae in self._nodes PID: 26632 cluster.py:673 - DEBUG - updating existing node i-d18befb0 in self._nodes PID: 26632 cluster.py:686 - DEBUG - returning self._nodes = [, , , , , , , , , ] PID: 26632 node.py:745 - INFO - Terminating node: master (i-ff8bef9e) PID: 26632 node.py:745 - INFO - Terminating node: node001 (i-c18befa0) PID: 26632 node.py:745 - INFO - Terminating node: node002 (i-c38befa2) PID: 26632 node.py:745 - INFO - Terminating node: node003 (i-c58befa4) PID: 26632 node.py:745 - INFO - Terminating node: node004 (i-c78befa6) PID: 26632 node.py:745 - INFO - Terminating node: node005 (i-c98befa8) PID: 26632 node.py:745 - INFO - Terminating node: node006 (i-cb8befaa) PID: 26632 node.py:745 - INFO - Terminating node: node007 (i-cd8befac) PID: 26632 node.py:745 - INFO - Terminating node: node008 (i-cf8befae) PID: 26632 node.py:745 - INFO - Terminating node: node009 (i-d18befb0) PID: 26632 cluster.py:1196 - INFO - Removing @sc-mycluster security group PID: 26632 cli.py:157 - ERROR - InvalidGroup.InUse: There are active instances using security group '@sc-mycluster' PID: 26635 config.py:515 - DEBUG - Loading config PID: 26635 config.py:108 - DEBUG - Loading file: /home/jpk37/.starcluster/config PID: 26635 awsutils.py:54 - DEBUG - creating self._conn w/ connection_authenticator kwargs = {'path': '/', 'region': None, 'port': None, 'is_secure': True} PID: 26635 cluster.py:670 - DEBUG - existing nodes: {} PID: 26635 cluster.py:686 - DEBUG - returning self._nodes = [] PID: 26635 cluster.py:670 - DEBUG - existing nodes: {} PID: 26635 cluster.py:686 - DEBUG - returning self._nodes = [] PID: 26635 cluster.py:670 - DEBUG - existing nodes: {} PID: 26635 cluster.py:686 - DEBUG - returning self._nodes = [] PID: 26635 cluster.py:1196 - INFO - Removing @sc-mycluster security group PID: 26780 config.py:515 - DEBUG - Loading config PID: 26780 config.py:108 - DEBUG - Loading file: /home/jpk37/.starcluster/config PID: 26780 config.py:515 - DEBUG - Loading config PID: 26780 config.py:108 - DEBUG - Loading file: /home/jpk37/.starcluster/config PID: 26780 awsutils.py:54 - DEBUG - creating self._conn w/ connection_authenticator kwargs = {'path': '/', 'region': None, 'port': None, 'is_secure': True} PID: 26780 start.py:167 - INFO - Using default cluster template: smallcluster PID: 26780 cluster.py:1333 - INFO - Validating cluster template settings... PID: 26780 cluster.py:872 - DEBUG - Launch map: node001 (ami: ami-0af31963, type: m1.xlarge)... PID: 26780 cluster.py:1355 - INFO - Cluster template settings are valid PID: 26780 cluster.py:1228 - INFO - Starting cluster... PID: 26780 cluster.py:891 - INFO - Launching a 2-node cluster... PID: 26780 cluster.py:908 - INFO - Launching a 2-node cluster... PID: 26780 cluster.py:872 - DEBUG - Launch map: node001 (ami: ami-0af31963, type: m1.xlarge)... PID: 26780 cluster.py:919 - DEBUG - Launching master (ami: ami-0af31963, type: m1.xlarge) PID: 26780 cluster.py:919 - DEBUG - Launching node001 (ami: ami-0af31963, type: m1.xlarge) PID: 26780 awsutils.py:175 - INFO - Creating security group @sc-mycluster... PID: 26780 cluster.py:1048 - INFO - Waiting for cluster to come up... (updating every 30s) PID: 26780 cluster.py:670 - DEBUG - existing nodes: {} PID: 26780 cluster.py:678 - DEBUG - adding node i-59781f38 to self._nodes list PID: 26780 cluster.py:678 - DEBUG - adding node i-5b781f3a to self._nodes list PID: 26780 cluster.py:686 - DEBUG - returning self._nodes = [, ] PID: 26780 cluster.py:1070 - INFO - Waiting for all nodes to be in a 'running' state... PID: 26780 cluster.py:670 - DEBUG - existing nodes: {u'i-5b781f3a': , u'i-59781f38': } PID: 26780 cluster.py:673 - DEBUG - updating existing node i-59781f38 in self._nodes PID: 26780 cluster.py:673 - DEBUG - updating existing node i-5b781f3a in self._nodes PID: 26780 cluster.py:686 - DEBUG - returning self._nodes = [, ] PID: 26780 cluster.py:670 - DEBUG - existing nodes: {u'i-5b781f3a': , u'i-59781f38': } PID: 26780 cluster.py:673 - DEBUG - updating existing node i-59781f38 in self._nodes PID: 26780 cluster.py:673 - DEBUG - updating existing node i-5b781f3a in self._nodes PID: 26780 cluster.py:686 - DEBUG - returning self._nodes = [, ] PID: 26780 cluster.py:670 - DEBUG - existing nodes: {u'i-5b781f3a': , u'i-59781f38': } PID: 26780 cluster.py:673 - DEBUG - updating existing node i-59781f38 in self._nodes PID: 26780 cluster.py:673 - DEBUG - updating existing node i-5b781f3a in self._nodes PID: 26780 cluster.py:686 - DEBUG - returning self._nodes = [, ] PID: 26780 cluster.py:670 - DEBUG - existing nodes: {u'i-5b781f3a': , u'i-59781f38': } PID: 26780 cluster.py:673 - DEBUG - updating existing node i-59781f38 in self._nodes PID: 26780 cluster.py:673 - DEBUG - updating existing node i-5b781f3a in self._nodes PID: 26780 cluster.py:686 - DEBUG - returning self._nodes = [, ] PID: 26780 cluster.py:670 - DEBUG - existing nodes: {u'i-5b781f3a': , u'i-59781f38': } PID: 26780 cluster.py:673 - DEBUG - updating existing node i-59781f38 in self._nodes PID: 26780 cluster.py:673 - DEBUG - updating existing node i-5b781f3a in self._nodes PID: 26780 cluster.py:686 - DEBUG - returning self._nodes = [, ] PID: 26780 cluster.py:1081 - INFO - Waiting for SSH to come up on all nodes... PID: 26780 ssh.py:61 - DEBUG - loading private key /home/jpk37/Downloads/blastkey.pem PID: 26780 ssh.py:68 - DEBUG - specified key does not end in either rsa or dsa, trying both PID: 26780 ssh.py:145 - DEBUG - Using private key /home/jpk37/Downloads/blastkey.pem (rsa) PID: 26780 ssh.py:82 - DEBUG - connecting to host ec2-50-17-175-68.compute-1.amazonaws.com on port 22 as user root PID: 26780 ssh.py:61 - DEBUG - loading private key /home/jpk37/Downloads/blastkey.pem PID: 26780 ssh.py:68 - DEBUG - specified key does not end in either rsa or dsa, trying both PID: 26780 ssh.py:145 - DEBUG - Using private key /home/jpk37/Downloads/blastkey.pem (rsa) PID: 26780 ssh.py:82 - DEBUG - connecting to host ec2-50-16-43-183.compute-1.amazonaws.com on port 22 as user root PID: 26780 cluster.py:1252 - INFO - The master node is ec2-50-17-175-68.compute-1.amazonaws.com PID: 26780 cluster.py:1253 - INFO - Setting up the cluster... PID: 26780 cluster.py:1127 - INFO - Attaching volume vol-b2ff16d8 to master node on /dev/sdz ... PID: 26780 cluster.py:1129 - DEBUG - resp = attaching PID: 26780 cluster.py:670 - DEBUG - existing nodes: {u'i-5b781f3a': , u'i-59781f38': } PID: 26780 cluster.py:673 - DEBUG - updating existing node i-59781f38 in self._nodes PID: 26780 cluster.py:673 - DEBUG - updating existing node i-5b781f3a in self._nodes PID: 26780 cluster.py:686 - DEBUG - returning self._nodes = [, ] PID: 26780 clustersetup.py:96 - INFO - Configuring hostnames... PID: 26780 ssh.py:164 - DEBUG - creating sftp connection PID: 26780 threadpool.py:108 - DEBUG - unfinished_tasks = 2 PID: 26780 ssh.py:164 - DEBUG - creating sftp connection PID: 26780 clustersetup.py:262 - INFO - Mounting EBS volume vol-b2ff16d8 on /db... PID: 26780 ssh.py:405 - DEBUG - /dev/sda1 on / type ext3 (rw) PID: 26780 ssh.py:405 - DEBUG - proc on /proc type proc (rw,noexec,nosuid,nodev) PID: 26780 ssh.py:405 - DEBUG - none on /sys type sysfs (rw,noexec,nosuid,nodev) PID: 26780 ssh.py:405 - DEBUG - none on /sys/kernel/debug type debugfs (rw) PID: 26780 ssh.py:405 - DEBUG - none on /sys/kernel/security type securityfs (rw) PID: 26780 ssh.py:405 - DEBUG - devtmpfs on /dev type devtmpfs (rw,mode=0755) PID: 26780 ssh.py:405 - DEBUG - none on /dev/pts type devpts (rw,noexec,nosuid,gid=5,mode=0620) PID: 26780 ssh.py:405 - DEBUG - none on /dev/shm type tmpfs (rw,nosuid,nodev) PID: 26780 ssh.py:405 - DEBUG - none on /var/run type tmpfs (rw,nosuid,mode=0755) PID: 26780 ssh.py:405 - DEBUG - none on /var/lock type tmpfs (rw,noexec,nosuid,nodev) PID: 26780 ssh.py:405 - DEBUG - none on /lib/init/rw type tmpfs (rw,nosuid,mode=0755) PID: 26780 ssh.py:405 - DEBUG - /dev/sdb on /mnt type ext3 (rw) PID: 26780 ssh.py:405 - DEBUG - nfsd on /proc/fs/nfsd type nfsd (rw) PID: 26780 ssh.py:254 - DEBUG - new /etc/fstab after removing regex ( /db ) matches: # /etc/fstab: static file system information. # proc /proc proc nodev,noexec,nosuid 0 0 /dev/sda1 / ext3 defaults 0 0 /dev/sdb /mnt auto defaults,comment=cloudconfig 0 0 PID: 26780 clustersetup.py:144 - INFO - Creating cluster user: sgeadmin (uid: 1001, gid: 1001) PID: 26780 threadpool.py:108 - DEBUG - unfinished_tasks = 2 PID: 26780 clustersetup.py:161 - DEBUG - user sgeadmin does not exist, creating... PID: 26780 clustersetup.py:161 - DEBUG - user sgeadmin does not exist, creating... PID: 26780 clustersetup.py:186 - INFO - Configuring scratch space for user: sgeadmin PID: 26780 threadpool.py:108 - DEBUG - unfinished_tasks = 2 PID: 26780 clustersetup.py:195 - INFO - Configuring /etc/hosts on each node PID: 26780 threadpool.py:108 - DEBUG - unfinished_tasks = 2 PID: 26780 ssh.py:254 - DEBUG - new /etc/hosts after removing regex (master|node001) matches: 127.0.0.1 localhost # The following lines are desirable for IPv6 capable hosts ::1 ip6-localhost ip6-loopback fe00::0 ip6-localnet ff00::0 ip6-mcastprefix ff02::1 ip6-allnodes ff02::2 ip6-allrouters ff02::3 ip6-allhosts PID: 26780 ssh.py:254 - DEBUG - new /etc/hosts after removing regex (master|node001) matches: 127.0.0.1 localhost # The following lines are desirable for IPv6 capable hosts ::1 ip6-localhost ip6-loopback fe00::0 ip6-localnet ff00::0 ip6-mcastprefix ff02::1 ip6-allnodes ff02::2 ip6-allrouters ff02::3 ip6-allhosts PID: 26780 clustersetup.py:314 - INFO - Configuring NFS... PID: 26780 ssh.py:405 - DEBUG - Rather than invoking init scripts through /etc/init.d, use the service(8) PID: 26780 ssh.py:405 - DEBUG - utility, e.g. service portmap start PID: 26780 ssh.py:405 - DEBUG - PID: 26780 ssh.py:405 - DEBUG - Since the script you are attempting to invoke has been converted to an PID: 26780 ssh.py:405 - DEBUG - Upstart job, you may also use the start(8) utility, e.g. start portmap PID: 26780 ssh.py:405 - DEBUG - * Exporting directories for NFS kernel daemon... PID: 26780 ssh.py:405 - DEBUG - ...done. PID: 26780 ssh.py:405 - DEBUG - * Starting NFS kernel daemon PID: 26780 ssh.py:405 - DEBUG - ...done. PID: 26780 ssh.py:405 - DEBUG - /dev/sda1 on / type ext3 (rw) PID: 26780 ssh.py:405 - DEBUG - proc on /proc type proc (rw,noexec,nosuid,nodev) PID: 26780 ssh.py:405 - DEBUG - none on /sys type sysfs (rw,noexec,nosuid,nodev) PID: 26780 ssh.py:405 - DEBUG - none on /sys/kernel/debug type debugfs (rw) PID: 26780 ssh.py:405 - DEBUG - none on /sys/kernel/security type securityfs (rw) PID: 26780 ssh.py:405 - DEBUG - devtmpfs on /dev type devtmpfs (rw,mode=0755) PID: 26780 ssh.py:405 - DEBUG - none on /dev/pts type devpts (rw,noexec,nosuid,gid=5,mode=0620) PID: 26780 ssh.py:405 - DEBUG - none on /dev/shm type tmpfs (rw,nosuid,nodev) PID: 26780 ssh.py:405 - DEBUG - none on /var/run type tmpfs (rw,nosuid,mode=0755) PID: 26780 ssh.py:405 - DEBUG - none on /var/lock type tmpfs (rw,noexec,nosuid,nodev) PID: 26780 ssh.py:405 - DEBUG - none on /lib/init/rw type tmpfs (rw,nosuid,mode=0755) PID: 26780 ssh.py:405 - DEBUG - /dev/sdb on /mnt type ext3 (rw) PID: 26780 ssh.py:405 - DEBUG - nfsd on /proc/fs/nfsd type nfsd (rw) PID: 26780 clustersetup.py:304 - INFO - Mounting shares for node node001 PID: 26780 threadpool.py:108 - DEBUG - unfinished_tasks = 1 PID: 26780 ssh.py:405 - DEBUG - Rather than invoking init scripts through /etc/init.d, use the service(8) PID: 26780 ssh.py:405 - DEBUG - utility, e.g. service portmap start PID: 26780 ssh.py:405 - DEBUG - PID: 26780 ssh.py:405 - DEBUG - Since the script you are attempting to invoke has been converted to an PID: 26780 ssh.py:405 - DEBUG - Upstart job, you may also use the start(8) utility, e.g. start portmap PID: 26780 ssh.py:402 - DEBUG - command mount -t devpts none /dev/pts failed with status 32 PID: 26780 ssh.py:405 - DEBUG - mount: none already mounted or /dev/pts busy PID: 26780 ssh.py:405 - DEBUG - mount: according to mtab, none is already mounted on /dev/pts PID: 26780 ssh.py:254 - DEBUG - new /etc/fstab after removing regex ( /home | /opt/sge6 | /db ) matches: # /etc/fstab: static file system information. # proc /proc proc nodev,noexec,nosuid 0 0 /dev/sda1 / ext3 defaults 0 0 /dev/sdb /mnt auto defaults,comment=cloudconfig 0 0 PID: 26780 threadpool.py:108 - DEBUG - unfinished_tasks = 1 PID: 26780 utils.py:61 - INFO - _setup_nfs took 0.120 mins PID: 26780 clustersetup.py:207 - INFO - Configuring passwordless ssh for root PID: 26780 node.py:396 - DEBUG - adding auth_key_contents PID: 26780 clustersetup.py:215 - INFO - Configuring passwordless ssh for sgeadmin PID: 26780 node.py:396 - DEBUG - adding auth_key_contents PID: 26780 node.py:404 - DEBUG - adding conn_pubkey_contents PID: 26780 ssh.py:402 - DEBUG - command pkill -9 sge failed with status 1 PID: 26780 ssh.py:402 - DEBUG - command rm /etc/init.d/sge* failed with status 1 PID: 26780 ssh.py:405 - DEBUG - rm: cannot remove `/etc/init.d/sge*': No such file or directory PID: 26780 ssh.py:405 - DEBUG - lx24-amd64 PID: 26780 ssh.py:405 - DEBUG - [: 359: 11: unexpected operator PID: 26780 ssh.py:405 - DEBUG - [: 359: 11: unexpected operator PID: 26780 ssh.py:402 - DEBUG - command pkill -9 sge failed with status 1 PID: 26780 ssh.py:402 - DEBUG - command rm /etc/init.d/sge* failed with status 1 PID: 26780 ssh.py:405 - DEBUG - rm: cannot remove `/etc/init.d/sge*': No such file or directory PID: 26780 ssh.py:405 - DEBUG - lx24-amd64 PID: 26780 ssh.py:405 - DEBUG - [: 359: 11: unexpected operator PID: 26780 ssh.py:405 - DEBUG - [: 359: 11: unexpected operator PID: 26780 clustersetup.py:360 - INFO - Installing Sun Grid Engine... PID: 26780 ssh.py:405 - DEBUG - Reading configuration from file ./ec2_sge.conf PID: 26780 ssh.py:405 - DEBUG - [H[2JInstall log can be found in: /opt/sge6/default/common/install_logs/qmaster_install_master_2011-08-05_18:35:55.log PID: 26780 ssh.py:405 - DEBUG - Install log can be found in: /opt/sge6/default/common/install_logs/execd_install_master_2011-08-05_18:35:59.log PID: 26780 ssh.py:405 - DEBUG - [: 359: 11: unexpected operator PID: 26780 ssh.py:405 - DEBUG - [: 359: 11: unexpected operator PID: 26780 ssh.py:405 - DEBUG - [: 359: 11: unexpected operator PID: 26780 ssh.py:405 - DEBUG - [: 359: 11: unexpected operator PID: 26780 ssh.py:405 - DEBUG - root@master modified "all.q" in cluster queue list PID: 26780 ssh.py:405 - DEBUG - adminhost "node001" already exists PID: 26780 ssh.py:405 - DEBUG - submithost "node001" already exists PID: 26780 threadpool.py:108 - DEBUG - unfinished_tasks = 1 PID: 26780 ssh.py:405 - DEBUG - lx24-amd64 PID: 26780 ssh.py:405 - DEBUG - [: 359: 11: unexpected operator PID: 26780 ssh.py:405 - DEBUG - [: 359: 11: unexpected operator PID: 26780 ssh.py:405 - DEBUG - adminhost "node001" already exists PID: 26780 ssh.py:405 - DEBUG - submithost "node001" already exists PID: 26780 threadpool.py:108 - DEBUG - unfinished_tasks = 1 PID: 26780 threadpool.py:108 - DEBUG - unfinished_tasks = 1 PID: 26780 threadpool.py:108 - DEBUG - unfinished_tasks = 1 PID: 26780 threadpool.py:108 - DEBUG - unfinished_tasks = 1 PID: 26780 ssh.py:405 - DEBUG - Reading configuration from file ./ec2_sge.conf PID: 26780 ssh.py:405 - DEBUG - Install log can be found in: /opt/sge6/default/common/install_logs/execd_install_node001_2011-08-05_18:36:04.log PID: 26780 ssh.py:405 - DEBUG - [: 359: 11: unexpected operator PID: 26780 ssh.py:405 - DEBUG - [: 359: 11: unexpected operator PID: 26780 ssh.py:405 - DEBUG - [: 359: 11: unexpected operator PID: 26780 ssh.py:405 - DEBUG - [: 359: 11: unexpected operator PID: 26780 ssh.py:405 - DEBUG - 4 PID: 26780 ssh.py:405 - DEBUG - 4 PID: 26780 ssh.py:405 - DEBUG - root@master added "orte" to parallel environment list PID: 26780 ssh.py:405 - DEBUG - Unable to find "orte" in "pe_list" of "all.q" - Adding new element. PID: 26780 ssh.py:405 - DEBUG - PID: 26780 ssh.py:405 - DEBUG - root@master modified "all.q" in cluster queue list PID: 26780 threadpool.py:96 - INFO - Shutting down threads... PID: 26780 threadpool.py:108 - DEBUG - unfinished_tasks = 3 PID: 26780 cluster.py:1243 - INFO - The cluster has been started and configured. Login to the master node as root by running: $ starcluster sshmaster mycluster When you are finished using the cluster, run: $ starcluster stop mycluster to shutdown the cluster and stop paying for service. If this cluster uses EBS instances then the 'stop' command above will put all nodes into a 'stopped' state. The cluster may then be restarted at a later time, without losing data, by passing the -x option to the 'start' command. To completely terminate an EBS cluster: $ starcluster terminate mycluster NOTE: Terminating an EBS cluster will destroy all volumes backing the nodes. PID: 26780 utils.py:72 - INFO - Starting cluster took 3.421 mins PID: 26780 ssh.py:536 - DEBUG - __del__ called PID: 26780 ssh.py:536 - DEBUG - __del__ called PID: 26821 config.py:515 - DEBUG - Loading config PID: 26821 config.py:108 - DEBUG - Loading file: /home/jpk37/.starcluster/config PID: 26821 awsutils.py:54 - DEBUG - creating self._conn w/ connection_authenticator kwargs = {'path': '/', 'region': None, 'port': None, 'is_secure': True} PID: 26821 cluster.py:670 - DEBUG - existing nodes: {} PID: 26821 cluster.py:678 - DEBUG - adding node i-59781f38 to self._nodes list PID: 26821 cluster.py:678 - DEBUG - adding node i-5b781f3a to self._nodes list PID: 26821 cluster.py:686 - DEBUG - returning self._nodes = [, ] PID: 26821 node.py:822 - DEBUG - using system's ssh client PID: 26821 node.py:825 - DEBUG - ssh_cmd: ssh -i /home/jpk37/Downloads/blastkey.pem root@ec2-50-17-175-68.compute-1.amazonaws.com PID: 26893 config.py:515 - DEBUG - Loading config PID: 26893 config.py:108 - DEBUG - Loading file: /home/jpk37/.starcluster/config PID: 26893 awsutils.py:54 - DEBUG - creating self._conn w/ connection_authenticator kwargs = {'path': '/', 'region': None, 'port': None, 'is_secure': True} PID: 26893 cluster.py:670 - DEBUG - existing nodes: {} PID: 26893 cluster.py:678 - DEBUG - adding node i-59781f38 to self._nodes list PID: 26893 cluster.py:678 - DEBUG - adding node i-5b781f3a to self._nodes list PID: 26893 cluster.py:686 - DEBUG - returning self._nodes = [, ] PID: 26893 cluster.py:1152 - INFO - Rebooting cluster... PID: 26893 cluster.py:1156 - INFO - Sleeping for 20 seconds... PID: 26893 cluster.py:1048 - INFO - Waiting for cluster to come up... (updating every 30s) PID: 26893 cluster.py:670 - DEBUG - existing nodes: {u'i-5b781f3a': , u'i-59781f38': } PID: 26893 cluster.py:673 - DEBUG - updating existing node i-59781f38 in self._nodes PID: 26893 cluster.py:673 - DEBUG - updating existing node i-5b781f3a in self._nodes PID: 26893 cluster.py:686 - DEBUG - returning self._nodes = [, ] PID: 26893 cluster.py:1070 - INFO - Waiting for all nodes to be in a 'running' state... PID: 26893 cluster.py:1081 - INFO - Waiting for SSH to come up on all nodes... PID: 26893 ssh.py:61 - DEBUG - loading private key /home/jpk37/Downloads/blastkey.pem PID: 26893 ssh.py:68 - DEBUG - specified key does not end in either rsa or dsa, trying both PID: 26893 ssh.py:145 - DEBUG - Using private key /home/jpk37/Downloads/blastkey.pem (rsa) PID: 26893 ssh.py:82 - DEBUG - connecting to host ec2-50-17-175-68.compute-1.amazonaws.com on port 22 as user root PID: 26893 ssh.py:61 - DEBUG - loading private key /home/jpk37/Downloads/blastkey.pem PID: 26893 ssh.py:68 - DEBUG - specified key does not end in either rsa or dsa, trying both PID: 26893 ssh.py:145 - DEBUG - Using private key /home/jpk37/Downloads/blastkey.pem (rsa) PID: 26893 ssh.py:82 - DEBUG - connecting to host ec2-50-16-43-183.compute-1.amazonaws.com on port 22 as user root PID: 26893 cluster.py:1252 - INFO - The master node is ec2-50-17-175-68.compute-1.amazonaws.com PID: 26893 cluster.py:1253 - INFO - Setting up the cluster... PID: 26893 cluster.py:1120 - INFO - Volume vol-b2ff16d8 already attached to master...skipping PID: 26893 cluster.py:670 - DEBUG - existing nodes: {u'i-5b781f3a': , u'i-59781f38': } PID: 26893 cluster.py:673 - DEBUG - updating existing node i-59781f38 in self._nodes PID: 26893 cluster.py:673 - DEBUG - updating existing node i-5b781f3a in self._nodes PID: 26893 cluster.py:686 - DEBUG - returning self._nodes = [, ] PID: 26893 clustersetup.py:96 - INFO - Configuring hostnames... PID: 26893 ssh.py:164 - DEBUG - creating sftp connection PID: 26893 threadpool.py:108 - DEBUG - unfinished_tasks = 2 PID: 26893 ssh.py:164 - DEBUG - creating sftp connection PID: 26893 threadpool.py:108 - DEBUG - unfinished_tasks = 1 PID: 26893 threadpool.py:108 - DEBUG - unfinished_tasks = 1 PID: 26893 threadpool.py:108 - DEBUG - unfinished_tasks = 1 PID: 26893 threadpool.py:108 - DEBUG - unfinished_tasks = 1 PID: 26893 threadpool.py:108 - DEBUG - unfinished_tasks = 1 PID: 26893 threadpool.py:108 - DEBUG - unfinished_tasks = 1 PID: 26893 threadpool.py:108 - DEBUG - unfinished_tasks = 1 PID: 26893 threadpool.py:108 - DEBUG - unfinished_tasks = 1 PID: 26893 threadpool.py:108 - DEBUG - unfinished_tasks = 1 PID: 26893 threadpool.py:108 - DEBUG - unfinished_tasks = 1 PID: 26893 threadpool.py:108 - DEBUG - unfinished_tasks = 1 PID: 26893 threadpool.py:108 - DEBUG - unfinished_tasks = 1 PID: 26893 threadpool.py:108 - DEBUG - unfinished_tasks = 1 PID: 26893 threadpool.py:108 - DEBUG - unfinished_tasks = 1 PID: 26893 threadpool.py:108 - DEBUG - unfinished_tasks = 1 PID: 26893 threadpool.py:108 - DEBUG - unfinished_tasks = 1 PID: 26893 threadpool.py:108 - DEBUG - unfinished_tasks = 1 PID: 26893 threadpool.py:108 - DEBUG - unfinished_tasks = 1 PID: 26893 threadpool.py:108 - DEBUG - unfinished_tasks = 1 PID: 26893 threadpool.py:96 - INFO - Shutting down threads... PID: 26893 threadpool.py:108 - DEBUG - unfinished_tasks = 3 PID: 26893 cli.py:166 - DEBUG - error occured in job (id=master): [Errno 104] Connection reset by peer Traceback (most recent call last): File "/usr/lib/python2.6/site-packages/StarCluster-0.92rc2-py2.6.egg/starcluster/threadpool.py", line 32, in run job.run() File "/usr/lib/python2.6/site-packages/StarCluster-0.92rc2-py2.6.egg/starcluster/threadpool.py", line 55, in run r = self.method(*self.args) File "/usr/lib/python2.6/site-packages/StarCluster-0.92rc2-py2.6.egg/starcluster/node.py", line 622, in set_hostname hostname_file = self.ssh.remote_file("/etc/hostname", "w") File "/usr/lib/python2.6/site-packages/StarCluster-0.92rc2-py2.6.egg/starcluster/ssh.py", line 266, in remote_file rfile = self.sftp.open(file, mode) File "/usr/lib/python2.6/site-packages/StarCluster-0.92rc2-py2.6.egg/starcluster/ssh.py", line 165, in sftp self._sftp = paramiko.SFTPClient.from_transport(self.transport) File "/usr/lib/python2.6/site-packages/paramiko-1.7.7.1-py2.6.egg/paramiko/sftp_client.py", line 102, in from_transport chan = t.open_session() File "/usr/lib/python2.6/site-packages/paramiko-1.7.7.1-py2.6.egg/paramiko/transport.py", line 658, in open_session return self.open_channel('session') File "/usr/lib/python2.6/site-packages/paramiko-1.7.7.1-py2.6.egg/paramiko/transport.py", line 746, in open_channel raise e error: [Errno 104] Connection reset by peer PID: 26893 cli.py:124 - ERROR - Oops! Looks like you've found a bug in StarCluster PID: 26893 cli.py:125 - ERROR - Debug file written to: /tmp/starcluster-debug-jpk37.log PID: 26893 cli.py:126 - ERROR - Look for lines starting with PID: 26893 PID: 26893 cli.py:127 - ERROR - Please submit this file, minus any private information, PID: 26893 cli.py:128 - ERROR - to starcluster@mit.edu PID: 26893 ssh.py:536 - DEBUG - __del__ called PID: 26893 ssh.py:536 - DEBUG - __del__ called