Re: [Starcluster] failed cluster / detached drive

From: Dan Yamins <no email>
Date: Fri, 30 Apr 2010 20:33:07 -0400

OK I just started a new cluster. I saw some weird things during the
start-up:

[snip part that was normal]
>>> Setting up the cluster...
>>> Mounting EBS volume vol-c3d927aa on /home...
>>> Using private key /Users/dyamins/amazon/id_rsa-gsg-keypair (rsa)
>>> Creating cluster user: gotdata
>>> Using private key /Users/dyamins/amazon/id_rsa-gsg-keypair (rsa)
>>> Configuring scratch space for user: gotdata
>>> Configuring /etc/hosts on each node
>>> Configuring NFS...
ssh.py:245 - ERROR - command mount -t devpts none /dev/pts failed with
status 32
ssh.py:245 - ERROR - command mount -t devpts none /dev/pts failed with
status 32
>>> Configuring passwordless ssh for root
ssh.py:245 - ERROR - command rm /root/.ssh/id_rsa* failed with status 1
>>> Configuring passwordless ssh for user: gotdata
>>> Using existing RSA ssh keys found for user: gotdata
>>> Installing Sun Grid Engine...
>>> Done Configuring Sun Grid Engine
>>> Running plugin govlovePlugin

The resulting cluster seems to be functioning normally (e.g. the volume is
mounted and I can log in to the nodes as both root and CLUSTER_USER), and my
SGE jobs seem to be working. I'll let you know if anything weird occurs.
Also I'd like to understand the error on line 245 of ssh.py above.

Thanks!

Dan



On Fri, Apr 30, 2010 at 8:20 PM, Dan Yamins <dyamins_at_gmail.com> wrote:

> Justin, I just had a strange situation where suddenly my cluster failed.
> here were the symptoms:
>
> 1) all my active ssh terminals timed out
> 2) i couldn't log back in as the CLUSTER_USER (I got the "permission denied
> (public key)" error -- though I could ssh in as root
> 3) the mounted EBS volume appears to have disappeared -- e.g. when I tried
> to cd to it from /root, it was reported as not existing.
> 4) the SGE "qstat" command failed to be recognized. (e.g. when i run
> "qstat -xml" as root I got an error in finding the qstat command.)
>
> It seems like my EBS drive might have detached ... but lots of things could
> have happened. Any thoughts?
>
> Anyway, I killed the cluster as i didn't want o keep paying for it. I'm
> starting another one now, and will let you know what the result it. If it
> happens again I'll keep the cluster up and let you know right away.
>
> Dan
>
>
Received on Fri Apr 30 2010 - 20:33:08 EDT