Re: [Starcluster] failed installing Sun Grid Engine...

From: Dan Yamins <no email>
Date: Wed, 17 Mar 2010 16:08:02 -0400

Sorry to be dumb ... but I still don't understand why that's the "fastest
processor on Amazon" ... I thought machine resources were a function of
instance type, not AMI -- once you determined the platform. Won't the
processor used for that fedora AMI be whatever is associated with the
instance type you choose? Can't you choose any of the 64-bit instance
types listed here: http://aws.amazon.com/ec2/instance-types
<http://aws.amazon.com/ec2/instance-types/>? I would think that the
"fastest processor on Amazon" would be
*High-CPU Extra Large Instance* c1.xlarge, which has 20 times the compute
power of the m1.small instance.

Won' t Starcluster's 64-bit AMI work just fine on c1.xlarge or any of the
other 64-bit-required instances? If so, does that not deliver the
performance you were seeking?






On Wed, Mar 17, 2010 at 3:14 PM, chuan gao <aggie.gao_at_gmail.com> wrote:

> That is the amazon one with fedora installed. it is 64 bit. ami-86db39ef .
> not surprising that it didn't work. :D
> Sorry guys.
>
>
> On Wed, Mar 17, 2010 at 2:38 PM, Dan Yamins <dyamins_at_gmail.com> wrote:
>
>> Which EC2 processors can you not use with either the 64-bit or 32 -bit
>> starcluster AMI?
>>
>>
>>
>>
>> On Wed, Mar 17, 2010 at 2:04 PM, chuan gao <aggie.gao_at_gmail.com> wrote:
>>
>>> Hi Justin,
>>> Sorry for the confusion, I used the wrong AMI IDs. stupid mistake that
>>> everybody can avoid by following the instructions. I saw it on the webpage
>>> but still went ahead and changed that because I want to use the fastest
>>> processor on amazon. :(
>>> thank you very much for the software, truly nice work.
>>> Chuan
>>>
>>>
>>>
>>> On Wed, Mar 17, 2010 at 1:51 PM, Justin Riley <jtriley_at_mit.edu> wrote:
>>>
>>>> -----BEGIN PGP SIGNED MESSAGE-----
>>>> Hash: SHA1
>>>>
>>>> Chuan,
>>>>
>>>> Responding to your other question concerning GSL and gotoBLAS:
>>>>
>>>> Installing these locally into a EBS volume is a perfectly reasonable
>>>> solution. In fact, this is how I envisioned most people storing their
>>>> software/data on StarCluster.
>>>>
>>>> The other way to do this is to launch an instance, install the software
>>>> globally (using apt-get and/or source packages), and then rebundle the
>>>> AMI. You could then tell starcluster to use this new AMI in your config
>>>> file.
>>>>
>>>> I will add GSL to the next version of StarCluster's AMI. goto blas must
>>>> be custom compiled. I'll have a look at this and if it's not too much
>>>> work I'll consider adding it in.
>>>>
>>>> Hope that helps,
>>>>
>>>> ~Justin
>>>>
>>>> On 03/17/2010 01:35 PM, chuan gao wrote:
>>>> > Hi Justin and Dan,
>>>> > Sorry for all the trouble, it's all my fault, it turned out that I
>>>> used
>>>> > the wrong AMI type. it worked like a charm after I changed that. I do
>>>> > have another question though, I need to use GSL and gotoBLAS for my
>>>> > computation, which I didn't find it installed on the cluster, I am
>>>> > thinking about compiling these locally on my EBS volume and include
>>>> the
>>>> > library in my code. is there any other better way?
>>>> > Thank you all guys for the help!
>>>> > Chuan
>>>> >
>>>> >
>>>> >
>>>> > On Wed, Mar 17, 2010 at 10:51 AM, Justin Riley <jtriley_at_mit.edu
>>>> > <mailto:jtriley_at_mit.edu>> wrote:
>>>> >
>>>> > Hi Chuan,
>>>> >
>>>> > Have you checked whether or not the EBS volume ever gets to an
>>>> > 'attached'
>>>> > state using either ElasticFox or the AWS web console?
>>>> >
>>>> > Also, you can run starcluster in debug mode by passing -d option:
>>>> >
>>>> > $ starcluster -d -s
>>>> >
>>>> > This will do the same thing as -s only with debug output enabled.
>>>> >
>>>> > Would you mind sending me that output? Please be careful about any
>>>> > sensitive
>>>> > data in the output if you cc this list.
>>>> >
>>>> > Thanks,
>>>> >
>>>> > ~Justin
>>>> >
>>>> >
>>>> > On Wednesday 17 March 2010 1:18:25 am chuan gao wrote:
>>>> > > Hi Justin and Dan,
>>>> > >
>>>> > > Sorry for the late reply cuz I was tied up on a few things.
>>>> > >
>>>> > > To Justin: I tried it without mount EBS and I still have that
>>>> > problem. I
>>>> > > believe it's related to this though cuz it happened at the last
>>>> > step of
>>>> > > configuring the cluster(all machines have been started by that
>>>> time).
>>>> > >
>>>> > > To Dan: I did tried reinstall paramiko (not in debug mode though
>>>> > since I
>>>> > > didn't find out how to do so,) and the system said installed
>>>> > sucessfully. I
>>>> > > also tried starting the cluster outside the directory without
>>>> success.
>>>> > >
>>>> > > I also tried the same thing on another machine which has ubuntu
>>>> on
>>>> > it and I
>>>> > > got the same error. I am not the administrator to that machine
>>>> but
>>>> > I guess
>>>> > > it should be ubuntu 9.04.
>>>> > >
>>>> > > any ideas? thank you guys very much for handling this.
>>>> > >
>>>> > > Chuan
>>>> > >
>>>> > > On Tue, Mar 16, 2010 at 7:13 PM, Justin Riley <jtriley_at_mit.edu
>>>> > <mailto:jtriley_at_mit.edu>> wrote:
>>>> > > > Hi Chuan,
>>>> > > >
>>>> > > > Sorry to hear you're having issues with StarCluster. I just
>>>> got
>>>> > back in
>>>> > > > town,
>>>> > > > sorry for the delayed response.
>>>> > > >
>>>> > > > Dan, as always, thanks for responding :D
>>>> > > >
>>>> > > > Could you try launching a cluster without using the EBS volume
>>>> > and let me
>>>> > > > know
>>>> > > > if it succeeds? Nicolas Pinto had a similar issue with
>>>> > StarCluster and I
>>>> > > > believe it only happened for him when using EBS.
>>>> > > >
>>>> > > > Thanks,
>>>> > > >
>>>> > > > ~Justin
>>>> > > >
>>>> > > > On Tuesday 16 March 2010 2:15:12 pm chuan gao wrote:
>>>> > > > > I am using the standard AMI
>>>> > > > > the OS is ubuntu 9.10
>>>> > > > > yes. I can ssh into AMI with no problem. actually, the error
>>>> > occured at
>>>> > > >
>>>> > > > the
>>>> > > >
>>>> > > > > last step when it is configuring NFS. I checked the AMI on
>>>> > amazon and
>>>> > > >
>>>> > > > they
>>>> > > >
>>>> > > > > have been started. and my permanent volume has been mounted.
>>>> > > > > concerning the error,
>>>> > > > > build/bdist.linux-i686/egg/paramiko/sftp_client.py
>>>> > > > > I looked into directory
>>>> > StarCluster-0.90.1/build/bdist.linux-i686 and
>>>> > > > > nothing is there. should there be
>>>> egg/paramiko/sftp_client.py ?
>>>> > > > > Thanks
>>>> > > > >
>>>> > > > > On Tue, Mar 16, 2010 at 1:46 PM, Dan Yamins <
>>>> dyamins_at_gmail.com
>>>> > <mailto:dyamins_at_gmail.com>> wrote:
>>>> > > > > > Justin is on this list, so I'm sure he'll respond soon.
>>>> > > > > >
>>>> > > > > > Are you using the standard AMI, or one you built yourself?
>>>> > > > > >
>>>> > > > > > Also, what is your operating system and version?
>>>> > > > > >
>>>> > > > > > Also, can you ssh into instances of the AMI, independently
>>>> of
>>>> > > > > > starcluster?
>>>> > > > > >
>>>> > > > > > On Tue, Mar 16, 2010 at 1:17 PM, chuan gao
>>>> > <aggie.gao_at_gmail.com <mailto:aggie.gao_at_gmail.com>>
>>>> > > >
>>>> > > > wrote:
>>>> > > > > >> Thanks Dan, I am pretty sure that I got paramiko
>>>> installed
>>>> > > > > >> correctly. I'll work on it a bit more.
>>>> > > > > >> will Justin have a chance to look at this email list and
>>>> try to
>>>> > > > > >> point out what could be the problem?
>>>> > > > > >>
>>>> > > > > >> On Tue, Mar 16, 2010 at 11:21 AM, Dan Yamins
>>>> > <dyamins_at_gmail.com <mailto:dyamins_at_gmail.com>>
>>>> > > >
>>>> > > > wrote:
>>>> > > > > >>> Hm. I'm not sure what the problem is (your version is
>>>> fine).
>>>> > > > > >>> Something is clearly wrong with either your paramiko
>>>> > installation
>>>> > > > > >>> or the way that starcluster is using it.
>>>> > > > > >>>
>>>> > > > > >>> Have you tried testing paramiko outside the context of
>>>> > starcluster?
>>>> > > > > >>> Try testing it for normal ssh usage. If that fails,
>>>> then
>>>> > you'll
>>>> > > > > >>> probably have identified the problem. (I don't know if
>>>> > you just
>>>> > > > > >>> installed it, perhaps reinstallation would then help.)
>>>> > > > > >>>
>>>> > > > > >>> If paramiko works normally, then perhaps there's an
>>>> > argument that
>>>> > > > > >>> is being passed on line 109 of starcluster/ssh.py that
>>>> is
>>>> > supposed
>>>> > > > > >>> to represent an existing file object, that somehow isn't
>>>> being
>>>> > > > > >>> properly created (probably earlier in the ssh cycle).
>>>> In this
>>>> > > > > >>> case, Justin
>>>> > > >
>>>> > > > (the
>>>> > > >
>>>> > > > > >>> creator of starcluster) should probably be the one to
>>>> > address your
>>>> > > > > >>> problem.
>>>> > > > > >>>
>>>> > > > > >>> Dan
>>>> > > > > >>>
>>>> > > > > >>> On Tue, Mar 16, 2010 at 10:57 AM, chuan gao
>>>> > <aggie.gao_at_gmail.com <mailto:aggie.gao_at_gmail.com>>
>>>> > > >
>>>> > > > wrote:
>>>> > > > > >>>> paramiko-1.7.6-py2.6.egg
>>>> > > > > >>>> Thanks for replying.
>>>> > > > > >>>>
>>>> > > > > >>>> On Tue, Mar 16, 2010 at 9:02 AM, Dan Yamins
>>>> > <dyamins_at_gmail.com <mailto:dyamins_at_gmail.com>>
>>>> > > >
>>>> > > > wrote:
>>>> > > > > >>>>> On Tue, Mar 16, 2010 at 1:05 AM, chuan gao
>>>> > <aggie.gao_at_gmail.com <mailto:aggie.gao_at_gmail.com>
>>>> > > > >
>>>> > > > >wrote:
>>>> > > > > >>>>>> Here is the error message:
>>>> > > > > >>>>>> >>> Installing Sun Grid Engine...
>>>> > > > > >>>>>>
>>>> > > > > >>>>>> [SNIP]
>>>> > > > > >>>>>>
>>>> > > > > >>>>>>
>>>> > > > > >>>>>> File
>>>> > "build/bdist.linux-i686/egg/paramiko/sftp_client.py",
>>>> > > > > >>>>>> line 675, in _read_response
>>>> > > > > >>>>>> File
>>>> > "build/bdist.linux-i686/egg/paramiko/sftp_client.py",
>>>> > > > > >>>>>> line 701, in _convert_status
>>>> > > > > >>>>>> IOError: [Errno 2] No such file
>>>> > > > > >>>>>
>>>> > > > > >>>>> What version of paramiko do you have installed?
>>>> > > > > >>>>>
>>>> > > > > >>>>>
>>>> > > > > >>>>> _______________________________________________
>>>> > > > > >>>>> Starcluster mailing list
>>>> > > > > >>>>> Starcluster_at_mit.edu <mailto:Starcluster_at_mit.edu>
>>>> > > > > >>>>> http://mailman.mit.edu/mailman/listinfo/starcluster
>>>> > >
>>>> >
>>>> >
>>>>
>>>> -----BEGIN PGP SIGNATURE-----
>>>> Version: GnuPG v2.0.14 (GNU/Linux)
>>>> Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/
>>>>
>>>> iEYEARECAAYFAkuhFqwACgkQ4llAkMfDcrkcEwCgi0EdVkC1/5Ne578BNYkQYMpO
>>>> +fYAn13tNTBQ2M2P29hgJoltz4nwj2DU
>>>> =Vuxo
>>>> -----END PGP SIGNATURE-----
>>>>
>>>
>>>
>>> _______________________________________________
>>> Starcluster mailing list
>>> Starcluster_at_mit.edu
>>> http://mailman.mit.edu/mailman/listinfo/starcluster
>>>
>>>
>>
>
Received on Wed Mar 17 2010 - 16:08:04 EDT