Re: Large cluster (125 nodes) launch failure

From: Joseph <Kyeong>
Date: Tue, 15 Mar 2011 23:23:26 +0000

Justin,

By the way, I forgot to mention that I am certainly interested in
profiling my large setup.
Just let me know how to do once you have done the next release.

Joseph


On Tue, Mar 15, 2011 at 10:36 PM, Kyeong Soo (Joseph) Kim
<kyeongsoo.kim_at_gmail.com> wrote:
> Justin,
>
> Thanks for the prompt and detailed response!
>
> BTW it's not just a matter of delay.
> I just responded to Jeff's e-mail regarding the partial installation
> of SGE; like Jeff, I experienced that 21 out of 50 nodes are not
> properly installed during the launching procedure and qhost shows only
> 29 execution nodes.
>
> Now I am a bit at a loss; because I believed that all 50 nodes were
> successfully running as SGE execution nodes, I submitted a huge array
> jobs (total 1000). They are running on 29 nodes, but 21 others are
> sitting idle.
>
> What should I do with those 21 nodes?
> Should I terminate the cluster (together with already running jobs)
> and start with an even smaller configuration?
>
> Regards,
> Joseph
>
>
> On Tue, Mar 15, 2011 at 10:29 PM, Justin Riley <jtriley_at_mit.edu> wrote:
>> -----BEGIN PGP SIGNED MESSAGE-----
>> Hash: SHA1
>>
>> Hi Joseph,
>>
>> Have a look here for Adam Kraut's account of launching an 80 node
>> StarCluster:
>>
>> http://mailman.mit.edu/pipermail/starcluster/2010-December/000552.html
>>
>> It's *definitely* a matter of a huge delay involved in setting up a
>> large number of nodes so be patient. This is because StarCluster was
>> originally intended for 10-20 nodes and uses a fairly naive approach of
>> waiting for all nodes to come up (including ssh) and then setting up
>> each node one-by-one. I have several ideas on how to substantially speed
>> this process up in the future but for now I'm concentrating on getting a
>> release out within the next couple weeks.
>>
>> However, after the release I'd be interested in profiling StarCluster
>> during such a large run and seeing where the slowness comes from in
>> detail. I'm definitely interested in getting StarCluster to a point
>> where it can launch large clusters in a reasonable amount of time. As it
>> stands now I need to drastically up my instance limit in order to test
>> and debug this properly. Would you be interested in profiling one of
>> your large runs and sending me the output some time after the next release?
>>
>> ~Justin
>>
>>
>>
>> On 03/15/2011 06:03 PM, Kyeong Soo (Joseph) Kim wrote:
>>> Hi Austin,
>>>
>>> Yes, I requested to increase the limit to 300 (got a confirmation, of
>>> course) and now successfully running a 50-node cluster (it took 32
>>> mins BTW).
>>> I wonder now if it simply is a matter of huge delay involved with
>>> setting up such a large number nodes.
>>>
>>> Regards,
>>> Joseph
>>>
>>>
>>> On Tue, Mar 15, 2011 at 9:53 PM, Austin Godber <godber_at_uberhip.com> wrote:
>>>> Does it work at 20 and fail at 21?  I think Amazon still has a 20 AMIs
>>>> limit, which you can request that they raise.  Have you done that?
>>>>
>>>> http://aws.amazon.com/ec2/faqs/#How_many_instances_can_I_run_in_Amazon_EC2
>>>>
>>>> Austin
>>>>
>>>> On 03/15/2011 05:29 PM, Kyeong Soo (Joseph) Kim wrote:
>>>>> Hi Justin and All,
>>>>>
>>>>> This is to report a failure in launching a large cluster with 125
>>>>> nodes (c1.xlarge).
>>>>>
>>>>> I tried to launch the said cluster two times but starcluster hung (for
>>>>> more than hours) at the following steps:
>>>>>
>>>>> .....
>>>>>
>>>>>>>> Launching node121 (ami: ami-2857a641, type: c1.xlarge)
>>>>>>>> Launching node122 (ami: ami-2857a641, type: c1.xlarge)
>>>>>>>> Launching node123 (ami: ami-2857a641, type: c1.xlarge)
>>>>>>>> Launching node124 (ami: ami-2857a641, type: c1.xlarge)
>>>>>>>> Creating security group _at_sc-hnrlcluster...
>>>>> Reservation:r-7c264911
>>>>>>>> Waiting for cluster to come up... (updating every 30s)
>>>>>>>> Waiting for all nodes to be in a 'running' state...
>>>>> 125/125 |||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
>>>>> 100%
>>>>>>>> Waiting for SSH to come up on all nodes...
>>>>> 125/125 |||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
>>>>> 100%
>>>>>>>> The master node is ec2-75-101-230-197.compute-1.amazonaws.com
>>>>>>>> Setting up the cluster...
>>>>>>>> Attaching volume vol-467ecc2e to master node on /dev/sdz ...
>>>>>>>> Configuring hostnames...
>>>>>>>> Mounting EBS volume vol-467ecc2e on /home...
>>>>>>>> Creating cluster user: kks (uid: 1001, gid: 1001)
>>>>>>>> Configuring scratch space for user: kks
>>>>>>>> Configuring /etc/hosts on each node
>>>>>
>>>>> I have succeeded with the configuration up to 15 nodes so far.
>>>>>
>>>>> Any idea?
>>>>>
>>>>> With Regards,
>>>>> Joseph
>>>>> --
>>>>> Kyeong Soo (Joseph) Kim, Ph.D.
>>>>> Senior Lecturer in Networking
>>>>> Room 112, Digital Technium
>>>>> Multidisciplinary Nanotechnology Centre, College of Engineering
>>>>> Swansea University, Singleton Park, Swansea SA2 8PP, Wales UK
>>>>> TEL: +44 (0)1792 602024
>>>>> EMAIL: k.s.kim_at_swansea.ac.uk
>>>>> HOME: http://iat-hnrl.swan.ac.uk/ (group)
>>>>>              http://iat-hnrl.swan.ac.uk/~kks/ (personal)
>>>>>
>>>>> _______________________________________________
>>>>> StarCluster mailing list
>>>>> StarCluster_at_mit.edu
>>>>> http://mailman.mit.edu/mailman/listinfo/starcluster
>>>>
>>>> _______________________________________________
>>>> StarCluster mailing list
>>>> StarCluster_at_mit.edu
>>>> http://mailman.mit.edu/mailman/listinfo/starcluster
>>>>
>>>
>>> _______________________________________________
>>> StarCluster mailing list
>>> StarCluster_at_mit.edu
>>> http://mailman.mit.edu/mailman/listinfo/starcluster
>>
>> -----BEGIN PGP SIGNATURE-----
>> Version: GnuPG v2.0.17 (GNU/Linux)
>> Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/
>>
>> iEYEARECAAYFAk1/6F0ACgkQ4llAkMfDcrnDXQCfap8Q42lXaahHbQu3No3pp5Nz
>> x3oAn2Rl50ImIxnWBxf9di1vaFgLm1AO
>> =wUaZ
>> -----END PGP SIGNATURE-----
>> _______________________________________________
>> StarCluster mailing list
>> StarCluster_at_mit.edu
>> http://mailman.mit.edu/mailman/listinfo/starcluster
>>
>
Received on Tue Mar 15 2011 - 19:23:27 EDT