Re: [Starcluster] Spot instances

From: Justin Riley <no email>
Date: Tue, 30 Mar 2010 09:41:57 -0400

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

Hi Joonas,

It appears that your cluster template high-cpu-xl has CLUSTER_SIZE=1
given that it extends the "small" cluster template in your config.

I believe your issue has to do with the cluster size in the template not
being the same as the number of instances launched in the _at_sc-cluster1
security group. One way to fix this would be to put CLUSTER_SIZE=4 in
your "high-cpu-xl" cluster template.

A quick fix would be to run:

$ starcluster start -x -s 4 high-cpu-xl cluster1

Let me know if that works for you or not.

Thanks,

~Justin



On 03/30/2010 09:00 AM, Joonas Paalasmaa wrote:
> Hi,
>
> I tried to set up a cluster as you suggested.
>
> First, I started four instances with the commands below.
>
> ec2-add-group "_at_sc-cluster1" -d "StarCluster cluster1"
> ec2-authorize "_at_sc-cluster1" -P tcp -p 22 -s 0.0.0.0/0
> ec2-request-spot-instances ami-a19e71c8 --group "_at_sc-masters" --group
> "_at_sc-cluster1" -m -z us-east-1a -p 1.0 -t c1.xlarge -n 1 -r one-time
> ec2-request-spot-instances ami-a19e71c8 --group "_at_sc-cluster1" -m -z
> us-east-1a -p 1.0 -t c1.xlarge -n 3 -r one-time
> starcluster start -x high-cpu-xl cluster1
>
> Once the instances were running, I ran the command "starcluster start
> -x high-cpu-xl cluster1". However, it did not go beyond the text
> "Waiting for cluster to start...".
>
> What could I be doing wrong?
>
> - Joonas
>
>
> Here's my config for the cluster:
>
>
> [cluster small]
> KEYNAME = pstam-keypair
>
> CLUSTER_SIZE = 1
>
> CLUSTER_USER = jpaalasm
> CLUSTER_SHELL = bash
>
> NODE_IMAGE_ID = ami-8f9e71e6
> NODE_INSTANCE_TYPE = m1.small
> AVAILABILITY_ZONE = us-east-1a
> VOLUMES = shhs
>
>
> [cluster high-cpu-xl]
> EXTENDS = small
> NODE_IMAGE_ID = ami-a19e71c8
> NODE_INSTANCE_TYPE = c1.xlarge
>
>
>
>
> On Wed, Mar 24, 2010 at 5:20 PM, Justin Riley <jtriley_at_mit.edu> wrote:
> Hi Joonas,
>
> The development code in github for the next version can use spot
> instances provided that you launch them manually.
>
> The next version of starcluster makes this possible because of the fact
> that it relies on creating/using security groups for accounting.
>
> By default StarCluster creates a security group for all starcluster
> master nodes called _at_sc-masters. When you launch a cluster, say called
> 'mycluster', StarCluster creates a new security group called
> _at_sc-mycluster. It then runs the master and assigns it to both
> _at_sc-masters and @sc-mycluster security groups and then runs the rest of
> the nodes, assigning them only to _at_sc-mycluster.
>
> So, with the new version, you can manually create a _at_sc-mycluster
> security group, launch your instances by adding the master to both the
> _at_sc-masters AND @sc-mycluster security groups and then launching the
> rest of the nodes in the _at_sc-mycluster security group only. Starcluster
> will then be able to launch using these instances, provided they're in a
> pending/running state, by running the following:
>
> $ starcluster start -x smallcluster mycluster
>
> This will then skip requesting new instances and wait for all existing
> instances in the _at_sc-mycluster group to come up and it will continue on
> normally from there.
>
> Does this make sense? I will document this in the next version. If
> you're interested in using the development version you will find docs
> for the new config in the github code (starcluster/docs/sphinx/*.rst).
>
> Hope that helps,
>
> ~Justin
>
>
>
>
> On 03/24/2010 04:33 AM, Joonas Paalasmaa wrote:
>>>> Hello,
>>>>
>>>> Are there any plans to support EC2 spot instances in starcluster?
>>>> Could it, for example, be implemented so that you specify the instance
>>>> IDs of of the launched StarCluster spot instances and starcluster
>>>> takes care of configuring a cluster from them?
>>>>
>>>> Thanks,
>>>> Joonas Paalasmaa
>>>> _______________________________________________
>>>> Starcluster mailing list
>>>> Starcluster_at_mit.edu
>>>> http://mailman.mit.edu/mailman/listinfo/starcluster
>
>>
> _______________________________________________
> Starcluster mailing list
> Starcluster_at_mit.edu
> http://mailman.mit.edu/mailman/listinfo/starcluster

-----BEGIN PGP SIGNATURE-----
Version: GnuPG v2.0.14 (GNU/Linux)
Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/

iEYEARECAAYFAkux/6UACgkQ4llAkMfDcrl0ogCcCj5IS+TUXGpPxH5nUEUyB46k
NIAAn2i1Exh3tTuqsOc6/JT8p4A2Yh5x
=56IO
-----END PGP SIGNATURE-----
Received on Tue Mar 30 2010 - 09:41:59 EDT