Re: Star cluster not creating instances on Amazon's East Region

From: sebastian rooks <no email>
Date: Wed, 1 Aug 2012 16:46:09 +0200

On Wed, Aug 1, 2012 at 7:39 AM, Justin Riley <jtriley_at_mit.edu> wrote:
> There would be some impact in terms of network latency given that, in
> theory, nodes *might* be assigned to different availability zones which
> means different data centers and thus higher network latency. This could
> affect the performance of things like NFS-shares on the cluster as well
> as any applications that require a lot of network communication between
> nodes but it's hard to quantify the exact performance hit in this case
> due to the randomness of where instances are allocated in the data
> centers.
>
> So depending on whether Amazon spreads the instances across multiple
> zones or not there could be a trade-off between network performance vs
> launch reliability.
>
> With that said I'm leaning towards changing the default in favor of
> launch reliability given that others have occassionally encountered
> similar errors from AWS concerning oversubscribed zones. Not to mention
> the network latency between nodes in general is suboptimal even when
> launching in the same zone given that nodes are often separated by many
> routers even within the same data center. My guess is that if possible,
> Amazon will choose the same zone for instances by default but I need to
> test this to make sure.
>
> If we made this switch I would also add a flag that enables the old
> behavior of trying to launch all nodes in the same availability zone for
> those that care and understand the risks. This way it's an optimization
> flag instead of a 'deoptimization' flag and the default behavior is more
> likely to 'just work' for users. I could also add an optional setting to
> the config to set the 'zone launch strategy' by default for different
> cluster configs.
>
> What do folks think?

When launching a cluster of cluster compute instances I certainly
expect them to be in the same availability zone.
I actually see the possibility of having nodes in several availability
zones as an optimisation for use cases that can afford the network
latency...

Regards,

  S

>
> ~Justin
>
> On Wed, Aug 01, 2012 at 10:00:42AM +0530, Vipin Shankar wrote:
>> Thanks Justin for the detailed response.
>> Would there be any performance impact if we nodes are created across AZs
>> (with the optional flag implementation) ?
>>
>> -Vipin
>>
>> -----Original Message-----
>> From: Justin Riley
>> Sent: Tuesday, July 31, 2012 11:16 PM
>> To: Erik Gafni
>> Cc: Ramit Bhardwaj ; starcluster_at_mit.edu ; Vipin Shankar
>> Subject: Re: [StarCluster] Star cluster not creating instances on Amazon's
>> East Region
>>
>> -----BEGIN PGP SIGNED MESSAGE-----
>> Hash: SHA1
>>
>> Hi Ramit,
>>
>> That error comes directly from AWS.
>>
>> In the case that you don't specify an availability zone in your
>> cluster config StarCluster first launches the master node, then
>> determines which zone Amazon chose *within* the us-east-1 region for
>> the master, and then launches the rest of the nodes using that zone in
>> order to improve cluster locality. Sometimes if the zone is
>> overloaded, as you encountered, the request to start the nodes fails.
>>
>> If you're using external EBS volumes with your cluster then you're
>> implicitly specifying an availability zone given that volumes can only
>> be attached to instances within their zone and thus StarCluster must
>> pick the volume's availability zone as the zone for the entire cluster.
>>
>> I think it's clear now that we should have an optional flag that
>> disables trying to launch all instances within the same zone and
>> simply lets Amazon pick the zone for each node (except for master in
>> case of EBS volumes). If folks have opinions on this I'd be happy to
>> hear them...
>>
>> For the time being, if you're not using external EBS volumes then you
>> can specify the zone to use when launching a cluster:
>>
>> $ starcluster start -a us-east-1a mycluster
>>
>> Otherwise if you're using external EBS volumes you will need to
>> snapshot your volume and recreate it in another zone that's available
>> in order for StarCluster to launch the instances in an alternate zone...
>>
>> HTH,
>>
>> ~Justin
>>
>> On 07/31/2012 01:20 PM, Erik Gafni wrote:
>> > It looks like that AWS region is at capacity and physically out of
>> > those instance types. Either try different instance types or
>> > switch regions.
>> >
>> > On Tue, Jul 31, 2012 at 4:58 AM, Ramit Bhardwaj
>> > <ramit.bhardwaj_at_sicadinc.com <mailto:ramit.bhardwaj_at_sicadinc.com>>
>> > wrote:
>> >
>> > Hello,
>> >
>> > We are using 'Star cluster' for creating clusters on Amazon Cloud.
>> > But off late, we are seeing the following error in our logs when we
>> > try to create a cluster on the US-East region:
>> >
>> > "!!! ERROR - Unsupported: The requested Availability Zone is
>> > currently constrained and we are no longer accepting new customer
>> > requests for t1/m1/c1/m2 instance types. Please retry your request
>> > by not specifying an Availability Zone or choosing us-east-1a,
>> > us-east-1d, us-east-1c."
>> >
>> > Is there anything that we are missing here? The exact setup was
>> > working perfectly till last month. We are seeing this problem since
>> > last one month.
>> >
>> > Also, in the forums i did not see anyone reporting this issue. Are
>> > there any changes to any policy or something? Please advice.
>> >
>> > Best Regards Ramit _______________________________________________
>> > StarCluster mailing list StarCluster_at_mit.edu
>> > <mailto:StarCluster_at_mit.edu>
>> > http://mailman.mit.edu/mailman/listinfo/starcluster
>> >
>> >
>>
>> -----BEGIN PGP SIGNATURE-----
>> Version: GnuPG v2.0.19 (GNU/Linux)
>> Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/
>>
>> iEYEARECAAYFAlAYGf0ACgkQ4llAkMfDcrmhkACeLNvrLUPsEWOpIX98E8r2rt6m
>> MkEAn2LlTsgjJ1iSrPkRiwgKXzciIpPb
>> =IDiw
>> -----END PGP SIGNATURE-----
>>
>
> _______________________________________________
> StarCluster mailing list
> StarCluster_at_mit.edu
> http://mailman.mit.edu/mailman/listinfo/starcluster
>
Received on Wed Aug 01 2012 - 10:46:10 EDT