Re: cluster creation best practice

From: Rajat Banerjee <no email>
Date: Fri, 7 Nov 2014 16:22:33 -0500

Are you requesting a particular type of instance, or in an unusual region
or AZ?

I work for AWS and though I have no definitive evidence, I doubt that you'd
have difficulty launching most types of instances in any of the big
regions, N. Virginia, Oregon, or Ireland.

On Fri, Nov 7, 2014 at 2:49 PM, Ramon Ramirez-Linan <rlinan_at_navteca.com>
wrote:

> Jennifer,
>
> Thanks for your answer although I was aware of all those limits and in
> fact we requested the limit increased a while ago.
>
> My question was more about how to check if AWS has the resources that I
> need before asking StartCluster to start creating the cluster.
>
> If I need a 300 nodes cluster and it fails after 250 due to AWS not having
> enough availability of an instance type, I will still have to pay for the
> 250 nodes for at least an hour.
> After the 250 nodes have been created I have the option of keep trying to
> add more nodes to complete to 300 but I think to optimize the cost it would
> be better if I have a way of knowing in advance if AWS is going to have 300
> EC2 for me.
>
>
>
> PS: I am an AWS Solutions Architect and my company is an APN and reseller
> partner.
>
>
> On Thu, Nov 6, 2014 at 12:30 PM, Jennifer Staab <jstaab_at_cs.unc.edu> wrote:
>
>> AWS does put limits on resources any one AWS account can use (see here
>> <http://docs.aws.amazon.com/general/latest/gr/aws_service_limits.html> for
>> details). Specifically see here
>> <http://aws.amazon.com/ec2/faqs/#How_many_instances_can_I_run_in_Amazon_EC2>
>> for limits regarding EC2s. You can make requests to AWS to increase limit
>> bounds. Also note there are certain actions AWS users are not allowed take
>> (see here <http://aws.amazon.com/aup/> ), if you violate any of those
>> policies Amazon has right to deny service to you.
>>
>> And another thing to consider is the time of the year. AWS started
>> because Amazon wanted to make more use of its resources when they were less
>> than active. I'm not sure what proportion of AWS resources are still used
>> by Amazon itself, but I would think with holidays approaching the EC2s
>> might be less than available as compared to the start of 2015.
>>
>> Good Luck.
>>
>> -Jennifer
>>
>>
>>
>> On 11/6/14 11:41 AM, Ramon Ramirez-Linan wrote:
>>
>> I need to create a 300 nodes cluster.
>>
>> 1. if there a way to query AWS in advance to see if they have the 300
>> EC2 availables? I dont want to find out after 299 nodes that they out of
>> that EC2 type. I am not sure if StarCluster already check for that.
>>
>> 2. Would it be better to create the cluster with less nodes and then
>> add them 20 at a time?
>>
>> Thanks in advance
>> Ramon
>>
>>
>> _______________________________________________
>> StarCluster mailing listStarCluster_at_mit.eduhttp://mailman.mit.edu/mailman/listinfo/starcluster
>>
>>
>>
>
> _______________________________________________
> StarCluster mailing list
> StarCluster_at_mit.edu
> http://mailman.mit.edu/mailman/listinfo/starcluster
>
>
Received on Fri Nov 07 2014 - 16:22:56 EST