StarCluster - Mailing List Archive

Re: cluster creation best practice

From: Ramon Ramirez-Linan <no email>
Date: Fri, 7 Nov 2014 14:49:17 -0500


Thanks for your answer although I was aware of all those limits and in fact
we requested the limit increased a while ago.

My question was more about how to check if AWS has the resources that I
need before asking StartCluster to start creating the cluster.

If I need a 300 nodes cluster and it fails after 250 due to AWS not having
enough availability of an instance type, I will still have to pay for the
250 nodes for at least an hour.
After the 250 nodes have been created I have the option of keep trying to
add more nodes to complete to 300 but I think to optimize the cost it would
be better if I have a way of knowing in advance if AWS is going to have 300
EC2 for me.

PS: I am an AWS Solutions Architect and my company is an APN and reseller

On Thu, Nov 6, 2014 at 12:30 PM, Jennifer Staab <> wrote:

> AWS does put limits on resources any one AWS account can use (see here
> <> for
> details). Specifically see here
> <>
> for limits regarding EC2s. You can make requests to AWS to increase limit
> bounds. Also note there are certain actions AWS users are not allowed take
> (see here <> ), if you violate any of those
> policies Amazon has right to deny service to you.
> And another thing to consider is the time of the year. AWS started because
> Amazon wanted to make more use of its resources when they were less than
> active. I'm not sure what proportion of AWS resources are still used by
> Amazon itself, but I would think with holidays approaching the EC2s might
> be less than available as compared to the start of 2015.
> Good Luck.
> -Jennifer
> On 11/6/14 11:41 AM, Ramon Ramirez-Linan wrote:
> I need to create a 300 nodes cluster.
> 1. if there a way to query AWS in advance to see if they have the 300
> EC2 availables? I dont want to find out after 299 nodes that they out of
> that EC2 type. I am not sure if StarCluster already check for that.
> 2. Would it be better to create the cluster with less nodes and then add
> them 20 at a time?
> Thanks in advance
> Ramon
> _______________________________________________
> StarCluster mailing listStarCluster_at_mit.edu
Received on Fri Nov 07 2014 - 14:49:25 EST
This archive was generated by hypermail 2.3.0.


Sort all by: