Re: cluster creation best practice
This archive was generated by
You are right, and it is not common to run out of capacity but not
impossible, check the startcluster error bellow.
We use US-EAST region
*> Starting stopped node: node011*
*>>> Starting stopped node: node012*
*>>> Starting stopped node: node013*
*!!! ERROR - InsufficientInstanceCapacity: We currently do not have
sufficient c3.4xlarge capacity in the Availability Zone you requested
(us-east-1b). Our system will be working on provisioning additional
capacity. You can currently get c3.4xlarge capacity by not specifying an
Availability Zone in your request or choosing us-east-1d, us-east-1a,
On Fri, Nov 7, 2014 at 4:22 PM, Rajat Banerjee <rajatb_at_post.harvard.edu>
> Are you requesting a particular type of instance, or in an unusual region
> or AZ?
> I work for AWS and though I have no definitive evidence, I doubt that
> you'd have difficulty launching most types of instances in any of the big
> regions, N. Virginia, Oregon, or Ireland.
> On Fri, Nov 7, 2014 at 2:49 PM, Ramon Ramirez-Linan <rlinan_at_navteca.com>
>> Thanks for your answer although I was aware of all those limits and in
>> fact we requested the limit increased a while ago.
>> My question was more about how to check if AWS has the resources that I
>> need before asking StartCluster to start creating the cluster.
>> If I need a 300 nodes cluster and it fails after 250 due to AWS not
>> having enough availability of an instance type, I will still have to pay
>> for the 250 nodes for at least an hour.
>> After the 250 nodes have been created I have the option of keep trying to
>> add more nodes to complete to 300 but I think to optimize the cost it would
>> be better if I have a way of knowing in advance if AWS is going to have 300
>> EC2 for me.
>> PS: I am an AWS Solutions Architect and my company is an APN and reseller
>> On Thu, Nov 6, 2014 at 12:30 PM, Jennifer Staab <jstaab_at_cs.unc.edu>
>>> AWS does put limits on resources any one AWS account can use (see here
>>> <http://docs.aws.amazon.com/general/latest/gr/aws_service_limits.html> for
>>> details). Specifically see here
>>> for limits regarding EC2s. You can make requests to AWS to increase limit
>>> bounds. Also note there are certain actions AWS users are not allowed take
>>> (see here <http://aws.amazon.com/aup/> ), if you violate any of those
>>> policies Amazon has right to deny service to you.
>>> And another thing to consider is the time of the year. AWS started
>>> because Amazon wanted to make more use of its resources when they were less
>>> than active. I'm not sure what proportion of AWS resources are still used
>>> by Amazon itself, but I would think with holidays approaching the EC2s
>>> might be less than available as compared to the start of 2015.
>>> Good Luck.
>>> On 11/6/14 11:41 AM, Ramon Ramirez-Linan wrote:
>>> I need to create a 300 nodes cluster.
>>> 1. if there a way to query AWS in advance to see if they have the 300
>>> EC2 availables? I dont want to find out after 299 nodes that they out of
>>> that EC2 type. I am not sure if StarCluster already check for that.
>>> 2. Would it be better to create the cluster with less nodes and then
>>> add them 20 at a time?
>>> Thanks in advance
>>> StarCluster mailing listStarCluster_at_mit.eduhttp://mailman.mit.edu/mailman/listinfo/starcluster
>> StarCluster mailing list
Received on Fri Nov 07 2014 - 16:41:43 EST