StarCluster - Mailing List Archive

Re: How does StarCluster track the clusters it's managing?

From: Steve Darnell <no email>
Date: Mon, 16 Mar 2015 20:39:01 +0000

Hi Raj,

Thanks for the reply. Manual clean-up is indeed required to deal with these rouge instances. It would be really convenient if loadbalancer resolved this scenario automatically once an hour. One can dream (or implement)…

Best regards,

From: [] On Behalf Of Rajat Banerjee
Sent: Monday, March 16, 2015 2:04 PM
To: Steve Darnell
Cc: Eduardo Gurgel Valente; Nicholas Chammas;
Subject: Re: [StarCluster] How does StarCluster track the clusters it's managing?

Sorry for the super-slow response.
The elastic load balancer parses the output of 'qhost' on the cluster:
I don't remember the exact reason for using that instead of the same logic as 'listclusters' above, but here's my guess a few years after the fact:
- Avoids another remote API call to AWS' tagging service to retrieve the tags for all instances within an account. This needs to be called every minute, so a speedy call to your cluster instead of to a remote API is beneficial
- qhost outputs the number of machines correctly configured and able to process work. If a machine shows up in 'listcluster' but not in 'qhost' it's likely not usable to process jobs, and would probably need manual cleanup.

On Tue, Mar 10, 2015 at 4:04 PM, Steve Darnell <<>> wrote:
On a related topic, does anyone know how the load balancing feature tracks the cluster and its compute nodes? I have gotten into situations where listclusters correctly reports that a cluster and its nodes are running (I can ssh into master and the nodes, etc.); however, loadbalance reports that the cluster is not running and refuses to balance the cluster.

Best regards,

From:<> [<>] On Behalf Of Eduardo Gurgel Valente
Sent: Tuesday, March 10, 2015 2:08 PM
To: Nicholas Chammas
Subject: Re: [StarCluster] How does StarCluster track the clusters it's managing?

Hi Nick,
   Look at the security group it creates. It follows a naming convention. In addition there are tags with encrypted information at play.

On Mon, Mar 9, 2015 at 11:16 PM, Nicholas Chammas <<>> wrote:


At this point in the StarCluster demo video<>, the presenter runs the following command to list all the clusters being managed by StarCluster:

starcluster listclusters

How does StarCluster track all the clusters it’s managing? Is it through the use of EC2 instance tags? A pointer to the relevant code would also be helpful.

I’m looking to implement a feature similar to listclusters but for spark-ec2<>. Tagging seems like the way to go to do that, but we had some issues with it<> when we used it with spark-ec2.

So I’m curious to know how StarCluster did things.


StarCluster mailing list<>

StarCluster mailing list<>

Received on Mon Mar 16 2015 - 16:39:01 EDT
This archive was generated by hypermail 2.3.0.


Sort all by: