StarCluster - Mailing List Archive

Re: [Starcluster] Spot instances

From: Justin Riley <no email>
Date: Thu, 15 Apr 2010 17:36:10 -0400

Hash: SHA1

Hi Thomas,

This is indeed a great idea and I plan to incorporate spot instances to
starcluster, however, not for this immediate release coming up.

The first step would be to allow starcluster to launch spot instances
via the start command given that I still haven't written an add/remove
node command for starcluster. The add/remove node commands are planned
but will take some time and significant testing before they're ready for
public use.

You're correct that jobs running on spot instances would be vulnerable
to being arbitrarily terminated. I'm not sure if SGE would auto
restart/migrate the jobs or not but their may be a way to configure it
to do so. I'd simply need to play around with this by trying it on a
test cluster and then seeing what's involved to configure SGE to
migrate/restart failed jobs. If you have ideas or experience in this
space I'd certainly love to hear them.

Hope that helps,


On 04/15/2010 04:07 PM, Thomas Deselaers wrote:
> Hi,
> As I am finding new features in AWS, I come up with new ideas.
> I have read on the mailinglist that adding and removing nodes to/from
> a cluster is a planned feature.
> Now I was wondering if this could be tied into the "spot instances".
> So, I would start a normal instance as master and as many spot
> instances as I get (for my price) as nodes.
> This might lead to some jobs crashing when spot instances are killed
> (without warning)... ideally, SGE would notice that the jobs crashed
> and restart them.
> Cheers,
> thomas

Version: GnuPG v2.0.14 (GNU/Linux)
Comment: Using GnuPG with Mozilla -

Received on Thu Apr 15 2010 - 17:36:12 EDT
This archive was generated by hypermail 2.3.0.


Sort all by: