StarCluster - Mailing List Archive

Re: A way to automatically terminate an EC2 instance

From: Napolitano, Diane <no email>
Date: Fri, 4 May 2012 10:46:33 -0700

Yeah, we're going to have to go with EBS, it seems. I'd just rsync the results back, but our servers aren't outside-accessible, of course; that would just be too convenient. :)

Thanks for pointing that out to me.

- Diane

-----Original Message-----
From: Rayson Ho [mailto:raysonlogin_at_yahoo.com]
Sent: Friday, May 04, 2012 12:39 PM
To: Napolitano, Diane; starcluster_at_mit.edu
Subject: Re: [StarCluster] A way to automatically terminate an EC2 instance

BTW, I recall we had discussions related to automatically transferring data back to the local machine...

I think you can add a few lines in starcluster/balancers/sge/__init__.py so that before it shuts down the master
node, it transfers data back to the local machine first.

If you don't want to modify starcluster, just create a separate EBS for the output directory and use the load
balancer to shutdown the nodes, or tell the balancer to not kill the master.


Rayson

=================================
Open Grid Scheduler / Grid Engine
http://gridscheduler.sourceforge.net/

Scalable Grid Engine Support Program
http://www.scalablelogic.com/



----- Original Message -----
From: "Napolitano, Diane" <dnapolitano_at_ets.org>
To: Rayson Ho <raysonlogin_at_yahoo.com>; "starcluster_at_mit.edu" <starcluster_at_mit.edu>
Cc:
Sent: Friday, May 4, 2012 11:53 AM
Subject: RE: [StarCluster] A way to automatically terminate an EC2 instance

Aha!  Perfect, thank you!

- Diane

-----Original Message-----
From: Rayson Ho [mailto:raysonlogin_at_yahoo.com]
Sent: Friday, May 04, 2012 11:03 AM
To: Napolitano, Diane; starcluster_at_mit.edu
Subject: Re: [StarCluster] A way to automatically terminate an EC2 instance

Use the Elastic Load Balancer to shutdown idle instances:


http://web.mit.edu/star/cluster/docs/latest/manual/load_balancer.html

The master instance can be shutdown as well, as long as you define an EBS that is not
the root filesystem of the instance.


Rayson

=================================
Open Grid Scheduler / Grid Engine
http://gridscheduler.sourceforge.net/

Scalable Grid Engine Support Program
http://www.scalablelogic.com/




----- Original Message -----
From: "Napolitano, Diane" <dnapolitano_at_ets.org>
To: "starcluster_at_mit.edu" <starcluster_at_mit.edu>
Cc:
Sent: Friday, May 4, 2012 9:41 AM
Subject: [StarCluster] A way to automatically terminate an EC2 instance

Hello, I hope someone will point me to the answer to my question, if it exists; if not, what I'm looking to do is the following:

Let's say I have an EC2 instance running several jobs that take some unknown number of hours (possibly as long as a day).  They generate a whole bunch of output files, which need to be downloaded back to my local servers before I can terminate the instance; I don't have S3 here.

Now, because these jobs could take almost a day to finish, what I'd like to do is NOT sit there watching them run all day. :)  I'd like some automatic way to tell EC2, be it with starcluster commands or otherwise, to logout, download all the data to my local servers, and terminate the instance the minute the jobs finish.  That way I'm not stuck watching the jobs all day, but I'm not wasting money on an idle instance should the jobs finish at a time when I'm doing something crazy like sleeping. :)

Any recommendations on how to do this?

Thanks,
Diane

_______________________________________________
StarCluster mailing list
StarCluster_at_mit.edu
http://mailman.mit.edu/mailman/listinfo/starcluster
Received on Fri May 04 2012 - 13:52:28 EDT
This archive was generated by hypermail 2.3.0.

Search:

Sort all by:

Date

Month

Thread

Author

Subject