Yeah, we're going to have to go with EBS, it seems. I'd just rsync the results back, but our servers aren't outside-accessible, of course; that would just be too convenient. :)
Thanks for pointing that out to me.
- Diane
-----Original Message-----
From: Rayson Ho [mailto:raysonlogin_at_yahoo.com]
Sent: Friday, May 04, 2012 12:39 PM
To: Napolitano, Diane; starcluster_at_mit.edu
Subject: Re: [StarCluster] A way to automatically terminate an EC2 instance
BTW, I recall we had discussions related to automatically transferring data back to the local machine...
I think you can add a few lines in starcluster/balancers/sge/__init__.py so that before it shuts down the master
node, it transfers data back to the local machine first.
If you don't want to modify starcluster, just create a separate EBS for the output directory and use the load
balancer to shutdown the nodes, or tell the balancer to not kill the master.
Rayson
=================================
Open Grid Scheduler / Grid Engine
http://gridscheduler.sourceforge.net/
Scalable Grid Engine Support Program
http://www.scalablelogic.com/
----- Original Message -----
From: "Napolitano, Diane" <dnapolitano_at_ets.org>
To: Rayson Ho <raysonlogin_at_yahoo.com>; "starcluster_at_mit.edu" <starcluster_at_mit.edu>
Cc:
Sent: Friday, May 4, 2012 11:53 AM
Subject: RE: [StarCluster] A way to automatically terminate an EC2 instance
Aha! Perfect, thank you!
- Diane
-----Original Message-----
From: Rayson Ho [mailto:raysonlogin_at_yahoo.com]
Sent: Friday, May 04, 2012 11:03 AM
To: Napolitano, Diane; starcluster_at_mit.edu
Subject: Re: [StarCluster] A way to automatically terminate an EC2 instance
Use the Elastic Load Balancer to shutdown idle instances:
http://web.mit.edu/star/cluster/docs/latest/manual/load_balancer.html
The master instance can be shutdown as well, as long as you define an EBS that is not
the root filesystem of the instance.
Rayson
=================================
Open Grid Scheduler / Grid Engine
http://gridscheduler.sourceforge.net/
Scalable Grid Engine Support Program
http://www.scalablelogic.com/
----- Original Message -----
From: "Napolitano, Diane" <dnapolitano_at_ets.org>
To: "starcluster_at_mit.edu" <starcluster_at_mit.edu>
Cc:
Sent: Friday, May 4, 2012 9:41 AM
Subject: [StarCluster] A way to automatically terminate an EC2 instance
Hello, I hope someone will point me to the answer to my question, if it exists; if not, what I'm looking to do is the following:
Let's say I have an EC2 instance running several jobs that take some unknown number of hours (possibly as long as a day). They generate a whole bunch of output files, which need to be downloaded back to my local servers before I can terminate the instance; I don't have S3 here.
Now, because these jobs could take almost a day to finish, what I'd like to do is NOT sit there watching them run all day. :) I'd like some automatic way to tell EC2, be it with starcluster commands or otherwise, to logout, download all the data to my local servers, and terminate the instance the minute the jobs finish. That way I'm not stuck watching the jobs all day, but I'm not wasting money on an idle instance should the jobs finish at a time when I'm doing something crazy like sleeping. :)
Any recommendations on how to do this?
Thanks,
Diane
_______________________________________________
StarCluster mailing list
StarCluster_at_mit.edu
http://mailman.mit.edu/mailman/listinfo/starcluster
Received on Fri May 04 2012 - 13:52:28 EDT