Indeed Steve! And you won't have to pay for or configure and manage all of those 1TB EBS volumes, either.
And a modification of my 'for ID' code (the old line wouldn't work with systems with attached drives):
for ID in $(starcluster lc mycluster 2>/dev/null | grep -e "^ *node[0-9][0-9][0-9]" -e "^ master" | awk '{print $3}'); do
There's probably a better way, too. :)
-Hugh
-----Original Message-----
From: starcluster-bounces_at_mit.edu [mailto:starcluster-bounces_at_mit.edu] On Behalf Of Steve Heistand
Sent: Thursday, September 18, 2014 10:50 AM
To: starcluster_at_mit.edu
Subject: Re: [StarCluster] Changing iinstance type for everynode in the cluster
-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1
while this is a good feature to have I will throw this out there for the original
person wanting to do this and everyone else.
if these instances are going to be long running you will get much much better
overall performance if the data is sitting in S3 and you copy the data each
instance needs to a local ephemeral disk assigned to that instance type.
(note this isnt retained across type changes)
The pipe from S3 to an instance is huge and the slight extra run time for
the transfer will be made up for in IO speed to the 1T of data.
and copy the data back to S3 when the job is done as its ephemeral after all :)
s
On 09/18/2014 07:43 AM, Rayson Ho wrote:
> Hugh, thanks for testing instance type change, it was very late last night and I did
> not have time to test it.
>
> Ramon, we can add code in StarCluster to provide support for instance type change,
> and in my original email I wanted to point out that you need to use StarCluster to
> stop the cluster first, as changing the instance type means changing the IP of the
> instance too, and thus a change of instance type without letting StarCluster know
> wouldn't work.
>
> Rayson
>
> ================================================== Open Grid Scheduler - The Official
> Open Source Grid Engine http://gridscheduler.sourceforge.net/
> http://gridscheduler.sourceforge.net/GridEngine/GridEngineCloud.html
>
> On Thu, Sep 18, 2014 at 10:22 AM, MacMullan, Hugh <hughmac_at_wharton.upenn.edu
>> wrote:
>
>> Wow, I can't believe I never thought of that. Just tested c3.large > c3.8xlarge
>> transformation, and it worked just fine ? AWS CLI for the command line win!
>>
>>
>>
>> starcluster start -i c3.large -c mycluster mycluster
>>
>> starcluster stop -c mycluster
>>
>> starcluster lc ## wait till the nodes are all 'stopped'
>>
>> for ID in $(starcluster lc mycluster 2>/dev/null | grep -e node[0-9][0-9][0-9] -e
>> master | awk '{print $3}'); do
>>
>> echo $ID
>>
>> # the magic:
>>
>> aws ec2 modify-instance-attribute --instance-id $ID --instance-type c3.8xlarge
>>
>> done
>>
>> starcluster start -x mycluster
>>
>>
>>
>> Seems like that could be pretty easily integrated into StarCluster.
>>
>>
>>
>> -Hugh
>>
>>
>>
>> *From:* Rayson Ho [mailto:raysonlogin_at_gmail.com] *Sent:* Thursday, September 18,
>> 2014 1:18 AM *To:* Ramon Ramirez-Linan *Cc:* MacMullan, Hugh; starcluster_at_mit.edu
>> *Subject:* Re: [StarCluster] Changing iinstance type for everynode in the cluster
>>
>>
>>
>> I've never tried something like this myself, but I believe it would work if you
>> stop the EBS cluster (*NOT* terminate), and then change the instance type from the
>> EC2 management Console, and then restart the cluster, then StarCluster would
>> happily reconfig the cluster for you:
>>
>>
>> http://star.mit.edu/cluster/docs/latest/manual/launch.html#stopping-an-ebs-backed-cluster
>>
>>
>>
Just a reboot won't do it, as the private IP address changes when you
>> change the instance type, and /etc/hosts would need to be changed again.
>>
>> (Test with a 2-node t1.micro cluster and change the type to t2.micro before doing
>> anything experimental on a real production cluster.)
>>
>>
>> Rayson
>>
>>
>>
>>
>> On Wed, Sep 17, 2014 at 10:53 PM, Ramon Ramirez-Linan <rlinan_at_navteca.com> wrote:
>>
>> Hi,
>>
>>
>>
>> Is there a way of changing the instance type for the entire cluster after it has
>> already be launched?
>>
>>
>>
>> I want to launch a 30 nodes cluster with a cheaper instance type to ingest data on
>> each node's volume. Each node has a 1TB EBS attached and there is no need to share
>> data between nodes (no nfs or anything). Then once the data is injected on each
>> node I want to be able to change the cluster instances to a more expensive one
>> (more CPU/RAM)
>>
>>
>>
>> Will cluster reboot let me do that?
>>
>>
>>
>> Thanks in advance
>>
>>
>>
>>
>>
>
>
>
> _______________________________________________ StarCluster mailing list
> StarCluster_at_mit.edu http://mailman.mit.edu/mailman/listinfo/starcluster
>
- --
************************************************************************
Steve Heistand NASA Ames Research Center
Email: steve.heistand_at_nasa.gov Steve Heistand/Mail Stop 258-6
Work Phone: (650) 604-4369 Bldg. 258, Rm. 232-5
Scientific & HPC Application P.O. Box 1
Development/Optimization Moffett Field, CA 94035-0001
************************************************************************
"Any opinions expressed are those of our alien overlords, not my own."
# For Remedy #
#Action: Resolve #
#Resolution: Resolved #
#Reason: No Further Action Required #
#Tier1: User Code #
#Tier2: Other #
#Tier3: Assistance #
#Notification: None #
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v2.0.14 (GNU/Linux)
iEYEARECAAYFAlQa8RcACgkQoBCTJSAkVrFGHgCgyGbIELl9+eKQabFLviDrI4BP
vsMAniWKHvcQDGFVFR8c74zQrObp/SUz
=mqvF
-----END PGP SIGNATURE-----
_______________________________________________
StarCluster mailing list
StarCluster_at_mit.edu
http://mailman.mit.edu/mailman/listinfo/starcluster
Received on Thu Sep 18 2014 - 10:58:17 EDT