StarCluster - Mailing List Archive

adding nodes to an existing cluster of a different instance type

From: Lyn Gerner <no email>
Date: Thu, 24 Jan 2013 10:00:02 -0800

Hi Ryan,

I experimented with addnode a while back:

1) Couldn't add an on-demand instance to a cluster initiated w/spot-bid
instances.

2) *Could* add a spot-bid instance to a cluster initiated w/on-demand
instances.

3) addnode command in any case will add whatever instance type you specify
with "-I", so you can run heterogeneously amongst the compute hosts (with
each instance potentially having a different bid price).

Cheers,
Lyn

---------- Forwarded message ----------
From: Ryan Golhar <ngsbioinformatics_at_gmail.com>
To: starcluster_at_mit.edu
Cc:
Date: Thu, 24 Jan 2013 12:39:47 -0500
Subject: [StarCluster] adding nodes to an existing cluster of a different
instance type
Is it possible to add a node to a running cluster that is different from
the instance type in the config file? I didn't see anything in the docs


---------- Forwarded message ----------
From: "MacMullan, Hugh" <hughmac_at_wharton.upenn.edu>
To: "'Ed Morris'" <ecm200_at_gmail.com>
Cc: "'starcluster_at_mit.edu'" <starcluster_at_mit.edu>
Date: Thu, 24 Jan 2013 17:40:56 +0000
Subject: Re: [StarCluster] Amazon AMI 64-bit Ubuntu 11 HVM-EBS for
eu-west-1 region

Hi Ed:****

** **

Woohoo, great news!****

** **

Lincoln Stein, maintainer of the VM::EC2 Perl module, has added HVM copy
functionality to his (already pretty awesome) module! The module includes
the Perl script migrate-ebs-image.pl, which makes copies a breeze.****

** **

I just tested it with a copy of a Scientific Linux 6.3 HVM StarCluster
image that I use frequently. Hereís the output:****

** **

# migrate-ebs-image.pl --from us-east-1 --to eu-west-1 ami-XXXXXXXXX****

[info] Scanning for existing staging servers and volumes in
http://ec2.us-east-1.amazonaws.com.****

[info] Scanning for existing staging servers and volumes in
http://ec2.eu-west-1.amazonaws.com.****

[info] Gathering information about image ami-XXXXXXXXX.****

[info] Copying EBS volumes attached to this image (this may take a long
time).****

[info] Copying snapshot snap-XXXXXXXX from us-east-1 to eu-west-1...**
**

[info] ...new snapshot = snap-XXXXXXXXXX; status = pending****

[info] Waiting for all snapshots to complete. This may take a long time.*
***

An image named 'starcluster-base-scientific-linux-6.3-x86_64-hvm-005'
already exists in destination region. Renamed to
'starcluster-base-scientific-linux-6.3-x86_64-hvm-005-d88a4dfe' *Ŗ** I
REALLY LIKE THIS!!!*****

[info] Registering snapshot in destination with the equivalent of:****

[info] ec2-register -n
'starcluster-base-scientific-linux-6.3-x86_64-hvm-005-XXXXX' -d 'Image
created _at_ 201301202215' -a x86_64 --kernel '' --ramdisk ''
--root-device-name /dev/sda1 -b /dev/sda1=snap-XXXXXXXXXXX:false:standard***
*

[info] Searching for a suitable HVM image in destination region****

[info] ...Found ami-011b1975 (suse-sles-11-sp2-hvm-v1.01.x86_64)****

[info] Launching an HVM staging server in the target region.
Heuristically choosing instance type of 'cc2.8xlarge' for this type of HVM..
****

[info] Waiting for instance to become ready.****

[info] Stopping instance temporarily to swap root volumes.****

[info] Detaching original root volume...****

[info] Creating and attaching new root volume..****

[info] Creating image in destination region...****

[info] Terminating the staging server****

New snapshot is now located in eu-west-1 under ami-XXXXXXXXX.****

** **

No muss, no fuss. As you can see, it has to launch a cc2.8xlarge to do the
copy, so itíll cost you to do the transfer. Itís currently in the
hvm-migrate alpha GIT tree:****

** **

https://github.com/lstein/LibVM-EC2-Perl/tree/hvm-migrate****

** **

Heíll be publishing to CPAN today or tomorrow after Windows AMI testing. It
will be VM::EC2 1.23 on CPAN.****

** **

So there you go.****

** **

-Hugh****

** **

** **

*From:* Ed Morris [mailto:ecm200_at_gmail.com]
*Sent:* Friday, January 18, 2013 12:43 PM
*To:* MacMullan, Hugh
*Cc:* starcluster_at_mit.edu
*Subject:* Re: [StarCluster] Amazon AMI 64-bit Ubuntu 11 HVM-EBS for
eu-west-1 region****

** **

** **

Hi Hugh.****

** **

Thanks for that, much appreciated.****

** **

I tried using the "snapshot" route, which allows you to transfer snapshots
between regions through the AWS web console (AWS documentation
here<http://docs.aws.amazon.com/AWSEC2/latest/UserGuide/ebs-copy-snapshot.html>),
but so far I have had no luck with regards to that. I started an instance
with the US-EAST-1 Ubuntu 11.10 x86_64 HVM AMI (not through Star Cluster),
and then created a snap shot of the root volume. I then transferred the
snapshot to EU-WEST-1 region and converted that into an AMI. One issue I
had, and have still not solved, was the "kernel ID" specification which was
needed during the creation of the AMI from the snap shot. I could not find
the kernel ID for the AMI in question, even though other flavours of Star
Cluster AMI's did have associated Kernel IDs. I came unstuck when I tried
to execute the AMI that I had transferred from the US-EAST-1 region to
EU-WEST-1 region, and I was told by StarCluster that the AMI was not HVM,
and therefore could not be executed on the CC2.8xlarge instance type. I am
positive that I am using the correct AMI which is HVM supported, as this is
currently the only HVM supported AMI that's available, and it's only
available in US-EAST-1.****

** **

I plan to try something slightly different, by creating a custom AMI using
the StarCluster tools, and then creating a snap shot from that AMI, and
transferring that. But again I may come up against the Kernel ID issue,
which I still don't know how to solve.****

** **

However some good news, I have read in a number of places on the web that
Amazon are developing a tool for transferring AMI's the same way you can
now transfer snap shots, and can be undertaken through the AWS console. I
haven't managed to find when this is expected to be implemented though.****

** **

There is also this that I found (Ylastic <http://ylastic.com/features.html>),
which amongst a lot of things, appears to suggest that it can migrate EBS
AMI's from region to region.****

** **

** **

On 18 January 2013 18:16, MacMullan, Hugh <hughmac_at_wharton.upenn.edu> wrote:
****

Hi Ed:****

 ****

I have no EU AMIs, but Iíve been curious about the second question you
have: whatís the Ďeasyí way to transfer AMIs? I poked around a bit today,
and it looks like Ďmigrate-ebs-image.plí, part of the VM::EC2 Perl module,
is a good solution Ö *except* that it converts my HVM AMIs to PV. Argh! J***
*

 ****

Iíve got a query in with the developer (Lincoln Stein) about HVM copies.
Iíll let you know what I discover.****

 ****

That said, if others have other *easy* ways to transfer AMIs Ö Iím all ears
too!****

 ****

-Hugh****

 ****

 ****

*From:* starcluster-bounces_at_mit.edu [mailto:starcluster-bounces_at_mit.edu] *On
Behalf Of *Ed Morris
*Sent:* Friday, January 18, 2013 5:26 AM
*To:* starcluster_at_mit.edu
*Subject:* [StarCluster] Amazon AMI 64-bit Ubuntu 11 HVM-EBS for eu-west-1
region****

 ****

Hi.****

 ****

Does anyone know if there is an AMI available using HVM version of Ubuntu
for the cc8.xlarge compute node for starcluster in the eu-west-1 region?****

 ****

I have been quite happily using the AMI ami-4583572c us-east-1
starcluster-base-ubuntu-11.10-x86_64-hvm (HVM-EBS) on the us-east region,
but couldn't find any HVM versions in the eu region.****

 ****

Any ideas? Can the HVM-EBS be easily transferred to the eu region?****

 ****

Ed.
****

 ****

-- 
Ed Morris
------------------------------------------------
email: ecm200_at_gmail.com
phone: ****
****
** **
-- 
Ed Morris
------------------------------------------------
email: ecm200_at_gmail.com
phone: ****
_______________________________________________
StarCluster mailing list
StarCluster_at_mit.edu
http://mailman.mit.edu/mailman/listinfo/starcluster
Received on Thu Jan 24 2013 - 13:00:03 EST
This archive was generated by hypermail 2.3.0.

Search:

Sort all by:

Date

Month

Thread

Author

Subject