Thank you very much Justin and Rayson for the explanation,
I actually started the cluster Yesterday and learned a few issues by try
and error, realising the already mounted EBS in the starcluster HVM AMI,
but terminates with it and needs downloading the data. I also realised the
unfeasibility of the thought of downloading an image, updating it on my
machine, and then upload it after testing and everytime I need it. Also
using a spotnode cost less, so I can experiment with less worries, but the
IO free-tier is small, and only tiny instances are in the free tier, and I
need cg1.4xlarge to use the GPU Cluster,
I tried the following steps:
http://instantbadger.blogspot.com.au/2009/09/how-to-create-and-save-ami-image-from.html
to bundle, upload and register the customised running AMI after my updates,
and the bundle took time indeed, but this seams like the alternative to the
scenario download/upload I was thinking of,
thanks again for your help,
Kind Regards,
On 18 May 2012 07:40, Rayson Ho <raysonlogin_at_gmail.com> wrote:
> On Thu, May 17, 2012 at 5:23 PM, Justin Riley <jtriley_at_mit.edu> wrote:
> > If you're trying to find a way to store data without having to keep the
> > cluster around there are much better approaches:
>
> Manal,
>
> Since you mentioned that you are new to AWS, you should be able to get
> the benefits of the free tier:
>
> http://aws.amazon.com/free/
>
> Basically, you can use 30 GB of EBS storage free of charge. On the
> other hand, if you pull data from AWS to your local harddrive
> everytime you are done with the instance, then you will need to pay
> for I/O & data transfer costs.
>
>
> Justin,
>
> I am trying to find the version of the NVIDIA Driver & CUDA toolkit
> installed on the CG1 AMIs, but instead of booting up an instance to
> find out what's there, is there a page that lists the additional
> software packages installed in the AMIs?? So far, I could only find:
>
> https://github.com/jtriley/StarCluster/issues/9
>
> Rayson
>
> ================================
> Open Grid Scheduler / Grid Engine
> http://gridscheduler.sourceforge.net/
>
> Scalable Grid Engine Support Program
> http://www.scalablelogic.com/
>
>
>
> >
> > 1. Use an external EBS volume with StarCluster and store your data
> > there. This will allow you to terminate the cluster and still preserve
> > your important data on the EBS volume. See:
> >
> > http://web.mit.edu/star/cluster/docs/latest/manual/volumes.html
> >
> > 2. If you don't wish to use external EBS volumes at all you can always
> > use starcluster's "get" command to manually download files from the
> > cluster before terminating, e.g.:
> >
> > $ starcluster get mycluster /path/to/results.tar.gz
> >
> > Similarly you can upload files using the "put" command:
> >
> > $ starcluster put mycluster /path/to/local/file /remote/path/
> >
> > See the following doc for more details:
> >
> > http://web.mit.edu/star/cluster/docs/latest/manual/putget.html
> >
> >> 4. After knowing how to download the image, I will need the command
> to
> >> start a new cluster using the image I have stored on my local disk. I
> >> think I have seen something to install from an image to the cluster
> >
> > Launching a cluster using a locally stored disk image is not supported.
> > The latency in doing this would be *insane*. Having to upload multiple
> > GBs and register an AMI everytime would make starting a cluster
> > *extremely* time-consuming and error-prone.
> >
> > As mentioned above you're better off creating a new AMI for your needs
> > and keeping the new AMI on Amazon. Then simply update your StarCluster
> > config to use your new AMI. If you need to save data before terminating
> > the cluster then either save your data to an external EBS volume or copy
> > the data using the 'get' command as mentined above.
> >
> > Hope that helps,
> >
> > ~Justin
> >
> > _______________________________________________
> > StarCluster mailing list
> > StarCluster_at_mit.edu
> > http://mailman.mit.edu/mailman/listinfo/starcluster
> >
>
>
>
> --
> ==================================================
> Open Grid Scheduler - The Official Open Source Grid Engine
> http://gridscheduler.sourceforge.net/
>
--
Kind Regards,
Manal Helal
Received on Thu May 17 2012 - 21:29:00 EDT