Re: [Starcluster] [newb] support c modules?

From: Justin Riley <no email>
Date: Mon, 31 May 2010 18:32:54 -0400

Hi Neal/Dan,

Sorry I've been out of town for the weekend. Dan correctly highlighted your
options for getting new software installed on a cluster launched by
StarCluster. If you need help with any of these methods please don't hesitate
to mail the list.

Thanks for your interest in StarCluster and thanks Dan for the prompt
response!

~Justin

Quoting Dan Yamins <dyamins_at_gmail.com>:

> On Fri, May 28, 2010 at 9:21 AM, Neal Becker <ndbecker2_at_gmail.com> wrote:
> Total newb here. I don't support I can add my own custom C modules?
>
>
> Neal,
>
> If you mean, compile your own packages on the machine(s) that in a cluster,
> you definitely can. Here are some options you have:
>
>
> 1) make a new amazon Machine Instance. Amazon machine instances (AMIs) are
> one of the main features of the underlying technology. The basic process
> for making an AMI with your custom software is this:
> -- boot up an instance of the existing machine instance that comes with
> starcluster -- but NOT as part of a cluster using the starcluster start
> command -- just as a standalone instance, say, by using Elasticfox.
> -- install the packages you want.
> -- create an AMI from the running instance by using the starcluster
> createimage command with appropriate arguments (run from your local
> machine).
>
> The process is very easy, but the last step takes a long time to complete
> --- so it's not something one does all the time. One does this, say at the
> beginning of a projec twhen one has a number of modules to add that iwll be
> useful for an extended period of time.
>
> 2) Just compile on the running cluster. You can of course just install C
> packages (or any other binaries) on a running cluster and they will work --
> but just for that instance, and you'll have to repeat the process when you
> start a new cluster.
>
> 3) Compile binaries on an EBS volume and link to those. Depending on your
> modules' configuration needs, you might be able to compile it in a directory
> on an attached EBS volume and link to those binaries from whatever
> application is using your C module. Of course, this might not work if you
> need to have compile things in particular locations (/usr/local or
> whatever), and depends on the specific EBS volume being attached whenever
> the package is needed ... but it might be easier.
>
> 4) Use a plugin. Starcluster comes with the idea of a "plugin" --- a piece
> of python code that is run after each cluster starts up. This code can
> written to do anything -- for instance, download and install your C
> module. If the compilation process isn't that long, this might be a good
> option too. See http://web.mit.edu/stardev/cluster/docs/plugins.html.
>
> I think this covers most options.
>
> Dan
>
>
>>
>
Received on Mon May 31 2010 - 18:32:56 EDT