On Wed, Oct 17, 2012 at 10:00 AM, Justin Riley <jtriley_at_mit.edu> wrote:
> -----BEGIN PGP SIGNED MESSAGE-----
> Hash: SHA1
>
> Hey Olivier,
>
> 1. I'm very close to releasing a new 12.04 AMI with IPython 0.13 on
> it. In the mean time you can use the git version of StarCluster which
> has support for 0.13 and create a custom SC AMI with latest IPython.
>
Justin - we intend to cut 0.13.1 this weekend (rc3 here:
http://archive.ipython.org/testing/0.13.1/)
Olivier - I wrote a few tiny plugins a while ago, when I needed to run dev
zeromq/pyzmq/ipython on starcluster:
https://github.com/minrk/starclusterplugins
I haven't used them in some time, so they may need a bit of fixing up if
starcluster APIs have changed.
Among them is a generic installer (EggSetup) that would let you use
easy_install (or pip with a tiny edit, though I specifically wanted to
fetch eggs to avoid recompilation) to install any packages you may need at
node startup.
>
> 2. It definitely is possible to extend the plugin to do this, however,
> this behavior should be off by default and configurable via the
> StarCluster config.
>
> 3. Yes I'm leaning towards doing this by default given that it ties up
> the entire SGE queue anyway.
>
> I'm certainly up for a pull request for 2 and 3 if you have time. I'm
> super busy at the moment playing catch up between finishing up next
> version, building the 12.04 AMIs, and responding to mailing list but I
> will eventually get to this. Feel free to file a bug for this as well...
>
> HTH,
>
> ~Justin
>
> On 10/16/2012 05:55 AM, Olivier Grisel wrote:
> > Hi,
> >
> > I have a couple of questions for the ipcluster plugin:
> >
> > 1- Is there an (dev) 64bit AMI with the latest stable release of
> > IPython (0.13) instead of 0.12? 2- Would it be possible to extend
> > the ipcluster plugin to have the nodes install the dev version of
> > ipython instead of the system version on startup? For instance by
> > cloning a tag or a specific branch (master) of the IPython github
> > repo. 3- Would be possible to make ipcluster not use SGE for
> > engine configuration and only use ssh instead, for instance as
> > documented in this very short blog post:
> >
> http://graal.ift.ulaval.ca/jeanfrancisroy/2012/09/27/using-ipython-for-parallel-computing/
>
That blog post, while helpful and short, actually *overstates* the
complexity of getting engines up and running, at least in a starcluster
environment. With the local network already configured, there would
actually be no SSH involved. No ipcluster config would be necessary, only
local launchers would need to be used, and the single command `ipcluster
engines [-n NUMBER] [--daemon]` on each node at startup would totally
suffice.
-MinRK
>
> >
> >
> The goal would be to speedup cluster startup by skipping the SGE
> > setup.
> >
> > If you think 2 and 3 are desirable contributions for the project I
> > can work on 2 pull requests for those.
> >
> > Best,
> >
>
> -----BEGIN PGP SIGNATURE-----
> Version: GnuPG v2.0.19 (GNU/Linux)
> Comment: Using GnuPG with Mozilla - http://www.enigmail.net/
>
> iEYEARECAAYFAlB+5DMACgkQ4llAkMfDcrnT2wCeKo0N4S5aQ2Cwn/p8unnltKLc
> 1+cAnRrsLIcF6DmP1rKZtyQI05K46aTm
> =BAek
> -----END PGP SIGNATURE-----
> _______________________________________________
> StarCluster mailing list
> StarCluster_at_mit.edu
> http://mailman.mit.edu/mailman/listinfo/starcluster
>
Received on Wed Oct 17 2012 - 15:31:02 EDT