Re: using ipython as a cluster

From: MinRK <no email>
Date: Wed, 27 Jul 2011 12:31:58 -0700

And I've been using an updated version of the plugin to work with
IPython 0.11 (parallel code has been completely rewritten, and is much
improved for certain activities). My plugins:

https://github.com/minrk/starclusterplugins

-MinRK

On Wed, Jul 27, 2011 at 12:14, Justin Riley <jtriley_at_mit.edu> wrote:
> -----BEGIN PGP SIGNED MESSAGE-----
> Hash: SHA1
>
> Hi Jonathan,
>
> The plugin still needs documenting but here's the basic flow:
>
> Put the plugin in the config:
>
> [plugin ipcluster]
> setup_class = starcluster.plugins.ipcluster.IPCluster
>
> Then put the plugin in a cluster template:
>
> [cluster smallcluster]
> ...
> plugins = ipcluster
>
> Then start a cluster:
>
> $ starcluster start myipcluster
>
> If you didn't put the plugin in the default cluster template (specified
> by default_template in the [global] section) you can specify the
> template like so:
>
> $ starcluster start -c smallcluster myipcluster
>
> This will start a cluster and run the ipcluster plugin which will setup
> an ipcluster session for your CLUSTER_USER (defaults to sgeadmin). Then
> to use ipcluster:
>
> $ starcluster sshmaster -u sgeadmin myipcluster
> sgeadmin_at_master~ ipython
> [~]> from IPython.kernel.client import MultiEngineClient
> [~]> mec = MultiEngineClient()
> [~]> mec.get_ids()
> [0, 1]
>
> In the above IPython session we see that we have two nodes in our
> ipcluster. In general, you will have as many clients as you have nodes
> in your StarCluster.
>
> After you have a MultiEngineClient instance you can then do things like:
>
> [~]> mec.map(lambda x: x**10, range(32)) # happens in parallel
> [~]> mec.execute('d=10; print d') # executes on all nodes in parallel
>
> For more details see:
>
> http://ipython.org/ipython-doc/rel-0.10.2/html/parallel/parallel_multiengine.html#quick-and-easy-parallelism
>
> HTH,
>
> ~Justin
>
> On 07/26/2011 11:56 PM, Jonathan Taylor wrote:
>> Hi,
>>
>> I was wondering if someone had any examples / documentations on how
>> the IPCluster plugin works in StarCluster.
>>
>> Thanks,
>>
>> Jonathan
>>
>> -- Jonathan Taylor Dept. of Statistics Sequoia Hall, 137
>>  390 Serra Mall Stanford, CA 94305 Tel:   650.723.9230 Fax:
>> 650.725.8977 Web: http://www-stat.stanford.edu/~jtaylo
>
> -----BEGIN PGP SIGNATURE-----
> Version: GnuPG v2.0.17 (GNU/Linux)
> Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/
>
> iEYEARECAAYFAk4wY50ACgkQ4llAkMfDcrl5IwCeINhKGiM7nQgIEN7Bec0cQsmJ
> yn4An3kpz6CPlxfNt0t/tHKtofPG9Nwy
> =jnrj
> -----END PGP SIGNATURE-----
> _______________________________________________
> StarCluster mailing list
> StarCluster_at_mit.edu
> http://mailman.mit.edu/mailman/listinfo/starcluster
>
Received on Wed Jul 27 2011 - 15:32:18 EDT