I figured this is how it would work and tried it but I have no machines in
my ipcluster so I was wondering if I had to invoke another command.
I did see starcluster say it started ipcluster:
>>> Running plugin ipcluster
>>> Starting ipcluster...
But, when I ssh to master, I don't seem to have any machines in the
MultiEngineClient:
jtaylo_at_gkf-ubuntu:~/work/regreg/code/regreg$ starcluster sshmaster --user
sgeadmin myipcluster
sgeadmin_at_master:~$ ipython
In [1]: from IPython.kernel.client import MultiEngineClient
/usr/lib/pymodules/python2.6/IPython/kernel/clientconnector.py:43:
DeprecationWarning: Importing class Tub directly from 'foolscap' is
deprecated since Foolscap 0.4.3. Please import foolscap.api.Tub instead
self.tub = Tub()
In [2]: mec = MultiEngineClient()
/usr/lib/pymodules/python2.6/IPython/kernel/multienginefc.py:86:
DeprecationWarning: Importing class Referenceable1 directly from 'foolscap'
is deprecated since Foolscap 0.4.3. Please import foolscap.api.Referenceable
instead
class FCSynchronousMultiEngineFromMultiEngine(Referenceable):
In [3]: mec.get_ids()
Out[3]: []
-- Jonathan
On Wed, Jul 27, 2011 at 12:14 PM, Justin Riley <jtriley_at_mit.edu> wrote:
> -----BEGIN PGP SIGNED MESSAGE-----
> Hash: SHA1
>
> Hi Jonathan,
>
> The plugin still needs documenting but here's the basic flow:
>
> Put the plugin in the config:
>
> [plugin ipcluster]
> setup_class = starcluster.plugins.ipcluster.IPCluster
>
> Then put the plugin in a cluster template:
>
> [cluster smallcluster]
> ...
> plugins = ipcluster
>
> Then start a cluster:
>
> $ starcluster start myipcluster
>
> If you didn't put the plugin in the default cluster template (specified
> by default_template in the [global] section) you can specify the
> template like so:
>
> $ starcluster start -c smallcluster myipcluster
>
> This will start a cluster and run the ipcluster plugin which will setup
> an ipcluster session for your CLUSTER_USER (defaults to sgeadmin). Then
> to use ipcluster:
>
> $ starcluster sshmaster -u sgeadmin myipcluster
> sgeadmin_at_master~ ipython
> [~]> from IPython.kernel.client import MultiEngineClient
> [~]> mec = MultiEngineClient()
> [~]> mec.get_ids()
> [0, 1]
>
> In the above IPython session we see that we have two nodes in our
> ipcluster. In general, you will have as many clients as you have nodes
> in your StarCluster.
>
> After you have a MultiEngineClient instance you can then do things like:
>
> [~]> mec.map(lambda x: x**10, range(32)) # happens in parallel
> [~]> mec.execute('d=10; print d') # executes on all nodes in parallel
>
> For more details see:
>
>
> http://ipython.org/ipython-doc/rel-0.10.2/html/parallel/parallel_multiengine.html#quick-and-easy-parallelism
>
> HTH,
>
> ~Justin
>
> On 07/26/2011 11:56 PM, Jonathan Taylor wrote:
> > Hi,
> >
> > I was wondering if someone had any examples / documentations on how
> > the IPCluster plugin works in StarCluster.
> >
> > Thanks,
> >
> > Jonathan
> >
> > -- Jonathan Taylor Dept. of Statistics Sequoia Hall, 137
> > 390 Serra Mall Stanford, CA 94305 Tel: 650.723.9230 Fax:
> > 650.725.8977 Web: http://www-stat.stanford.edu/~jtaylo
>
> -----BEGIN PGP SIGNATURE-----
> Version: GnuPG v2.0.17 (GNU/Linux)
> Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/
>
> iEYEARECAAYFAk4wY50ACgkQ4llAkMfDcrl5IwCeINhKGiM7nQgIEN7Bec0cQsmJ
> yn4An3kpz6CPlxfNt0t/tHKtofPG9Nwy
> =jnrj
> -----END PGP SIGNATURE-----
>
--
Jonathan Taylor
Dept. of Statistics
Sequoia Hall, 137
390 Serra Mall
Stanford, CA 94305
Tel: 650.723.9230
Fax: 650.725.8977
Web: http://www-stat.stanford.edu/~jtaylo
Received on Wed Jul 27 2011 - 16:04:59 EDT