Re: using ipython as a cluster

From: MinRK <no email>
Date: Wed, 27 Jul 2011 13:45:24 -0700

Sorry, the message for doing `Client('starcluster.json')` is for
connecting to the client from your laptop (or whatever machine you are
running the starcluster scripts from). You shouldn't need to pass any
arguments to connect a client that is running on the cluster. Just:
`rc = Client()`.

-MinRK

On Wed, Jul 27, 2011 at 13:38, Jonathan Taylor
<jonathan.taylor_at_stanford.edu> wrote:
> I tried these new plugins which seemed to install fine but something's still
> not quite right -- trying to get a Client hangs:
>
> sgeadmin_at_master:~$ ipython
> Python 2.6.5 (r265:79063, Apr 16 2010, 13:09:56)
> Type "copyright", "credits" or "license" for more information.
>
> IPython 0.11.dev -- An enhanced Interactive Python.
> ?         -> Introduction and overview of IPython's features.
> %quickref -> Quick reference.
> help      -> Python's own help system.
> object?   -> Details about 'object', use 'object??' for extra details.
>
> In [1]: from IPython.parallel import Client
>
> In [2]: rc = Client('starcluster.json')
> ---------------------------------------------------------------------------
> AssertionError                            Traceback (most recent call last)
> /home/sgeadmin/<ipython-input-2-c67e0a34263a> in <module>()
> ----> 1 rc = Client('starcluster.json')
>
> /usr/local/lib/python2.6/dist-packages/ipython-0.11.dev-py2.6.egg/IPython/parallel/client/client.pyc
> in __init__(self, url_or_file, profile, profile_dir, ipython_dir, context,
> debug, exec_key, sshserver, sshkey, password, paramiko, timeout,
> **extra_args)
>     313                 if self._cd:
>     314                     url_or_file =
> os.path.join(self._cd.security_dir, url_or_file)
> --> 315                 assert os.path.exists(url_or_file), "Not a valid
> connection file or url: %r"%url_or_file
>     316             with open(url_or_file) as f:
>     317                 cfg = json.loads(f.read())
>
> AssertionError: Not a valid connection file or url:
> u'/home/sgeadmin/.ipython/profile_default/security/starcluster.json'
>
> ---------
>
> There are only two JSON files on the master node:
>
> sgeadmin_at_master:~$ !fi
> find . -name \*json
> ./.ipython/profile_default/security/ipcontroller-client.json
> ./.ipython/profile_default/security/ipcontroller-engine.json
>
>
>
> -- Jonathan
>
> On Wed, Jul 27, 2011 at 12:31 PM, MinRK <benjaminrk_at_gmail.com> wrote:
>>
>> And I've been using an updated version of the plugin to work with
>> IPython 0.11 (parallel code has been completely rewritten, and is much
>> improved for certain activities).  My plugins:
>>
>> https://github.com/minrk/starclusterplugins
>>
>> -MinRK
>>
>> On Wed, Jul 27, 2011 at 12:14, Justin Riley <jtriley_at_mit.edu> wrote:
>> > -----BEGIN PGP SIGNED MESSAGE-----
>> > Hash: SHA1
>> >
>> > Hi Jonathan,
>> >
>> > The plugin still needs documenting but here's the basic flow:
>> >
>> > Put the plugin in the config:
>> >
>> > [plugin ipcluster]
>> > setup_class = starcluster.plugins.ipcluster.IPCluster
>> >
>> > Then put the plugin in a cluster template:
>> >
>> > [cluster smallcluster]
>> > ...
>> > plugins = ipcluster
>> >
>> > Then start a cluster:
>> >
>> > $ starcluster start myipcluster
>> >
>> > If you didn't put the plugin in the default cluster template (specified
>> > by default_template in the [global] section) you can specify the
>> > template like so:
>> >
>> > $ starcluster start -c smallcluster myipcluster
>> >
>> > This will start a cluster and run the ipcluster plugin which will setup
>> > an ipcluster session for your CLUSTER_USER (defaults to sgeadmin). Then
>> > to use ipcluster:
>> >
>> > $ starcluster sshmaster -u sgeadmin myipcluster
>> > sgeadmin_at_master~ ipython
>> > [~]> from IPython.kernel.client import MultiEngineClient
>> > [~]> mec = MultiEngineClient()
>> > [~]> mec.get_ids()
>> > [0, 1]
>> >
>> > In the above IPython session we see that we have two nodes in our
>> > ipcluster. In general, you will have as many clients as you have nodes
>> > in your StarCluster.
>> >
>> > After you have a MultiEngineClient instance you can then do things like:
>> >
>> > [~]> mec.map(lambda x: x**10, range(32)) # happens in parallel
>> > [~]> mec.execute('d=10; print d') # executes on all nodes in parallel
>> >
>> > For more details see:
>> >
>> >
>> > http://ipython.org/ipython-doc/rel-0.10.2/html/parallel/parallel_multiengine.html#quick-and-easy-parallelism
>> >
>> > HTH,
>> >
>> > ~Justin
>> >
>> > On 07/26/2011 11:56 PM, Jonathan Taylor wrote:
>> >> Hi,
>> >>
>> >> I was wondering if someone had any examples / documentations on how
>> >> the IPCluster plugin works in StarCluster.
>> >>
>> >> Thanks,
>> >>
>> >> Jonathan
>> >>
>> >> -- Jonathan Taylor Dept. of Statistics Sequoia Hall, 137
>> >>  390 Serra Mall Stanford, CA 94305 Tel:   650.723.9230 Fax:
>> >> 650.725.8977 Web: http://www-stat.stanford.edu/~jtaylo
>> >
>> > -----BEGIN PGP SIGNATURE-----
>> > Version: GnuPG v2.0.17 (GNU/Linux)
>> > Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/
>> >
>> > iEYEARECAAYFAk4wY50ACgkQ4llAkMfDcrl5IwCeINhKGiM7nQgIEN7Bec0cQsmJ
>> > yn4An3kpz6CPlxfNt0t/tHKtofPG9Nwy
>> > =jnrj
>> > -----END PGP SIGNATURE-----
>> > _______________________________________________
>> > StarCluster mailing list
>> > StarCluster_at_mit.edu
>> > http://mailman.mit.edu/mailman/listinfo/starcluster
>> >
>
>
>
> --
> Jonathan Taylor
> Dept. of Statistics
> Sequoia Hall, 137
> 390 Serra Mall
> Stanford, CA 94305
> Tel:   650.723.9230
> Fax:   650.725.8977
> Web: http://www-stat.stanford.edu/~jtaylo
>
Received on Wed Jul 27 2011 - 16:45:45 EDT