I tried these new plugins which seemed to install fine but something's still
not quite right -- trying to get a Client hangs:
sgeadmin_at_master:~$ ipython
Python 2.6.5 (r265:79063, Apr 16 2010, 13:09:56)
Type "copyright", "credits" or "license" for more information.
IPython 0.11.dev -- An enhanced Interactive Python.
? -> Introduction and overview of IPython's features.
%quickref -> Quick reference.
help -> Python's own help system.
object? -> Details about 'object', use 'object??' for extra details.
In [1]: from IPython.parallel import Client
In [2]: rc = Client('starcluster.json')
---------------------------------------------------------------------------
AssertionError Traceback (most recent call last)
/home/sgeadmin/<ipython-input-2-c67e0a34263a> in <module>()
----> 1 rc = Client('starcluster.json')
/usr/local/lib/python2.6/dist-packages/ipython-0.11.dev-py2.6.egg/IPython/parallel/client/client.pyc
in __init__(self, url_or_file, profile, profile_dir, ipython_dir, context,
debug, exec_key, sshserver, sshkey, password, paramiko, timeout,
**extra_args)
313 if self._cd:
314 url_or_file =
os.path.join(self._cd.security_dir, url_or_file)
--> 315 assert os.path.exists(url_or_file), "Not a valid
connection file or url: %r"%url_or_file
316 with open(url_or_file) as f:
317 cfg = json.loads(f.read())
AssertionError: Not a valid connection file or url:
u'/home/sgeadmin/.ipython/profile_default/security/starcluster.json'
---------
There are only two JSON files on the master node:
sgeadmin_at_master:~$ !fi
find . -name \*json
./.ipython/profile_default/security/ipcontroller-client.json
./.ipython/profile_default/security/ipcontroller-engine.json
-- Jonathan
On Wed, Jul 27, 2011 at 12:31 PM, MinRK <benjaminrk_at_gmail.com> wrote:
> And I've been using an updated version of the plugin to work with
> IPython 0.11 (parallel code has been completely rewritten, and is much
> improved for certain activities). My plugins:
>
> https://github.com/minrk/starclusterplugins
>
> -MinRK
>
> On Wed, Jul 27, 2011 at 12:14, Justin Riley <jtriley_at_mit.edu> wrote:
> > -----BEGIN PGP SIGNED MESSAGE-----
> > Hash: SHA1
> >
> > Hi Jonathan,
> >
> > The plugin still needs documenting but here's the basic flow:
> >
> > Put the plugin in the config:
> >
> > [plugin ipcluster]
> > setup_class = starcluster.plugins.ipcluster.IPCluster
> >
> > Then put the plugin in a cluster template:
> >
> > [cluster smallcluster]
> > ...
> > plugins = ipcluster
> >
> > Then start a cluster:
> >
> > $ starcluster start myipcluster
> >
> > If you didn't put the plugin in the default cluster template (specified
> > by default_template in the [global] section) you can specify the
> > template like so:
> >
> > $ starcluster start -c smallcluster myipcluster
> >
> > This will start a cluster and run the ipcluster plugin which will setup
> > an ipcluster session for your CLUSTER_USER (defaults to sgeadmin). Then
> > to use ipcluster:
> >
> > $ starcluster sshmaster -u sgeadmin myipcluster
> > sgeadmin_at_master~ ipython
> > [~]> from IPython.kernel.client import MultiEngineClient
> > [~]> mec = MultiEngineClient()
> > [~]> mec.get_ids()
> > [0, 1]
> >
> > In the above IPython session we see that we have two nodes in our
> > ipcluster. In general, you will have as many clients as you have nodes
> > in your StarCluster.
> >
> > After you have a MultiEngineClient instance you can then do things like:
> >
> > [~]> mec.map(lambda x: x**10, range(32)) # happens in parallel
> > [~]> mec.execute('d=10; print d') # executes on all nodes in parallel
> >
> > For more details see:
> >
> >
> http://ipython.org/ipython-doc/rel-0.10.2/html/parallel/parallel_multiengine.html#quick-and-easy-parallelism
> >
> > HTH,
> >
> > ~Justin
> >
> > On 07/26/2011 11:56 PM, Jonathan Taylor wrote:
> >> Hi,
> >>
> >> I was wondering if someone had any examples / documentations on how
> >> the IPCluster plugin works in StarCluster.
> >>
> >> Thanks,
> >>
> >> Jonathan
> >>
> >> -- Jonathan Taylor Dept. of Statistics Sequoia Hall, 137
> >> 390 Serra Mall Stanford, CA 94305 Tel: 650.723.9230 Fax:
> >> 650.725.8977 Web: http://www-stat.stanford.edu/~jtaylo
> >
> > -----BEGIN PGP SIGNATURE-----
> > Version: GnuPG v2.0.17 (GNU/Linux)
> > Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/
> >
> > iEYEARECAAYFAk4wY50ACgkQ4llAkMfDcrl5IwCeINhKGiM7nQgIEN7Bec0cQsmJ
> > yn4An3kpz6CPlxfNt0t/tHKtofPG9Nwy
> > =jnrj
> > -----END PGP SIGNATURE-----
> > _______________________________________________
> > StarCluster mailing list
> > StarCluster_at_mit.edu
> > http://mailman.mit.edu/mailman/listinfo/starcluster
> >
>
--
Jonathan Taylor
Dept. of Statistics
Sequoia Hall, 137
390 Serra Mall
Stanford, CA 94305
Tel: 650.723.9230
Fax: 650.725.8977
Web: http://www-stat.stanford.edu/~jtaylo
Received on Wed Jul 27 2011 - 16:38:23 EDT