limiting the number of ipengines started on each node?

From: Jascha Sohl-Dickstein <no email>
Date: Mon, 19 Aug 2013 16:43:45 -0700

Hello,

Due to limitations on the number of simultaneous active network connections, IPython.parallel is unable to handle more than about 250 engines. (see the discussion at http://python.6.x6.nabble.com/IPython-User-ipcluster-Too-many-open-files-tcp-listener-cpp-213-td4978725.html )

On an mx.large instance, with 8 cores, and with an engine started for each core, this means that an ipcluster is limited to around 30 nodes. However, a single ipengine is capable of using all the cores on a node (for instance, if it's doing matrix algebra). I would like to be able to start a cluster with fewer ipengines than cores. That is, I would like the ipcontroller call on each node to have a "-n [constant number of engines]" rather than "-n [number of cores]".

This is probably easy to change in a config file somewhere, so apologies if this should be obvious, but I don't (yet) know where to look.

Thank you very much for any help!

-Jascha
Received on Mon Aug 19 2013 - 19:43:50 EDT