limiting the number of ipengines started on each node?
This archive was generated by
Due to limitations on the number of simultaneous active network connections, IPython.parallel is unable to handle more than about 250 engines. (see the discussion at http://python.6.x6.nabble.com/IPython-User-ipcluster-Too-many-open-files-tcp-listener-cpp-213-td4978725.html
On an mx.large instance, with 8 cores, and with an engine started for each core, this means that an ipcluster is limited to around 30 nodes. However, a single ipengine is capable of using all the cores on a node (for instance, if it's doing matrix algebra). I would like to be able to start a cluster with fewer ipengines than cores. That is, I would like the ipcontroller call on each node to have a "-n [constant number of engines]" rather than "-n [number of cores]".
This is probably easy to change in a config file somewhere, so apologies if this should be obvious, but I don't (yet) know where to look.
Thank you very much for any help!
Received on Mon Aug 19 2013 - 19:43:50 EDT