Re: limiting the number of ipengines started on each node?

From: Fernando Perez <no email>
Date: Thu, 31 Oct 2013 18:03:13 -0700

Hi all,

I just wanted to ping on this same question, in case one of the experts is
around...

Can this be configured at the starcluster level?

Thanks!

f


On Mon, Aug 19, 2013 at 4:43 PM, Jascha Sohl-Dickstein
<jascha_at_stanford.edu>wrote:

> Hello,
>
> Due to limitations on the number of simultaneous active network
> connections, IPython.parallel is unable to handle more than about 250
> engines. (see the discussion at
> http://python.6.x6.nabble.com/IPython-User-ipcluster-Too-many-open-files-tcp-listener-cpp-213-td4978725.html)
>
> On an mx.large instance, with 8 cores, and with an engine started for each
> core, this means that an ipcluster is limited to around 30 nodes. However,
> a single ipengine is capable of using all the cores on a node (for
> instance, if it's doing matrix algebra). I would like to be able to start
> a cluster with fewer ipengines than cores. That is, I would like the
> ipcontroller call on each node to have a "-n [constant number of engines]"
> rather than "-n [number of cores]".
>
> This is probably easy to change in a config file somewhere, so apologies
> if this should be obvious, but I don't (yet) know where to look.
>
> Thank you very much for any help!
>
> -Jascha
>
>
> _______________________________________________
> StarCluster mailing list
> StarCluster_at_mit.edu
> http://mailman.mit.edu/mailman/listinfo/starcluster
>

-- 
Fernando Perez (_at_fperez_org; http://fperez.org)
fperez.net-at-gmail: mailing lists only (I ignore this when swamped!)
fernando.perez-at-berkeley: contact me here for any direct mail

Received on Thu Oct 31 2013 - 21:03:46 EDT