StarCluster - Mailing List Archive

Re: limiting the number of ipengines started on each node?

From: Fernando Perez <no email>
Date: Thu, 31 Oct 2013 18:03:13 -0700

Hi all,

I just wanted to ping on this same question, in case one of the experts is

Can this be configured at the starcluster level?



On Mon, Aug 19, 2013 at 4:43 PM, Jascha Sohl-Dickstein

> Hello,
> Due to limitations on the number of simultaneous active network
> connections, IPython.parallel is unable to handle more than about 250
> engines. (see the discussion at
> On an mx.large instance, with 8 cores, and with an engine started for each
> core, this means that an ipcluster is limited to around 30 nodes. However,
> a single ipengine is capable of using all the cores on a node (for
> instance, if it's doing matrix algebra). I would like to be able to start
> a cluster with fewer ipengines than cores. That is, I would like the
> ipcontroller call on each node to have a "-n [constant number of engines]"
> rather than "-n [number of cores]".
> This is probably easy to change in a config file somewhere, so apologies
> if this should be obvious, but I don't (yet) know where to look.
> Thank you very much for any help!
> -Jascha
> _______________________________________________
> StarCluster mailing list

Fernando Perez (_at_fperez_org; mailing lists only (I ignore this when swamped!)
fernando.perez-at-berkeley: contact me here for any direct mail
Received on Thu Oct 31 2013 - 21:03:46 EDT
This archive was generated by hypermail 2.3.0.


Sort all by: