Re: limiting the number of ipengines started on each node?
This archive was generated by
thanks so much! I'll test it and will report back if there are any issues.
On Fri, Nov 1, 2013 at 12:14 AM, Niklas Krumm <nkrumm_at_gmail.com> wrote:
> Hi Jascha and Fernando,
> I am not an expert. However I did take a look... and while there doesn't
> seem to be an option built in for this, thanks to your detailed
> instructions it was easy to parameterize this as an option for the plugin
> (I named it MAX_ENGINES_PER_NODE). You can see the changes needed here:
> and then just add the variable to the [plugin ipcluster] section of your
> config file, eg:
> Full disclosure...I have not yet tested these changes! good luck :)
> On Oct 31, 2013, at 6:03 PM, Fernando Perez wrote:
> Hi all,
> I just wanted to ping on this same question, in case one of the experts is
> Can this be configured at the starcluster level?
> On Mon, Aug 19, 2013 at 4:43 PM, Jascha Sohl-Dickstein <
> jascha_at_stanford.edu> wrote:
>> Due to limitations on the number of simultaneous active network
>> connections, IPython.parallel is unable to handle more than about 250
>> engines. (see the discussion at
>> On an mx.large instance, with 8 cores, and with an engine started for
>> each core, this means that an ipcluster is limited to around 30 nodes.
>> However, a single ipengine is capable of using all the cores on a node
>> (for instance, if it's doing matrix algebra). I would like to be able to
>> start a cluster with fewer ipengines than cores. That is, I would like the
>> ipcontroller call on each node to have a "-n [constant number of engines]"
>> rather than "-n [number of cores]".
>> This is probably easy to change in a config file somewhere, so apologies
>> if this should be obvious, but I don't (yet) know where to look.
>> Thank you very much for any help!
>> StarCluster mailing list
> Fernando Perez (_at_fperez_org; http://fperez.org)
> fperez.net-at-gmail: mailing lists only (I ignore this when swamped!)
> fernando.perez-at-berkeley: contact me here for any direct mail
> StarCluster mailing list
Fernando Perez (_at_fperez_org; http://fperez.org)
fperez.net-at-gmail: mailing lists only (I ignore this when swamped!)
fernando.perez-at-berkeley: contact me here for any direct mail
Received on Fri Nov 01 2013 - 12:31:37 EDT