Re: limiting the number of ipengines started on each node?

From: Fernando Perez <no email>
Date: Fri, 1 Nov 2013 17:39:09 -0700

ps - did you submit this for upstreaming into the official repo?


On Fri, Nov 1, 2013 at 9:30 AM, Fernando Perez <fperez.net_at_gmail.com> wrote:

> Hi Niklas,
>
> thanks so much! I'll test it and will report back if there are any issues.
>
> Best,
>
> f
>
>
> On Fri, Nov 1, 2013 at 12:14 AM, Niklas Krumm <nkrumm_at_gmail.com> wrote:
>
>> Hi Jascha and Fernando,
>>
>> I am not an expert. However I did take a look... and while there doesn't
>> seem to be an option built in for this, thanks to your detailed
>> instructions it was easy to parameterize this as an option for the plugin
>> (I named it MAX_ENGINES_PER_NODE). You can see the changes needed here:
>>
>>
>> https://github.com/nkrumm/StarCluster/commit/9670dde20b0ed8fbd8a62ed97650892faea01703
>>
>> and then just add the variable to the [plugin ipcluster] section of your
>> config file, eg:
>> MAX_ENGINES_PER_NODE=1
>>
>> Full disclosure...I have not yet tested these changes! good luck :)
>> ~Nik
>>
>> On Oct 31, 2013, at 6:03 PM, Fernando Perez wrote:
>>
>> Hi all,
>>
>> I just wanted to ping on this same question, in case one of the experts
>> is around...
>>
>> Can this be configured at the starcluster level?
>>
>> Thanks!
>>
>> f
>>
>>
>> On Mon, Aug 19, 2013 at 4:43 PM, Jascha Sohl-Dickstein <
>> jascha_at_stanford.edu> wrote:
>>
>>> Hello,
>>>
>>> Due to limitations on the number of simultaneous active network
>>> connections, IPython.parallel is unable to handle more than about 250
>>> engines. (see the discussion at
>>> http://python.6.x6.nabble.com/IPython-User-ipcluster-Too-many-open-files-tcp-listener-cpp-213-td4978725.html)
>>>
>>> On an mx.large instance, with 8 cores, and with an engine started for
>>> each core, this means that an ipcluster is limited to around 30 nodes.
>>> However, a single ipengine is capable of using all the cores on a node
>>> (for instance, if it's doing matrix algebra). I would like to be able to
>>> start a cluster with fewer ipengines than cores. That is, I would like the
>>> ipcontroller call on each node to have a "-n [constant number of engines]"
>>> rather than "-n [number of cores]".
>>>
>>> This is probably easy to change in a config file somewhere, so apologies
>>> if this should be obvious, but I don't (yet) know where to look.
>>>
>>> Thank you very much for any help!
>>>
>>> -Jascha
>>>
>>>
>>> _______________________________________________
>>> StarCluster mailing list
>>> StarCluster_at_mit.edu
>>> http://mailman.mit.edu/mailman/listinfo/starcluster
>>>
>>
>>
>>
>> --
>> Fernando Perez (_at_fperez_org; http://fperez.org)
>> fperez.net-at-gmail: mailing lists only (I ignore this when swamped!)
>> fernando.perez-at-berkeley: contact me here for any direct mail
>> _______________________________________________
>> StarCluster mailing list
>> StarCluster_at_mit.edu
>> http://mailman.mit.edu/mailman/listinfo/starcluster
>>
>>
>>
>
>
> --
> Fernando Perez (_at_fperez_org; http://fperez.org)
> fperez.net-at-gmail: mailing lists only (I ignore this when swamped!)
> fernando.perez-at-berkeley: contact me here for any direct mail
>

-- 
Fernando Perez (_at_fperez_org; http://fperez.org)
fperez.net-at-gmail: mailing lists only (I ignore this when swamped!)
fernando.perez-at-berkeley: contact me here for any direct mail

Received on Fri Nov 01 2013 - 20:39:42 EDT