StarCluster - Mailing List Archive

Re: loadbalancer race condition with plugin.on_add_node()

From: Justin Riley <no email>
Date: Tue, 15 Mar 2011 17:58:14 -0400

Hash: SHA1

Hi Jeff,

My apologies for the delay. I agree there should be a way to execute a
plugin before a node is added to SGE, however, I think there are also
valid use cases for wanting to be able to run after SGE has been
configured. For instance a plugin may need to submit new jobs/cleanup
old jobs or inspect the current state of the queue. Fortunately there's
a solution that meets both needs and does not require adding more plugin

I've had requests in the past for an option to disable SGE altogether
which I've implemented in the latest github code (see
I'm working now to extract the SGE-related code into a completely
separate plugin that, for now, gets executed by StarCluster by default
unless disable_queue=True in the config. This helps to gradually migrate
SGE to a plugin given that some folks still expect it to be there

With SGE as a plugin you could then completely control the order like so:

[plugin SGE]
setup_class = starcluster.plugins.sge.SGEPlugin

[cluster smallcluster]
# this line tells StarCluster not to configure SGE by default
disable_queue = True
# now add your plugins, including SGE, in the order you want executed
plugins = myplugfirst, SGE, myotherplugin

Does this make sense? I'll send an update to the list once the SGE code
has been converted to a plugin on github...


On 02/21/2011 02:50 PM, Jeff White wrote:
> I believe there is a race condition when nodes are added via the
> loadbalancer. The configured plugins are run after the new node is added
> to SGE. The issue is our plugin needs to do some setup on the new node
> before jobs get submitted to the new node. It would be better if plugins
> were run before the node is available to run jobs. If there is a use
> case for running a plugin after the node is added to SGE, then more
> hooks should be added, e.g. on_add_node_before_sge(),
> on_add_node_after_sge().
> jeff

Version: GnuPG v2.0.17 (GNU/Linux)
Comment: Using GnuPG with Mozilla -

Received on Tue Mar 15 2011 - 17:58:37 EDT
This archive was generated by hypermail 2.3.0.


Sort all by: