Re: [Starcluster] Using starcluster as a library
This archive was generated by
Thanks for your response, that clears it up, and makes perfect
sense. I think thats a good habit I should develop, avoid leaving
anything in the global namespace. I am a bit new to python and am just
getting the hang of it, so reading though starcluster was a useful exercise.
Nice job on the article and the release.
On 05/18/2010 03:41 PM, Justin Riley wrote:
> -----BEGIN PGP SIGNED MESSAGE-----
> Hash: SHA1
> Hi Austin,
> I apologize for the delayed response on this, I was busy putting
> together an article about StarCluster for www.hpcinthecloud.com over the
> past weekend.
> Glad to hear you've figured out how to get past the problem with the
> code returning after launching a cluster.
> For those that are interested I believe the reason you have to go
> through all of the nodes and close them before exiting is because your
> code is being executed in the global module scope (ie __name__ ==
> "__main__"). In this scope, the node objects are still around and their
> __del__ methods have not been automatically called because garbage
> collection hasn't kicked in. I ran into this problem developing the very
> first version of StarCluster.
> If you look at the starcluster.ssh.Connection class you'll see that its
> __del__ method calls close() on the ssh connection automatically. This
> means that as soon as the objects can be garbage collected their delete
> methods will be called and the connections closed automatically.
> The way to properly handle this is to write your program in a function
> and call that function from __main__ rather than putting the code
> directly in the global scope like so:
> def main():
> cfg = starcluster.config.StarClusterConfig(); cfg.load()
> sc = cfg.get_cluster_template('smallcluster', 'mynewcluster')
> if sc.is_valid():
> if __name__ == "__main__":
> This way the objects are created in the function scope rather than in
> the global scope. This means any objects in the function that are not
> stored somewhere will be destroyed when the function returns. This is
> why you don't see me iterating through nodes and closing them anywhere
> in StarCluster; I don't need to given that garbage collection will close
> them automatically when the process is finished.
> I haven't tested the above code but my guess is that it will fix the
> need to close the nodes manually. Please let me know if this isn't the case.
> Hope that helps,
> On 05/15/2010 05:11 PM, Austin Godber wrote:
>> For those following along at home, lsof clearly revealed open SSH
>> connections (actually I only checked with lsof with a single node
>> cluster). So if I loop over all the clusters nodes and do _ssh.close()
>> on each one, the program exits. I didn't see anything in cli.py that
>> does this and there was no other obvious function that appeared to do
>> this. I am not sure if this is intended behavior or I have missed
>> Have a good day,
>> PS - The following is a working program. It was tested on a two node
>> cluster, I am not sure whether it was just the master or other nodes
>> that had connections open.
>> #!/usr/bin/env python
>> import starcluster.config
>> if __name__ == "__main__":
>> cfg = starcluster.config.StarClusterConfig(); cfg.load()
>> sc = cfg.get_cluster_template('smallcluster', 'mynewcluster')
>> if sc.is_valid():
>> # close all SSH connections so we can exit:
>> for node in sc._nodes:
>> On 05/15/2010 09:22 AM, Austin Godber wrote:
>>> I am playing with using starcluster as a library and its been
>>> straightforward and easy to use so far with the exception of after I
>>> start the cluster the code doesn't return. Cluster launches fine. Are
>>> there some connections I need to close? I have taken a look at
>>> cluster.start and theres nothing apparent.
>>> Anyone know offhand? Otherwise I will dig into it this afternoon.
>>> - Austin
>>> The code's trivial and essentially the snippet from the end of cluster.py:
>>> #!/usr/bin/env python
>>> import starcluster.config
>>> if __name__ == "__main__":
>>> cfg = starcluster.config.StarClusterConfig(); cfg.load()
>>> sc = cfg.get_cluster_template('smallcluster', 'mynewcluster')
>>> if sc.is_valid():
>> Starcluster mailing list
> -----BEGIN PGP SIGNATURE-----
> Version: GnuPG v2.0.14 (GNU/Linux)
> Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/
> -----END PGP SIGNATURE-----
Received on Tue May 18 2010 - 20:14:03 EDT