Re: [Starcluster] Using starcluster as a library

From: Austin Godber <no email>
Date: Tue, 18 May 2010 20:14:00 -0400

Hi Justin,
     Thanks for your response, that clears it up, and makes perfect
sense. I think thats a good habit I should develop, avoid leaving
anything in the global namespace. I am a bit new to python and am just
getting the hang of it, so reading though starcluster was a useful exercise.

Nice job on the article and the release.

Austin

On 05/18/2010 03:41 PM, Justin Riley wrote:
> -----BEGIN PGP SIGNED MESSAGE-----
> Hash: SHA1
>
> Hi Austin,
>
> I apologize for the delayed response on this, I was busy putting
> together an article about StarCluster for www.hpcinthecloud.com over the
> past weekend.
>
> Glad to hear you've figured out how to get past the problem with the
> code returning after launching a cluster.
>
> For those that are interested I believe the reason you have to go
> through all of the nodes and close them before exiting is because your
> code is being executed in the global module scope (ie __name__ ==
> "__main__"). In this scope, the node objects are still around and their
> __del__ methods have not been automatically called because garbage
> collection hasn't kicked in. I ran into this problem developing the very
> first version of StarCluster.
>
> If you look at the starcluster.ssh.Connection class you'll see that its
> __del__ method calls close() on the ssh connection automatically. This
> means that as soon as the objects can be garbage collected their delete
> methods will be called and the connections closed automatically.
>
> The way to properly handle this is to write your program in a function
> and call that function from __main__ rather than putting the code
> directly in the global scope like so:
>
>
> def main():
> cfg = starcluster.config.StarClusterConfig(); cfg.load()
> sc = cfg.get_cluster_template('smallcluster', 'mynewcluster')
> if sc.is_valid():
> sc.start(create=True)
>
> if __name__ == "__main__":
> main()
>
> This way the objects are created in the function scope rather than in
> the global scope. This means any objects in the function that are not
> stored somewhere will be destroyed when the function returns. This is
> why you don't see me iterating through nodes and closing them anywhere
> in StarCluster; I don't need to given that garbage collection will close
> them automatically when the process is finished.
>
> I haven't tested the above code but my guess is that it will fix the
> need to close the nodes manually. Please let me know if this isn't the case.
>
> Hope that helps,
>
> ~Justin
>
> On 05/15/2010 05:11 PM, Austin Godber wrote:
>
>> For those following along at home, lsof clearly revealed open SSH
>> connections (actually I only checked with lsof with a single node
>> cluster). So if I loop over all the clusters nodes and do _ssh.close()
>> on each one, the program exits. I didn't see anything in cli.py that
>> does this and there was no other obvious function that appeared to do
>> this. I am not sure if this is intended behavior or I have missed
>> something.
>>
>> Have a good day,
>> Austin
>>
>>
>> PS - The following is a working program. It was tested on a two node
>> cluster, I am not sure whether it was just the master or other nodes
>> that had connections open.
>>
>>
>> #!/usr/bin/env python
>> import starcluster.config
>>
>> if __name__ == "__main__":
>> cfg = starcluster.config.StarClusterConfig(); cfg.load()
>> sc = cfg.get_cluster_template('smallcluster', 'mynewcluster')
>> if sc.is_valid():
>> sc.start(create=True)
>> # close all SSH connections so we can exit:
>> for node in sc._nodes:
>> node._ssh.close()
>>
>>
>> On 05/15/2010 09:22 AM, Austin Godber wrote:
>>
>>> I am playing with using starcluster as a library and its been
>>> straightforward and easy to use so far with the exception of after I
>>> start the cluster the code doesn't return. Cluster launches fine. Are
>>> there some connections I need to close? I have taken a look at
>>> cluster.start and theres nothing apparent.
>>>
>>> Anyone know offhand? Otherwise I will dig into it this afternoon.
>>>
>>> Thanks!
>>>
>>> - Austin
>>>
>>>
>>> The code's trivial and essentially the snippet from the end of cluster.py:
>>>
>>>
>>> #!/usr/bin/env python
>>>
>>> import starcluster.config
>>>
>>> if __name__ == "__main__":
>>> cfg = starcluster.config.StarClusterConfig(); cfg.load()
>>> sc = cfg.get_cluster_template('smallcluster', 'mynewcluster')
>>> if sc.is_valid():
>>> sc.start(create=True)
>>>
>>>
>> _______________________________________________
>> Starcluster mailing list
>> Starcluster_at_mit.edu
>> http://mailman.mit.edu/mailman/listinfo/starcluster
>>
> -----BEGIN PGP SIGNATURE-----
> Version: GnuPG v2.0.14 (GNU/Linux)
> Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/
>
> iEYEARECAAYFAkvy7U0ACgkQ4llAkMfDcrleEACgjXyma7YDeceMm77m74CJLyTg
> Y9MAnRAYEMv2JS5K0HBGVN2DS7HMbZBo
> =zV/k
> -----END PGP SIGNATURE-----
>
>
Received on Tue May 18 2010 - 20:14:03 EDT