-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1
Hi Austin,
I apologize for the delayed response on this, I was busy putting
together an article about StarCluster for www.hpcinthecloud.com over the
past weekend.
Glad to hear you've figured out how to get past the problem with the
code returning after launching a cluster.
For those that are interested I believe the reason you have to go
through all of the nodes and close them before exiting is because your
code is being executed in the global module scope (ie __name__ ==
"__main__"). In this scope, the node objects are still around and their
__del__ methods have not been automatically called because garbage
collection hasn't kicked in. I ran into this problem developing the very
first version of StarCluster.
If you look at the starcluster.ssh.Connection class you'll see that its
__del__ method calls close() on the ssh connection automatically. This
means that as soon as the objects can be garbage collected their delete
methods will be called and the connections closed automatically.
The way to properly handle this is to write your program in a function
and call that function from __main__ rather than putting the code
directly in the global scope like so:
def main():
cfg = starcluster.config.StarClusterConfig(); cfg.load()
sc = cfg.get_cluster_template('smallcluster', 'mynewcluster')
if sc.is_valid():
sc.start(create=True)
if __name__ == "__main__":
main()
This way the objects are created in the function scope rather than in
the global scope. This means any objects in the function that are not
stored somewhere will be destroyed when the function returns. This is
why you don't see me iterating through nodes and closing them anywhere
in StarCluster; I don't need to given that garbage collection will close
them automatically when the process is finished.
I haven't tested the above code but my guess is that it will fix the
need to close the nodes manually. Please let me know if this isn't the case.
Hope that helps,
~Justin
On 05/15/2010 05:11 PM, Austin Godber wrote:
> For those following along at home, lsof clearly revealed open SSH
> connections (actually I only checked with lsof with a single node
> cluster). So if I loop over all the clusters nodes and do _ssh.close()
> on each one, the program exits. I didn't see anything in cli.py that
> does this and there was no other obvious function that appeared to do
> this. I am not sure if this is intended behavior or I have missed
> something.
>
> Have a good day,
> Austin
>
>
> PS - The following is a working program. It was tested on a two node
> cluster, I am not sure whether it was just the master or other nodes
> that had connections open.
>
>
> #!/usr/bin/env python
> import starcluster.config
>
> if __name__ == "__main__":
> cfg = starcluster.config.StarClusterConfig(); cfg.load()
> sc = cfg.get_cluster_template('smallcluster', 'mynewcluster')
> if sc.is_valid():
> sc.start(create=True)
> # close all SSH connections so we can exit:
> for node in sc._nodes:
> node._ssh.close()
>
>
> On 05/15/2010 09:22 AM, Austin Godber wrote:
>> I am playing with using starcluster as a library and its been
>> straightforward and easy to use so far with the exception of after I
>> start the cluster the code doesn't return. Cluster launches fine. Are
>> there some connections I need to close? I have taken a look at
>> cluster.start and theres nothing apparent.
>>
>> Anyone know offhand? Otherwise I will dig into it this afternoon.
>>
>> Thanks!
>>
>> - Austin
>>
>>
>> The code's trivial and essentially the snippet from the end of cluster.py:
>>
>>
>> #!/usr/bin/env python
>>
>> import starcluster.config
>>
>> if __name__ == "__main__":
>> cfg = starcluster.config.StarClusterConfig(); cfg.load()
>> sc = cfg.get_cluster_template('smallcluster', 'mynewcluster')
>> if sc.is_valid():
>> sc.start(create=True)
>>
> _______________________________________________
> Starcluster mailing list
> Starcluster_at_mit.edu
> http://mailman.mit.edu/mailman/listinfo/starcluster
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v2.0.14 (GNU/Linux)
Comment: Using GnuPG with Mozilla -
http://enigmail.mozdev.org/
iEYEARECAAYFAkvy7U0ACgkQ4llAkMfDcrleEACgjXyma7YDeceMm77m74CJLyTg
Y9MAnRAYEMv2JS5K0HBGVN2DS7HMbZBo
=zV/k
-----END PGP SIGNATURE-----
Received on Tue May 18 2010 - 15:41:06 EDT