Re: StarCluster Plugins

From: Cory Dolphin <no email>
Date: Fri, 28 Mar 2014 13:56:46 -0400

Is there a central repository of user-contributed plugins anywhere?
Regardless, I have both a contribution and a question:

1. Has anyone experimented with using SGE profile in an IPython cluster? I
wish to enable starcluster's load balancing, but since it depends on SGE's
queue length, I do not believe it will behave as expected.
2. I had some issues with remote engines dying during a particular job, so
I edited the ipcluster plugin to use supervisord to watch the processes, if
anyone else finds it useful, it is on Github with the creative name
StarClusterReliableIPCluster<https://github.com/wcdolphin/StarClusterReliableIPCluster>

Cheers,
Cory


On Mon, Mar 17, 2014 at 3:29 PM, Sergio Mafra <sergiohmafra_at_gmail.com>wrote:

> Good move Dmitry..
>
> If someone has more plugins who wants to share.. be welcomed here.
>
> All best,
>
> Sergio
>
>
> On Mon, Mar 17, 2014 at 3:45 PM, Dmitry Serenbrennikov <dmitry_at_adchemy.com
> > wrote:
>
>> Same here.
>>
>>
>> Here's another plugin that I have written. It integrates Chef such that
>> master nodes are configured as chef clients/nodes.
>>
>> https://gist.github.com/nitecoder/9605618
>> <https://gist.github.com/nitecoder/9605618>
>> This is a Chef plugin for StarCluster. It's really my first ever
>> StarCluster plugin, plus I'm very new to Chef as well. So please take with
>> a grain of salt. Any feedback is welcome! Especially, a few things are not
>> quite to my liking: chef's authentication system seems to presume that
>> validation.pem file is widely distributed. I'm attempting to limit the need
>> for this file being everywhere (by u
>> This is a Chef plugin for StarCluster. It's really my first ever
>> StarCluster plugin, plus I'm very new to Chef as well. So please take with
>> a grain of salt. Any feedback is welcome! Especially, a few things are not
>> quite to my liking: chef's authentication system seems to presume that
>> validation.pem file is widely distributed. I'm attempting to limit the need
>> for this file being everywhere (by using chef bootstrap). It works, but
>> doesn't seem like the right approach. Am I just misunderstanding Chef
>> somehow? currently it's not possible to tell if starcluster node is
>> terminated or just stopped. If one deletes chef client on stop, then start
>> doesn't work. I chose to delete the node but leave the client. But this
>> also doesn't seem like the right approach. - Gist is a simple way to share
>> snippets of text and code with others.
>> Read more... <https://gist.github.com/nitecoder/9605618>
>> It's kind of clunky but seems to work. In particular, I've attempted
>> to de-provision the node (if not the client) when cluster terminates. But
>> I'm new to Chef and StarCluster both, so this could be totally missing the
>> way Chef is supposed to operate.
>>
>>
>> Any feedback is welcome!
>>
>>
>> Thanks!
>>
>> -Dmitry
>>
>>
>>
>>
>> ------------------------------
>> *From:* starcluster-bounces_at_mit.edu <starcluster-bounces_at_mit.edu> on
>> behalf of Jacob Barhak <jacob.barhak_at_gmail.com>
>> *Sent:* Sunday, March 16, 2014 9:21 PM
>> *To:* Sergio Mafra
>> *Cc:* starcluster_at_mit.edu
>> *Subject:* Re: [StarCluster] StarCluster Plugins
>>
>> Thanks Sergio,
>>
>> You have a second there.
>>
>> People were very helpful to me in the past when I asked questions and
>> they suggested plugins that solved my problems. However, I still do not
>> understand too much about them to the point I can write one on my own, and
>> I saw others in the list that ask similar questions about plugins.
>>
>> The real question is if anyone who understands enough is willing to
>> take this effort of a centralized repository?
>>
>> If someone does choose to invest in this it will help for sure and may
>> grow to be a useful tool set.
>>
>> Jacob
>>
>>
>>
>> On Sat, Mar 15, 2014 at 3:08 PM, Sergio Mafra <sergiohmafra_at_gmail.com>wrote:
>>
>>> Hi fellows,
>>>
>>> I´ve got no doubt that StarCluster is a fantastic tool. This helps
>>> lots of people on creating and managing an array of clusters.
>>> But there is more.. the plugins.. They can help us improve what is
>>> really great.
>>> The problem is knowing how to really develop and find them. They are
>>> scattered on internet.
>>> I think that if someone (loving soul) creates a catalog of plugins,
>>> starting with "How to develop a StarCluster plugin for dummies"... this
>>> should be something. Better if Justin and Rayson cloud enhance the
>>> StarCluster´s site (
>>> http://star.mit.edu/cluster/docs/latest/manual/plugins.html#plugin-system)
>>> to be like a micro git.
>>> Tell me what do you think.
>>>
>>> Sergio
>>>
>>> _______________________________________________
>>> StarCluster mailing list
>>> StarCluster_at_mit.edu
>>> http://mailman.mit.edu/mailman/listinfo/starcluster
>>>
>>>
>>
>
> _______________________________________________
> StarCluster mailing list
> StarCluster_at_mit.edu
> http://mailman.mit.edu/mailman/listinfo/starcluster
>
>
Received on Fri Mar 28 2014 - 13:56:48 EDT