Hi Paolo,
Well it probably won't be the *next* version given that I need to put
out a patch release today/tomorrow but it should make it in to the next
major release.
As far as the config goes the only thing that will change vs using a
plugin to do this is that the cluster template itself would now support
a user_data option which would allow you to specify user_data in
multiple templates easily rather than having to create separate plugin
sections for each template in order to specify different scripts. We
should also add a command line flag to the 'start' command that
specifies a single user_data file to pass to all nodes in the cluster.
~Justin
On 2/9/12 9:50 AM, Paolo Di Tommaso wrote:
> Dear Justin, all,
>
> I think this will be a quite useful enhancement to StarCluster.
>
> Actually I tried to solve the problem writing a custom plugin, but the
main limitation was that I had to define the data/file to pass as
user_data in the plugin definition block (as a plugin parameter) of the
configuration file, and this was really unhandy.
>
>
> So I'm really happy to read that next version will include this feature.
>
>
> Thanks,
> Paolo
>
>
>
> On Feb 8, 2012, at 9:00 PM, Justin Riley wrote:
>
> Awesome, thanks! Yea, it may not be universal but at least it allows
> standard shell scripts (and apparently YAML) to work as expected where
> it previously did not so I think it's worth while including. I'm just
> speculating but my guess is scripts are probably the most common
> user-data.
>
> Once we have the code in and tested the next step is to document how
> to use user-data with StarCluster and add it to the user manual...
>
> ~Justin
>
> On 02/07/2012 09:38 PM, Joydeep Sen Sarma wrote:
> >>> ok - i think the patch is pretty simple - let me file a pull
> >>> request. i had added the user_data param as well ..
> >>>
> >>> i think the '#' strategy works for yaml as well (looking one
> >>> message below). i was just afraid it wasn't universal.
> >>>
> >>> On Tue, Feb 7, 2012 at 11:03 PM, Justin Riley
> >>> <justin.t.riley_at_gmail.com <mailto:justin.t.riley_at_gmail.com>>
> >>> wrote:
> >>>
> >>> Hi Paolo/Joydeep,
> >>>
> >>> You're correct that StarCluster currently uses the user-data to
> >>> store node-aliases which at the moment kills the ability to utilize
> >>> user-data scripts. It is possible to run user-scripts on the entire
> >>> cluster fairly easily, however, using a StarCluster plugin. I'm
> >>> happy to provide an example or perhaps include it in a future
> >>> release if you're interested.
> >>>
> >>> The reason for storing the alias in user-data is that I needed some
> >>> way to 'stick' a name to a node at launch time. The new tagging API
> >>> for EC2 is useful but it's still not possible to assign names at
> >>> launch time. My current approach is to use the user-data at launch
> >>> time, then later fetch it from user-data once the instance is up
> >>> and apply an alias tag. This is useful for example if things crash
> >>> somewhere in the middle - the nodes are guaranteed to be named the
> >>> same thing the next time around because of the user-data.
> >>>
> >>> Your solution of effectively 'commenting' the alias line in
> >>> user-data so that other scripts can run is not a bad solution
> >>> actually. I'd be willing to accept a pull request for this and
> >>> include the functionality in a future release. We need to add a
> >>> USER_DATA config parameter to the cluster config that would let
> >>> users point to a user-data script.
> >>>
> >>> Would you be interested in contributing this back? I'm happy to
> >>> help you along with the fork/pull request process if you're
> >>> willing. There's also a contribute guide that gives an overview of
> >>> the process:
> >>>
> >>> http://web.mit.edu/star/cluster/docs/latest/contribute.html
> >>>
> >>> ~Justin
> >>>
> >>>
> >>> On 01/24/2012 11:41 AM, Joydeep Sen Sarma wrote:
> >>>> It doesn't.
> >>>>
> >>>> Currently the user data is used by starcluster itself to store
> >>> node aliases.
> >>>>
> >>>> I had a similar requirement as yours - but in my case - the user
> >>>> data was restricted to shell scripts. So what i did for my use
> >>>> case was
> >>> hack
> >>>> the starcluster code to use the first line of the user data -
> >>> prepended
> >>>> with a '#' for node aliases - and the rest of the user data could
> >>>> be used as a shell script (filled in when launching the cluster
> >>>> from the configuration).
> >>>>
> >>>> it seemed pretty hacky to me - so didn't try to contribute back.
> >>>>
> >>>> there may be other ways of achieving this outcome. for example -
> >>>> one could take the cluster-name and the instance alias (node001
> >>>> etc.) and then map that to a location in s3
> >>>> (s3://urbucket/cluster-name/node001) and then store instance
> >>>> specific user data in that location. the boot scripts can get the
> >>>> cluster name and the instance alias from ec2 apis and then use
> >>>> that to locate the instance data and get rolling.
> >>>>
> >>>> On Tue, Jan 24, 2012 at 9:22 PM, Paolo Di Tommaso
> >>>> <Paolo.DiTommaso_at_crg.eu <mailto:Paolo.DiTommaso_at_crg.eu>
> >>> <mailto:Paolo.DiTommaso_at_crg.eu <mailto:Paolo.DiTommaso_at_crg.eu>>>
> >>> wrote:
> >>>>
> >>>> Dear all,
> >>>>
> >>>> Does StarCluster provide any support for EC2 user data feature?
> >>>>
> >>>> http://www.turnkeylinux.org/blog/ec2-userdata
> >>>>
> >>>>
> >>>> I would like to use this mechanism to specify some extra
> >>>> instance configuration at node boot time.
> >>>>
> >>>> If no, is there any other alternative?
> >>>>
> >>>>
> >>>> Thank you, Paolo Di Tommaso
> >>>>
> >>>> Software Engineer Comparative Bioinformatics Group Centre de
> >>>> Regulacio Genomica (CRG) Dr. Aiguader, 88 08003 Barcelona, Spain
> >>>>
> >>>>
> >>>> _______________________________________________ StarCluster
> >>>> mailing list StarCluster_at_mit.edu <mailto:StarCluster_at_mit.edu>
> >>> <mailto:StarCluster_at_mit.edu <mailto:StarCluster_at_mit.edu>>
> >>>> http://mailman.mit.edu/mailman/listinfo/starcluster
> >>>>
> >>>>
> >>>>
> >>>>
> >>>> _______________________________________________ StarCluster
> >>>> mailing list StarCluster_at_mit.edu <mailto:StarCluster_at_mit.edu>
> >>>> http://mailman.mit.edu/mailman/listinfo/starcluster
> >>>
> >>>
> >>>
> >>>
> >>> This body part will be downloaded on demand.
>
>> _______________________________________________
>> StarCluster mailing list
>> StarCluster_at_mit.edu
>> http://mailman.mit.edu/mailman/listinfo/starcluster
>
>
> _______________________________________________
> StarCluster mailing list
> StarCluster_at_mit.edu
> http://mailman.mit.edu/mailman/listinfo/starcluster
Received on Thu Feb 09 2012 - 10:31:15 EST
This archive was generated by
hypermail 2.3.0.