Re: EC2 User data

From: Paolo Di Tommaso <no email>
Date: Thu, 9 Feb 2012 15:50:05 +0100

Dear Justin, all,

I think this will be a quite useful enhancement to StarCluster.

Actually I tried to solve the problem writing a custom plugin, but the main limitation was that I had to define the data/file to pass as user_data in the plugin definition block (as a plugin parameter) of the configuration file, and this was really unhandy.


So I'm really happy to read that next version will include this feature.


Thanks,
Paolo
   


On Feb 8, 2012, at 9:00 PM, Justin Riley wrote:

> -----BEGIN PGP SIGNED MESSAGE-----
> Hash: SHA1
>
> Awesome, thanks! Yea, it may not be universal but at least it allows
> standard shell scripts (and apparently YAML) to work as expected where
> it previously did not so I think it's worth while including. I'm just
> speculating but my guess is scripts are probably the most common
> user-data.
>
> Once we have the code in and tested the next step is to document how
> to use user-data with StarCluster and add it to the user manual...
>
> ~Justin
>
> On 02/07/2012 09:38 PM, Joydeep Sen Sarma wrote:
>> ok - i think the patch is pretty simple - let me file a pull
>> request. i had added the user_data param as well ..
>>
>> i think the '#' strategy works for yaml as well (looking one
>> message below). i was just afraid it wasn't universal.
>>
>> On Tue, Feb 7, 2012 at 11:03 PM, Justin Riley
>> <justin.t.riley_at_gmail.com <mailto:justin.t.riley_at_gmail.com>>
>> wrote:
>>
>> Hi Paolo/Joydeep,
>>
>> You're correct that StarCluster currently uses the user-data to
>> store node-aliases which at the moment kills the ability to utilize
>> user-data scripts. It is possible to run user-scripts on the entire
>> cluster fairly easily, however, using a StarCluster plugin. I'm
>> happy to provide an example or perhaps include it in a future
>> release if you're interested.
>>
>> The reason for storing the alias in user-data is that I needed some
>> way to 'stick' a name to a node at launch time. The new tagging API
>> for EC2 is useful but it's still not possible to assign names at
>> launch time. My current approach is to use the user-data at launch
>> time, then later fetch it from user-data once the instance is up
>> and apply an alias tag. This is useful for example if things crash
>> somewhere in the middle - the nodes are guaranteed to be named the
>> same thing the next time around because of the user-data.
>>
>> Your solution of effectively 'commenting' the alias line in
>> user-data so that other scripts can run is not a bad solution
>> actually. I'd be willing to accept a pull request for this and
>> include the functionality in a future release. We need to add a
>> USER_DATA config parameter to the cluster config that would let
>> users point to a user-data script.
>>
>> Would you be interested in contributing this back? I'm happy to
>> help you along with the fork/pull request process if you're
>> willing. There's also a contribute guide that gives an overview of
>> the process:
>>
>> http://web.mit.edu/star/cluster/docs/latest/contribute.html
>>
>> ~Justin
>>
>>
>> On 01/24/2012 11:41 AM, Joydeep Sen Sarma wrote:
>>> It doesn't.
>>>
>>> Currently the user data is used by starcluster itself to store
>> node aliases.
>>>
>>> I had a similar requirement as yours - but in my case - the user
>>> data was restricted to shell scripts. So what i did for my use
>>> case was
>> hack
>>> the starcluster code to use the first line of the user data -
>> prepended
>>> with a '#' for node aliases - and the rest of the user data could
>>> be used as a shell script (filled in when launching the cluster
>>> from the configuration).
>>>
>>> it seemed pretty hacky to me - so didn't try to contribute back.
>>>
>>> there may be other ways of achieving this outcome. for example -
>>> one could take the cluster-name and the instance alias (node001
>>> etc.) and then map that to a location in s3
>>> (s3://urbucket/cluster-name/node001) and then store instance
>>> specific user data in that location. the boot scripts can get the
>>> cluster name and the instance alias from ec2 apis and then use
>>> that to locate the instance data and get rolling.
>>>
>>> On Tue, Jan 24, 2012 at 9:22 PM, Paolo Di Tommaso
>>> <Paolo.DiTommaso_at_crg.eu <mailto:Paolo.DiTommaso_at_crg.eu>
>> <mailto:Paolo.DiTommaso_at_crg.eu <mailto:Paolo.DiTommaso_at_crg.eu>>>
>> wrote:
>>>
>>> Dear all,
>>>
>>> Does StarCluster provide any support for EC2 user data feature?
>>>
>>> http://www.turnkeylinux.org/blog/ec2-userdata
>>>
>>>
>>> I would like to use this mechanism to specify some extra
>>> instance configuration at node boot time.
>>>
>>> If no, is there any other alternative?
>>>
>>>
>>> Thank you, Paolo Di Tommaso
>>>
>>> Software Engineer Comparative Bioinformatics Group Centre de
>>> Regulacio Genomica (CRG) Dr. Aiguader, 88 08003 Barcelona, Spain
>>>
>>>
>>> _______________________________________________ StarCluster
>>> mailing list StarCluster_at_mit.edu <mailto:StarCluster_at_mit.edu>
>> <mailto:StarCluster_at_mit.edu <mailto:StarCluster_at_mit.edu>>
>>> http://mailman.mit.edu/mailman/listinfo/starcluster
>>>
>>>
>>>
>>>
>>> _______________________________________________ StarCluster
>>> mailing list StarCluster_at_mit.edu <mailto:StarCluster_at_mit.edu>
>>> http://mailman.mit.edu/mailman/listinfo/starcluster
>>
>>
>>
>>
>> This body part will be downloaded on demand.
>
> -----BEGIN PGP SIGNATURE-----
> Version: GnuPG v2.0.17 (GNU/Linux)
> Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/
>
> iEYEARECAAYFAk8y1HoACgkQ4llAkMfDcrl64wCglpNHQYKfAjfrubiBTVXSxCDN
> ersAnRdUZ94ku8OcWvxM7r4IXzIn8jc1
> =UZq3
> -----END PGP SIGNATURE-----
> _______________________________________________
> StarCluster mailing list
> StarCluster_at_mit.edu
> http://mailman.mit.edu/mailman/listinfo/starcluster
Received on Thu Feb 09 2012 - 09:50:10 EST