Re: Live load balancing node AMI Swapping

From: Rajat Banerjee <no email>
Date: Wed, 25 Oct 2017 14:18:57 -0400

Hi Colby,
Sorry for the misinformation. I looked at the code more and you are correct:

https://github.com/jtriley/StarCluster/blob/develop/starcluster/balancers/sge/__init__.py#L727

calls cluster.add_nodes(need_to_add) and doesn't fill the optional param
'image_id' that would have helped your use case.
https://github.com/jtriley/StarCluster/blob/develop/starcluster/cluster.py#L179

that uses self.node_image_id:
https://github.com/jtriley/StarCluster/blob/develop/starcluster/cluster.py#L938

self.node_image_id is set during cluster's __init__ . I have a vague memory
that this gets stored in the cluster's TAGs in EC2. Do you see a big JSON
blob in the tags?

Sorry for the misinformation before. If you're running in develop mode, you
can modify the call to add_nodes in balancers/sge/__init__.py to take the
image_id from your current config.

Rajat

On Wed, Oct 25, 2017 at 1:30 PM, Colby Taperts <
colby.taperts_at_codewilling.com> wrote:

> Hey Raj,
>
> It does not seem to pick up the new node AMI name and uses the old AMI for
> the node, even though it is not in the ~/.starcluster/config.
>
> My steps to reproduce are:
> 1. Setup config with different master/node AMI's
> 2. Start load balancing
> 3. Stop Load Balancing
> 4. Change node AMI
> 5. Start Load balancing
>
> If I were to terminate the starcluster and start it again then the AMI for
> the cluster will be updated. Is there another ephemeral location for the
> starcluster configuration between starting and terminating the cluster? I
> did look through the source but could not find the reference.
>
>
>
>
>
> On Wed, Oct 25, 2017 at 11:48 AM, Rajat Banerjee <rajatb_at_post.harvard.edu>
> wrote:
>
>> Hi Cody,
>> I don't see why not. I haven't touched the load balancer code in a few
>> years. But from my memory, it reads the config file on startup whenever you
>> run `starcluster loadbalance starcluster-name` and will pick up the new
>> AMI then. So if you kill the last loadbalance process, swap the AMI in the
>> config file, then restart, the load balancer should begin adding the new
>> AMI. The new AMI must be compatible with starcluster obviously, and be
>> ready for starcluster and SGE bootstrapping. This is easy to verify by
>> simply using it to start a new cluster as a test.
>>
>> Give it a try and let us know?
>> Best,
>> Raj
>>
>> On Wed, Oct 25, 2017 at 12:32 PM, Colby Taperts <
>> colby.taperts_at_codewilling.com> wrote:
>>
>>> Hello,
>>>
>>> Would it be possible to change the node AMI after the StarCluster is up?
>>>
>>> for example, use the below setup to initialize the StarCluster
>>>
>>> master_image_id = ami-0330d16a
>>> master_instance_type = m1.small
>>> node_image_id = ami-0330d16a
>>> node_instance_type = m1.small
>>>
>>>
>>> then change the config to:
>>>
>>> master_image_id = ami-0330d16a
>>> master_instance_type = m1.small
>>> node_image_id = ami-0111111 # New AMI name
>>> node_instance_type = m1.small
>>>
>>> Start `starcluster loadbalance starcluster-name`
>>>
>>> and have the new node AMI created when a node is needed
>>>
>>> Or would I need to destroy the cluster if I need to set up the new AMI?
>>>
>>> Thanks so much!
>>> -Colby Taperts
>>>
>>> _______________________________________________
>>> StarCluster mailing list
>>> StarCluster_at_mit.edu
>>> http://mailman.mit.edu/mailman/listinfo/starcluster
>>>
>>>
>>
>
>
> --
> Colby Taperts
> 3488 Brentwood Drive
> <https://maps.google.com/?q=3488+Brentwood+Drive%C2%A0%C2%A0+Suite+104%C2%A0+%C2%A0+Baton+Rouge,+LA+70809&entry=gmail&source=g>
> Suite 104
> <https://maps.google.com/?q=3488+Brentwood+Drive%C2%A0%C2%A0+Suite+104%C2%A0+%C2%A0+Baton+Rouge,+LA+70809&entry=gmail&source=g>
> Baton Rouge, LA 70809
> <https://maps.google.com/?q=3488+Brentwood+Drive%C2%A0%C2%A0+Suite+104%C2%A0+%C2%A0+Baton+Rouge,+LA+70809&entry=gmail&source=g>
>
>
Received on Wed Oct 25 2017 - 14:19:33 EDT