Re: Live load balancing node AMI Swapping

From: Colby Taperts <no email>
Date: Wed, 25 Oct 2017 13:24:12 -0500

I am super happy you were able to recall this and let me know. Thank you so
much!

On Wed, Oct 25, 2017 at 1:18 PM, Rajat Banerjee <rajatb_at_post.harvard.edu>
wrote:

> Hi Colby,
> Sorry for the misinformation. I looked at the code more and you are
> correct:
>
> https://github.com/jtriley/StarCluster/blob/develop/
> starcluster/balancers/sge/__init__.py#L727
>
> calls cluster.add_nodes(need_to_add) and doesn't fill the optional param
> 'image_id' that would have helped your use case.
> https://github.com/jtriley/StarCluster/blob/develop/
> starcluster/cluster.py#L179
>
> that uses self.node_image_id:
> https://github.com/jtriley/StarCluster/blob/develop/
> starcluster/cluster.py#L938
>
> self.node_image_id is set during cluster's __init__ . I have a vague
> memory that this gets stored in the cluster's TAGs in EC2. Do you see a big
> JSON blob in the tags?
>
> Sorry for the misinformation before. If you're running in develop mode,
> you can modify the call to add_nodes in balancers/sge/__init__.py to take
> the image_id from your current config.
>
> Rajat
>
> On Wed, Oct 25, 2017 at 1:30 PM, Colby Taperts <
> colby.taperts_at_codewilling.com> wrote:
>
>> Hey Raj,
>>
>> It does not seem to pick up the new node AMI name and uses the old AMI
>> for the node, even though it is not in the ~/.starcluster/config.
>>
>> My steps to reproduce are:
>> 1. Setup config with different master/node AMI's
>> 2. Start load balancing
>> 3. Stop Load Balancing
>> 4. Change node AMI
>> 5. Start Load balancing
>>
>> If I were to terminate the starcluster and start it again then the AMI
>> for the cluster will be updated. Is there another ephemeral location for
>> the starcluster configuration between starting and terminating the cluster?
>> I did look through the source but could not find the reference.
>>
>>
>>
>>
>>
>> On Wed, Oct 25, 2017 at 11:48 AM, Rajat Banerjee <rajatb_at_post.harvard.edu
>> > wrote:
>>
>>> Hi Cody,
>>> I don't see why not. I haven't touched the load balancer code in a few
>>> years. But from my memory, it reads the config file on startup whenever you
>>> run `starcluster loadbalance starcluster-name` and will pick up the new
>>> AMI then. So if you kill the last loadbalance process, swap the AMI in the
>>> config file, then restart, the load balancer should begin adding the new
>>> AMI. The new AMI must be compatible with starcluster obviously, and be
>>> ready for starcluster and SGE bootstrapping. This is easy to verify by
>>> simply using it to start a new cluster as a test.
>>>
>>> Give it a try and let us know?
>>> Best,
>>> Raj
>>>
>>> On Wed, Oct 25, 2017 at 12:32 PM, Colby Taperts <
>>> colby.taperts_at_codewilling.com> wrote:
>>>
>>>> Hello,
>>>>
>>>> Would it be possible to change the node AMI after the StarCluster is up?
>>>>
>>>> for example, use the below setup to initialize the StarCluster
>>>>
>>>> master_image_id = ami-0330d16a
>>>> master_instance_type = m1.small
>>>> node_image_id = ami-0330d16a
>>>> node_instance_type = m1.small
>>>>
>>>>
>>>> then change the config to:
>>>>
>>>> master_image_id = ami-0330d16a
>>>> master_instance_type = m1.small
>>>> node_image_id = ami-0111111 # New AMI name
>>>> node_instance_type = m1.small
>>>>
>>>> Start `starcluster loadbalance starcluster-name`
>>>>
>>>> and have the new node AMI created when a node is needed
>>>>
>>>> Or would I need to destroy the cluster if I need to set up the new AMI?
>>>>
>>>> Thanks so much!
>>>> -Colby Taperts
>>>>
>>>> _______________________________________________
>>>> StarCluster mailing list
>>>> StarCluster_at_mit.edu
>>>> http://mailman.mit.edu/mailman/listinfo/starcluster
>>>>
>>>>
>>>
>>
>>
>> --
>> Colby Taperts
>> 3488 Brentwood Drive
>> <https://maps.google.com/?q=3488+Brentwood+Drive%C2%A0%C2%A0+Suite+104%C2%A0+%C2%A0+Baton+Rouge,+LA+70809&entry=gmail&source=g>
>> Suite 104
>> <https://maps.google.com/?q=3488+Brentwood+Drive%C2%A0%C2%A0+Suite+104%C2%A0+%C2%A0+Baton+Rouge,+LA+70809&entry=gmail&source=g>
>> Baton Rouge, LA 70809
>> <https://maps.google.com/?q=3488+Brentwood+Drive%C2%A0%C2%A0+Suite+104%C2%A0+%C2%A0+Baton+Rouge,+LA+70809&entry=gmail&source=g>
>>
>>
>
>

-- 
  Colby Taperts
  3488 Brentwood Drive
  Suite 104
  Baton Rouge, LA 70809

Received on Wed Oct 25 2017 - 14:25:02 EDT