Re: Using MIT StarCluster on VPC

From: Rayson Ho <no email>
Date: Thu, 13 Mar 2014 20:27:01 -0400

On Thu, Mar 13, 2014 at 8:09 PM, Sergio Mafra <sergiohmafra_at_gmail.com> wrote:
> Thanks for the explanation and tips for this situation that should be a
> common place when StarCluster moves to private subnets.
> Other option could be assigning a secondary IP to master using EC2 cli,
> where this IP is fixed.

Yes, that would work too -- you can use the AWS management console to
attach an ENI (Elastic Network Interface), or an Elastic IP if that
VPC subnet is a public subnet.

You shouldn't need to change anything with the ENI approach, and users
will log directly onto the master node and submit jobs. However, if
you want to block SSH login, then you will need to make sure that the
qmaster is listening on the additional ENI's IP:

http://gridscheduler.sourceforge.net/howto/multi_intrfcs.html

Rayson

==================================================
Open Grid Scheduler - The Official Open Source Grid Engine
http://gridscheduler.sourceforge.net/
http://gridscheduler.sourceforge.net/GridEngine/GridEngineCloud.html


>
> All best,
>
> Sérgio
>
> Em quinta-feira, 13 de março de 2014, Dmitry Serenbrennikov
> <dmitry_at_adchemy.com> escreveu:
>
>> Hi Sergio,
>>
>> To add to Rayson's answer, perhaps you can consider using DNS entries
>> instead of fixed IP addresses?
>> In our case, that's what we are doing. (Honestly, I didn't even know you
>> could request specific IPs...)
>>
>> I wrote a small plugin to have starcluster automatically maintain Route53
>> entries for each master and the nodes.
>> Feel free to use it as is or modify:
>> https://gist.github.com/nitecoder/9536584#file-starcluster-route53-plugin
>>
>> By the way, this is only my second plugin ever, so I would very much
>> appreciate any critique / improvements from the list!
>>
>> Cheers!
>> -Dmitry
>>
>>
>> ________________________________________
>> From: starcluster-bounces_at_mit.edu <starcluster-bounces_at_mit.edu> on behalf
>> of Rayson Ho <raysonlogin_at_gmail.com>
>> Sent: Thursday, March 13, 2014 1:09 PM
>> To: Sergio Mafra
>> Cc: starcluster_at_mit.edu
>> Subject: Re: [StarCluster] Using MIT StarCluster on VPC
>>
>> On Thu, Mar 13, 2014 at 3:34 PM, Sergio Mafra <sergiohmafra_at_gmail.com>
>> wrote:
>> > Since we´d like to assign to each master, a specific private ip, I´d
>> > like to
>> > know if this can be done in StarCluster or by CLI commands. so my users
>> > can
>> > always connect in their machines using the same procedure.
>>
>> Short answer: currently there is no support for that in StarCluster.
>>
>> Longer answer: While the boto (AWS Python) run_instances() API does
>> offer support for assigning specific private IP address (by using
>> parameter "private_ip_address") in a VPC subnet, there is a limitation
>> of starting only 1 instance when private_ip_address is specified.
>>
>> Thus, if you want to do it that way, the StarCluster code will need to
>> pass in the "private_ip_address" parameter when starting the master
>> node. Then the slave nodes will need to be started in another
>> run_instances() call. It should be doable (with some code changes) as
>> just running a 1-node cluster, then adding new nodes is my recommended
>> way of starting large clusters in EC2.
>>
>> Rayson
>>
>> ==================================================
>> Open Grid Scheduler - The Official Open Source Grid Engine
>> http://gridscheduler.sourceforge.net/
>> http://gridscheduler.sourceforge.net/GridEngine/GridEngineCloud.html
>>
>>
>> >
>> > All best,
>> >
>> > Sergio
>> >
>> >
>> > _______________________________________________
>> > StarCluster mailing list
>> > StarCluster_at_mit.edu
>> > http://mailman.mit.edu/mailman/listinfo/starcluster
>> >
>>
>> _______________________________________________
>> StarCluster mailing list
>> StarCluster_at_mit.edu
>> http://mailman.mit.edu/mailman/listinfo/starcluster
Received on Thu Mar 13 2014 - 20:27:03 EDT