Re: request of new features of starcluster

From: Rayson Ho <no email>
Date: Mon, 26 Jan 2015 10:49:17 -0500

On Tue, Jan 13, 2015 at 12:35 PM, Jian Feng <freedafeng_at_gmail.com> wrote:

> But on the other hand, your idea to assign two security groups can solve
> my problem too. I just did not know we can assign multiple sg to an
> instance. (I did not see this functionality in the UI.) I guess we can only
> do this programmatically? Could you or anyone please elaborate how this is
> done?
>

The standard EC2 python API supports passing in a list of security groups
when you request for EC2 instances -- you will notice that
"security_groups" & "security_group_ids" (for VPC) both accept a list of
groups:

https://boto.readthedocs.org/en/latest/ref/ec2.html#boto.ec2.connection.EC2Connection.run_instances

In fact, I hacked the code in StarCluster so that it uses both my
pre-created security group and the group that it creates during the
StarCluster bootstrap process, and the code change was minimal.

Rayson

==================================================
Open Grid Scheduler - The Official Open Source Grid Engine
http://gridscheduler.sourceforge.net/
http://gridscheduler.sourceforge.net/GridEngine/GridEngineCloud.html




>
> Thanks!
>
> On Tue, Jan 13, 2015 at 7:51 AM, Ramon Ramirez-Linan <rlinan_at_navteca.com>
> wrote:
>
>> Hi,
>>
>> For question 1
>> What do you think about adding an option where the user can specify a SG
>> to server as a template for the Cluster's SG?
>> So if I have a production SG (psg) StarCluster can copy that group
>> psg_sc, that way the workflow of destroying a cluster will be safer since
>> psg_sc only exist for the purpose of that cluster. I think this option
>> would be easier to implement.
>>
>> For question 2,
>> I would also like to see this feature, what I have been doing for now is
>> to bake the compute node storage in the AMI
>>
>>
>>
>>
>> On Mon, Jan 12, 2015 at 11:45 AM, Rayson Ho <raysonlogin_at_gmail.com>
>> wrote:
>>
>>> Hi Jian,
>>>
>>> StarCluster puts information in the cluster security group to identify
>>> whether a cluster exists or not (you may have encountered the message " !!!
>>> ERROR - Cluster 'sc' already exists.". While we can rename or clear the
>>> tags added to a security group, forcing StarCluster to use a predefined
>>> security group can affect the current code we have.
>>>
>>> IMO, a better way to satisfy your need is to add your security group
>>> (the one that's used as the traffic source in the production security
>>> group) together with the security group create by Starcluster to each
>>> cluster node. This way, only 1 or 2 functions in the existing StarCluster
>>> code need to be changed.
>>>
>>> One the other hand, forcing StarCluster to use a predefined security
>>> group can cause issues as StarCluster's cleanup code won't be able to
>>> delete your group, because it is referenced in the production security
>>> group. While StarCluster probably shouldn't touch your predefined security
>>> group at all if one is supplied, it does change how StarCluster cleans up
>>> the security group when you terminate the cluster.
>>>
>>> Finally, an added benefit is that you can document what that security
>>> group is for in your audit log, and also apply that security group to any
>>> instances that need to access the production cluster. This way, you can
>>> have multiple StarClusters and/or other services access the production
>>> cluster, while at the same time keeping the traffic from flowing through
>>> each other.
>>>
>>> Rayson
>>>
>>> ==================================================
>>> Open Grid Scheduler - The Official Open Source Grid Engine
>>> http://gridscheduler.sourceforge.net/
>>> http://gridscheduler.sourceforge.net/GridEngine/GridEngineCloud.html
>>>
>>>
>>>
>>> On Tue, Jan 6, 2015 at 3:52 PM, Jian Feng <freedafeng_at_gmail.com> wrote:
>>>
>>>> Dear StarCluster devs,
>>>>
>>>> As a loyal starcluster user, I am wondering if it's ok for me to submit
>>>> a feature request to meet needs of more people and businesses. I'd like to
>>>> see
>>>>
>>>> 1. security group separated from cluster creation. In AWS the sg and
>>>> instances are two totally independent things. Security group and live
>>>> without any instances. This is convenient because we'd like to reuse the
>>>> security group for different clusters. This is especially true when we need
>>>> to fine tune the security rules for different type of work. In my case, I
>>>> need my starcluster cluster to access the production cluster data. So I
>>>> will have to change the production security group rules every time I create
>>>> a new star cluster. This is not efficient and not easy to be fully
>>>> automated.
>>>> It's totally fine to keep the current setting but give people an
>>>> option to reuse an existing security group.
>>>>
>>>> 2. local ebs disk storage for each node. I see a great amount
>>>> applications have this needs. This is crucial for any distributed
>>>> persistent data storage.
>>>>
>>>> I understand that starcluster is open source so people can tailor it to
>>>> their own needs. The thing is some functionalities can help people to
>>>> improve productivity a lot so they might be worth adding.
>>>>
>>>> Thanks!
>>>>
>>>> _______________________________________________
>>>> StarCluster mailing list
>>>> StarCluster_at_mit.edu
>>>> http://mailman.mit.edu/mailman/listinfo/starcluster
>>>>
>>>>
>>>
>>> _______________________________________________
>>> StarCluster mailing list
>>> StarCluster_at_mit.edu
>>> http://mailman.mit.edu/mailman/listinfo/starcluster
>>>
>>>
>>
>
Received on Mon Jan 26 2015 - 10:49:19 EST