Re: request of new features of starcluster

From: Jian Feng <no email>
Date: Tue, 27 Jan 2015 15:06:31 -0800

Thanks Rayson!

On Mon, Jan 26, 2015 at 7:49 AM, Rayson Ho <raysonlogin_at_gmail.com> wrote:

> On Tue, Jan 13, 2015 at 12:35 PM, Jian Feng <freedafeng_at_gmail.com> wrote:
>
>> But on the other hand, your idea to assign two security groups can solve
>> my problem too. I just did not know we can assign multiple sg to an
>> instance. (I did not see this functionality in the UI.) I guess we can only
>> do this programmatically? Could you or anyone please elaborate how this is
>> done?
>>
>
> The standard EC2 python API supports passing in a list of security groups
> when you request for EC2 instances -- you will notice that
> "security_groups" & "security_group_ids" (for VPC) both accept a list of
> groups:
>
>
> https://boto.readthedocs.org/en/latest/ref/ec2.html#boto.ec2.connection.EC2Connection.run_instances
>
> In fact, I hacked the code in StarCluster so that it uses both my
> pre-created security group and the group that it creates during the
> StarCluster bootstrap process, and the code change was minimal.
>
> Rayson
>
> ==================================================
> Open Grid Scheduler - The Official Open Source Grid Engine
> http://gridscheduler.sourceforge.net/
> http://gridscheduler.sourceforge.net/GridEngine/GridEngineCloud.html
>
>
>
>
>>
>> Thanks!
>>
>> On Tue, Jan 13, 2015 at 7:51 AM, Ramon Ramirez-Linan <rlinan_at_navteca.com>
>> wrote:
>>
>>> Hi,
>>>
>>> For question 1
>>> What do you think about adding an option where the user can specify a SG
>>> to server as a template for the Cluster's SG?
>>> So if I have a production SG (psg) StarCluster can copy that group
>>> psg_sc, that way the workflow of destroying a cluster will be safer since
>>> psg_sc only exist for the purpose of that cluster. I think this option
>>> would be easier to implement.
>>>
>>> For question 2,
>>> I would also like to see this feature, what I have been doing for now is
>>> to bake the compute node storage in the AMI
>>>
>>>
>>>
>>>
>>> On Mon, Jan 12, 2015 at 11:45 AM, Rayson Ho <raysonlogin_at_gmail.com>
>>> wrote:
>>>
>>>> Hi Jian,
>>>>
>>>> StarCluster puts information in the cluster security group to identify
>>>> whether a cluster exists or not (you may have encountered the message " !!!
>>>> ERROR - Cluster 'sc' already exists.". While we can rename or clear the
>>>> tags added to a security group, forcing StarCluster to use a predefined
>>>> security group can affect the current code we have.
>>>>
>>>> IMO, a better way to satisfy your need is to add your security group
>>>> (the one that's used as the traffic source in the production security
>>>> group) together with the security group create by Starcluster to each
>>>> cluster node. This way, only 1 or 2 functions in the existing StarCluster
>>>> code need to be changed.
>>>>
>>>> One the other hand, forcing StarCluster to use a predefined security
>>>> group can cause issues as StarCluster's cleanup code won't be able to
>>>> delete your group, because it is referenced in the production security
>>>> group. While StarCluster probably shouldn't touch your predefined security
>>>> group at all if one is supplied, it does change how StarCluster cleans up
>>>> the security group when you terminate the cluster.
>>>>
>>>> Finally, an added benefit is that you can document what that security
>>>> group is for in your audit log, and also apply that security group to any
>>>> instances that need to access the production cluster. This way, you can
>>>> have multiple StarClusters and/or other services access the production
>>>> cluster, while at the same time keeping the traffic from flowing through
>>>> each other.
>>>>
>>>> Rayson
>>>>
>>>> ==================================================
>>>> Open Grid Scheduler - The Official Open Source Grid Engine
>>>> http://gridscheduler.sourceforge.net/
>>>> http://gridscheduler.sourceforge.net/GridEngine/GridEngineCloud.html
>>>>
>>>>
>>>>
>>>> On Tue, Jan 6, 2015 at 3:52 PM, Jian Feng <freedafeng_at_gmail.com> wrote:
>>>>
>>>>> Dear StarCluster devs,
>>>>>
>>>>> As a loyal starcluster user, I am wondering if it's ok for me to
>>>>> submit a feature request to meet needs of more people and businesses. I'd
>>>>> like to see
>>>>>
>>>>> 1. security group separated from cluster creation. In AWS the sg and
>>>>> instances are two totally independent things. Security group and live
>>>>> without any instances. This is convenient because we'd like to reuse the
>>>>> security group for different clusters. This is especially true when we need
>>>>> to fine tune the security rules for different type of work. In my case, I
>>>>> need my starcluster cluster to access the production cluster data. So I
>>>>> will have to change the production security group rules every time I create
>>>>> a new star cluster. This is not efficient and not easy to be fully
>>>>> automated.
>>>>> It's totally fine to keep the current setting but give people an
>>>>> option to reuse an existing security group.
>>>>>
>>>>> 2. local ebs disk storage for each node. I see a great amount
>>>>> applications have this needs. This is crucial for any distributed
>>>>> persistent data storage.
>>>>>
>>>>> I understand that starcluster is open source so people can tailor it
>>>>> to their own needs. The thing is some functionalities can help people to
>>>>> improve productivity a lot so they might be worth adding.
>>>>>
>>>>> Thanks!
>>>>>
>>>>> _______________________________________________
>>>>> StarCluster mailing list
>>>>> StarCluster_at_mit.edu
>>>>> http://mailman.mit.edu/mailman/listinfo/starcluster
>>>>>
>>>>>
>>>>
>>>> _______________________________________________
>>>> StarCluster mailing list
>>>> StarCluster_at_mit.edu
>>>> http://mailman.mit.edu/mailman/listinfo/starcluster
>>>>
>>>>
>>>
>>
>
Received on Tue Jan 27 2015 - 18:06:33 EST