Re: request of new features of starcluster

From: Jian Feng <no email>
Date: Tue, 13 Jan 2015 09:35:07 -0800

Hi Rayson and Ramon,

Thanks for the clarifications and comments!

Sorry for replying late -- was stuck in some production issues.

I understand that starcluster uses security group as the cluster
identification so that it knows which ec2 instances belongs to which
cluster. This is totally fine and I couldn't find any better way to ID the
cluster instances. What I was proposing is to decouple the
(creation/deletion) of cluster instances and the security group. It still
holds that there is at most one cluster using any specific security group.
The only difference is that the security group won't be deleted when we
terminate a cluster, and it won't (or does not need to) be created when we
create a cluster.

But on the other hand, your idea to assign two security groups can solve my
problem too. I just did not know we can assign multiple sg to an instance.
(I did not see this functionality in the UI.) I guess we can only do this
programmatically? Could you or anyone please elaborate how this is done?

Thanks!

On Tue, Jan 13, 2015 at 7:51 AM, Ramon Ramirez-Linan <rlinan_at_navteca.com>
wrote:

> Hi,
>
> For question 1
> What do you think about adding an option where the user can specify a SG
> to server as a template for the Cluster's SG?
> So if I have a production SG (psg) StarCluster can copy that group psg_sc,
> that way the workflow of destroying a cluster will be safer since psg_sc
> only exist for the purpose of that cluster. I think this option would be
> easier to implement.
>
> For question 2,
> I would also like to see this feature, what I have been doing for now is
> to bake the compute node storage in the AMI
>
>
>
>
> On Mon, Jan 12, 2015 at 11:45 AM, Rayson Ho <raysonlogin_at_gmail.com> wrote:
>
>> Hi Jian,
>>
>> StarCluster puts information in the cluster security group to identify
>> whether a cluster exists or not (you may have encountered the message " !!!
>> ERROR - Cluster 'sc' already exists.". While we can rename or clear the
>> tags added to a security group, forcing StarCluster to use a predefined
>> security group can affect the current code we have.
>>
>> IMO, a better way to satisfy your need is to add your security group (the
>> one that's used as the traffic source in the production security group)
>> together with the security group create by Starcluster to each cluster
>> node. This way, only 1 or 2 functions in the existing StarCluster code need
>> to be changed.
>>
>> One the other hand, forcing StarCluster to use a predefined security
>> group can cause issues as StarCluster's cleanup code won't be able to
>> delete your group, because it is referenced in the production security
>> group. While StarCluster probably shouldn't touch your predefined security
>> group at all if one is supplied, it does change how StarCluster cleans up
>> the security group when you terminate the cluster.
>>
>> Finally, an added benefit is that you can document what that security
>> group is for in your audit log, and also apply that security group to any
>> instances that need to access the production cluster. This way, you can
>> have multiple StarClusters and/or other services access the production
>> cluster, while at the same time keeping the traffic from flowing through
>> each other.
>>
>> Rayson
>>
>> ==================================================
>> Open Grid Scheduler - The Official Open Source Grid Engine
>> http://gridscheduler.sourceforge.net/
>> http://gridscheduler.sourceforge.net/GridEngine/GridEngineCloud.html
>>
>>
>>
>> On Tue, Jan 6, 2015 at 3:52 PM, Jian Feng <freedafeng_at_gmail.com> wrote:
>>
>>> Dear StarCluster devs,
>>>
>>> As a loyal starcluster user, I am wondering if it's ok for me to submit
>>> a feature request to meet needs of more people and businesses. I'd like to
>>> see
>>>
>>> 1. security group separated from cluster creation. In AWS the sg and
>>> instances are two totally independent things. Security group and live
>>> without any instances. This is convenient because we'd like to reuse the
>>> security group for different clusters. This is especially true when we need
>>> to fine tune the security rules for different type of work. In my case, I
>>> need my starcluster cluster to access the production cluster data. So I
>>> will have to change the production security group rules every time I create
>>> a new star cluster. This is not efficient and not easy to be fully
>>> automated.
>>> It's totally fine to keep the current setting but give people an
>>> option to reuse an existing security group.
>>>
>>> 2. local ebs disk storage for each node. I see a great amount
>>> applications have this needs. This is crucial for any distributed
>>> persistent data storage.
>>>
>>> I understand that starcluster is open source so people can tailor it to
>>> their own needs. The thing is some functionalities can help people to
>>> improve productivity a lot so they might be worth adding.
>>>
>>> Thanks!
>>>
>>> _______________________________________________
>>> StarCluster mailing list
>>> StarCluster_at_mit.edu
>>> http://mailman.mit.edu/mailman/listinfo/starcluster
>>>
>>>
>>
>> _______________________________________________
>> StarCluster mailing list
>> StarCluster_at_mit.edu
>> http://mailman.mit.edu/mailman/listinfo/starcluster
>>
>>
>
Received on Tue Jan 13 2015 - 12:35:10 EST