Re: _at_sc-core tag exceeds maximum length of 255 chars (#348)

From: Justin Riley <no email>
Date: Thu, 12 Dec 2013 16:15:33 -0500

Just following up to the list: this issue occured because of an
unsupported change in a fork of StarCluster. Nevertheless we're working
on a fix for this but this shouldn't be an issue for folks using the
official repo.

~Justin

On Thu, Dec 12, 2013 at 03:34:22PM -0500, Jennifer Staab wrote:
> I have a feeling you were about to suggest the fix that I have outlined
> below. It involves no code change just a change in how you set up your
> cluster template such that your _at_sc-core tag squeezes within the 255
> character limit.
> Things to note about EC2-VPC and the this issue: 
> 1) EC2-VPC limits their tag lengths to 255 characters (this is not true
> for EC2-Classic)
> 2) _at_sc-core tag and (non-truncated) value are needed for starcluster to
> function properly
> 3) _at_sc-core tag values are serialized (via json) and compressed to make
> them usable but reduce their size
> 4) changing zlib.compress() function to use level 9 (highest compression)
> instead of default 6 doesn't solve the problem
> SOLUTION
> Changing your [cluster]/cluster template configuration in the
> configuration file such that 'master_instance_type' and
> 'node_instance_type' are of same type (like m1.small) AND
> 'master_image_id' and 'node_image_id' use the same ID (like ami-xxxxxxxx)
> will allow compression such that the _at_sc-core tag value can be compressed
> to a size that is under 255 characters.
> You can use VPC even if your master and worker nodes are of different
> types or use different AMI's. To allow this type of variation and not
> violate the 255 character limit, I would suggest creating a cluster
> template with the AMI id and type you want for the master node for both
> master and (worker) node. Then use starcluster addnode with -i and -I
> options to add all the worker nodes you want of the TYPE and AMI ID you
> want. I have tried it and it seems to work perfectly.
> Just in case someone runs into a similar problem in the future, below are
> the dictionary of values that are serialized and compressed to create the
> _at_sc-core tag's value:
> {'disable_cloudinit': False, 'master_instance_type': 'm1.small',
> 'disable_queue': False, 'subnet_id': 'subnet-xxxxxxxx',
> 'availability_zone': 'us-east-1b', 'node_image_id': 'ami-xxxxxxxx',
> 'node_instance_type': 'm1.small', 'use_nfs_crossmnt': True,
> 'cluster_size': 1, 'master_image_id': 'ami-xxxxxxxx', 'dns_prefix': False}
> Thanks,
> -Jennifer
>
> On Thu, Dec 12, 2013 at 1:04 PM, Justin Riley
> <[1]notifications_at_github.com> wrote:
>
> [2]_at_jmstaab Yep, that's the issue. The reason I asked for the config is
> I'm curious what is making the value that long. I have a more general
> fix for this but in the mean time would you mind pasting your [cluster]
> section? Thanks!
>
> —
> Reply to this email directly or [3]view it on GitHub.[IMG]
>
> References
>
> Visible links
> 1. mailto:notifications_at_github.com
> 2. https://github.com/jmstaab
> 3. https://github.com/jtriley/StarCluster/issues/348#issuecomment-30446236

> _______________________________________________
> StarCluster mailing list
> StarCluster_at_mit.edu
> http://mailman.mit.edu/mailman/listinfo/starcluster




Received on Thu Dec 12 2013 - 16:15:35 EST