Re: [Starcluster] Bug VolumeCreator _request_instance

From: Justin Riley <no email>
Date: Thu, 08 Jul 2010 18:25:35 -0400

Hi Austin,

Did you mean createvolume? If that's the case, yes it does and sorry
this wasn't
immediately obvious to you. There are some docs here:

http://web.mit.edu/stardev/cluster/docs/volumes.html#using-starcluster-to-create-partition-format-a-new-ebs-volume-recommended

Perhaps I should mention these things on the homepage (ie createvolume,
createimage, etc) and possibly also the PyPI page. Any other suggestions for
making users aware of this functionality?

Thanks,

~Justin

Quoting Austin Godber <godber_at_uberhip.com>:

> Oh man, I just implemented my own createcluster, I didn't realize
> starcluster did it already.
>
> Austin
>
> On 07/08/2010 06:16 PM, Justin Riley wrote:
>> Hi Joachim,
>>
>> Thanks for reporting this. Adding an --instance-type option to the
>> createvolume
>> command is on my todo list now and this should fix your issue.
>>
>> For the record though, there's no need to use a 64bit image to create a new
>> volume. The reason I chose m1.small as a default is because this instance is
>> *only* used to mount the volume and automate the
>> partitioning/formatting. For
>> this there really isn't any advantage to using a 64bit image and using
>> m1.small
>> is your cheapest option.
>>
>> I would recommend using a 32bit alestic image (alestic.com), or *any* 32bit
>> image, in the eu-west-1 zone with the createvolume command. This avoids
>> needing
>> a different instance type. Regardless, I'll be adding checks for invalid
>> instance_type/image_id combos though in the VolumeCreator class in
>> addition to
>> allowing instance_type to be passed to the createvolume action at
>> the command
>> line.
>>
>> Also, I've decided to replicate the StarCluster AMIs in all
>> availability zones
>> so that folks outside the us-east-1 zone do not have to migrate the AMIs
>> themselves. The whole point of StarCluster is to abstract most of
>> the details
>> of using EC2 to create HPC clusters in the cloud. Having users
>> migrate the AMI
>> themselves to other zones is suboptimal. I'll send a notice to the
>> list in the
>> next week or so when the us-west/eu-west/ap-souteast AMIs are available
>>
>> Hope that helps and thanks again for reporting.
>>
>> ~Justin
>>
>> Quoting Joachim Van den Bogaert<joachim_at_ccl.kuleuven.be>:
>>
>>
>>> Hi Justin,
>>>
>>>
>>>
>>> I tried to create a volume with the following command:
>>>
>>>
>>>
>>> joachim_at_tm1:~$ starcluster createvolume 100 eu-west-1b --image-id
>>> ami-6be5cf1f
>>>
>>>
>>>
>>>
>>>>>> Using keypair sge
>>>>>>
>>>
>>>>>> Requesting host instance in zone eu-west-1b to attach volume to...
>>>>>>
>>>
>>>>>> Creating security group _at_sc-volumecreator...
>>>>>>
>>>
>>>>>> No instance in group _at_sc-volumecreator for zone eu-west-1b, launching
>>>>>>
>>> one now.
>>>
>>>
>>>
>>> volume.py:186 - ERROR - exception thrown: EC2ResponseError: 400 Bad Request
>>>
>>> <?xml version="1.0"?>
>>>
>>> <Response>
>>>
>>> <Errors>
>>>
>>> <Error>
>>>
>>> <Code>InvalidParameterValue</Code>
>>>
>>> <Message>The requested instance type's architecture (i386) does not
>>> match the architecture in the manifest for ami-6be5cf1f (x86_64)</Message>
>>>
>>> </Error>
>>>
>>> </Errors>
>>>
>>> <RequestID>4ff61a79-6a82-480e-b1dd-e26d8965eb70</RequestID>
>>>
>>> </Response>
>>>
>>>
>>>
>>> I was looking for an architecture parameter, but I couldn't find one
>>>
>>>
>>>
>>> (volume.py line 32 - 56)
>>>
>>>
>>>
>>> def _request_instance(self, zone):
>>>
>>> for i in self.security_group.instances():
>>>
>>> if i.state in ['pending','running'] and i.placement == zone:
>>>
>>> log.info("Using existing instance %s in group %s" % \
>>>
>>> (i.id,self.security_group.name))
>>>
>>> self._instance = Node(i, self._key_location, 'vol_host')
>>>
>>> break
>>>
>>> if not self._instance:
>>>
>>> log.info("No instance in group %s for zone %s, launching one
>>> now." % \
>>>
>>> (self.security_group.name, zone))
>>>
>>> self._resv = self._ec2.run_instances(image_id=self._image_id,
>>>
>>> instance_type='m1.small',
>>>
>>> min_count=1, max_count=1,
>>>
>>> security_groups=[self.security_group.name],
>>>
>>> key_name=self._keypair,
>>>
>>> placement=zone)
>>>
>>> instance = self._resv.instances[0]
>>>
>>> self._instance = Node(instance, self._key_location, 'vol_host')
>>>
>>> s = Spinner()
>>>
>>> log.info("Waiting for instance %s to come up..." %
>>> self._instance.id)
>>>
>>> s.start()
>>>
>>> while not self._instance.is_up():
>>>
>>> time.sleep(15)
>>>
>>> s.stop()
>>>
>>> return self._instance
>>>
>>>
>>>
>>> Apparently the code in line 43
>>>
>>>
>>>
>>> instance_type='m1.small'
>>>
>>>
>>>
>>> throws an error if you try to create a volume with a 64-bit
>>> StarCluster AMI,
>>> since you can't launch an m1.small with 64 bit images.
>>>
>>>
>>>
>>> I am in the eu zone, so I can't use the AMI defined in
>>> image_id=static.BASE_AMI_32.
>>>
>>> With the 64-bit image I use instead, an extra parameter could prevent this
>>> error
>>>
>>>
>>>
>>> After I changed the line into
>>>
>>>
>>>
>>> instance_type='m1.large'
>>>
>>>
>>>
>>> the volume was created successfully.
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>> Kind regards,
>>>
>>> Joachim
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>
>> _______________________________________________
>> Starcluster mailing list
>> Starcluster_at_mit.edu
>> http://mailman.mit.edu/mailman/listinfo/starcluster
>>
>>
>
> _______________________________________________
> Starcluster mailing list
> Starcluster_at_mit.edu
> http://mailman.mit.edu/mailman/listinfo/starcluster
>
Received on Thu Jul 08 2010 - 18:25:37 EDT