Re: [Starcluster] Bug VolumeCreator _request_instance

From: Austin Godber <no email>
Date: Thu, 08 Jul 2010 18:32:36 -0400

Hi Justin,
     Yeah, my apologies, I meant createvolume. I don't know that you
could make it more clear. I had just assumed that I had to do it
myself. Its documented pretty well. Though, perhaps in the commands
"Available Actions" output, if you said
"Create, partition, and format a new EBS volume for use with StarCluster
(this launches an AMI to do the work)"
or something along those lines it would be more explicit.

I had really just assumed that they had to be made yourself. I bet most
users take the meaning correctly.

Austin


On 07/08/2010 06:25 PM, Justin Riley wrote:
> Hi Austin,
>
> Did you mean createvolume? If that's the case, yes it does and sorry
> this wasn't
> immediately obvious to you. There are some docs here:
>
> http://web.mit.edu/stardev/cluster/docs/volumes.html#using-starcluster-to-create-partition-format-a-new-ebs-volume-recommended
>
>
> Perhaps I should mention these things on the homepage (ie createvolume,
> createimage, etc) and possibly also the PyPI page. Any other
> suggestions for
> making users aware of this functionality?
>
> Thanks,
>
> ~Justin
>
> Quoting Austin Godber <godber_at_uberhip.com>:
>
>> Oh man, I just implemented my own createcluster, I didn't realize
>> starcluster did it already.
>>
>> Austin
>>
>> On 07/08/2010 06:16 PM, Justin Riley wrote:
>>> Hi Joachim,
>>>
>>> Thanks for reporting this. Adding an --instance-type option to the
>>> createvolume
>>> command is on my todo list now and this should fix your issue.
>>>
>>> For the record though, there's no need to use a 64bit image to
>>> create a new
>>> volume. The reason I chose m1.small as a default is because this
>>> instance is
>>> *only* used to mount the volume and automate the
>>> partitioning/formatting. For
>>> this there really isn't any advantage to using a 64bit image and using
>>> m1.small
>>> is your cheapest option.
>>>
>>> I would recommend using a 32bit alestic image (alestic.com), or
>>> *any* 32bit
>>> image, in the eu-west-1 zone with the createvolume command. This avoids
>>> needing
>>> a different instance type. Regardless, I'll be adding checks for
>>> invalid
>>> instance_type/image_id combos though in the VolumeCreator class in
>>> addition to
>>> allowing instance_type to be passed to the createvolume action at
>>> the command
>>> line.
>>>
>>> Also, I've decided to replicate the StarCluster AMIs in all
>>> availability zones
>>> so that folks outside the us-east-1 zone do not have to migrate the
>>> AMIs
>>> themselves. The whole point of StarCluster is to abstract most of
>>> the details
>>> of using EC2 to create HPC clusters in the cloud. Having users
>>> migrate the AMI
>>> themselves to other zones is suboptimal. I'll send a notice to the
>>> list in the
>>> next week or so when the us-west/eu-west/ap-souteast AMIs are available
>>>
>>> Hope that helps and thanks again for reporting.
>>>
>>> ~Justin
>>>
>>> Quoting Joachim Van den Bogaert<joachim_at_ccl.kuleuven.be>:
>>>
>>>
>>>> Hi Justin,
>>>>
>>>>
>>>>
>>>> I tried to create a volume with the following command:
>>>>
>>>>
>>>>
>>>> joachim_at_tm1:~$ starcluster createvolume 100 eu-west-1b --image-id
>>>> ami-6be5cf1f
>>>>
>>>>
>>>>
>>>>
>>>>>>> Using keypair sge
>>>>>>>
>>>>
>>>>>>> Requesting host instance in zone eu-west-1b to attach volume to...
>>>>>>>
>>>>
>>>>>>> Creating security group _at_sc-volumecreator...
>>>>>>>
>>>>
>>>>>>> No instance in group _at_sc-volumecreator for zone eu-west-1b,
>>>>>>> launching
>>>>>>>
>>>> one now.
>>>>
>>>>
>>>>
>>>> volume.py:186 - ERROR - exception thrown: EC2ResponseError: 400 Bad
>>>> Request
>>>>
>>>> <?xml version="1.0"?>
>>>>
>>>> <Response>
>>>>
>>>> <Errors>
>>>>
>>>> <Error>
>>>>
>>>> <Code>InvalidParameterValue</Code>
>>>>
>>>> <Message>The requested instance type's architecture (i386) does not
>>>> match the architecture in the manifest for ami-6be5cf1f
>>>> (x86_64)</Message>
>>>>
>>>> </Error>
>>>>
>>>> </Errors>
>>>>
>>>> <RequestID>4ff61a79-6a82-480e-b1dd-e26d8965eb70</RequestID>
>>>>
>>>> </Response>
>>>>
>>>>
>>>>
>>>> I was looking for an architecture parameter, but I couldn't find one
>>>>
>>>>
>>>>
>>>> (volume.py line 32 - 56)
>>>>
>>>>
>>>>
>>>> def _request_instance(self, zone):
>>>>
>>>> for i in self.security_group.instances():
>>>>
>>>> if i.state in ['pending','running'] and i.placement ==
>>>> zone:
>>>>
>>>> log.info("Using existing instance %s in group %s" % \
>>>>
>>>> (i.id,self.security_group.name))
>>>>
>>>> self._instance = Node(i, self._key_location,
>>>> 'vol_host')
>>>>
>>>> break
>>>>
>>>> if not self._instance:
>>>>
>>>> log.info("No instance in group %s for zone %s,
>>>> launching one
>>>> now." % \
>>>>
>>>> (self.security_group.name, zone))
>>>>
>>>> self._resv =
>>>> self._ec2.run_instances(image_id=self._image_id,
>>>>
>>>> instance_type='m1.small',
>>>>
>>>> min_count=1, max_count=1,
>>>>
>>>> security_groups=[self.security_group.name],
>>>>
>>>> key_name=self._keypair,
>>>>
>>>> placement=zone)
>>>>
>>>> instance = self._resv.instances[0]
>>>>
>>>> self._instance = Node(instance, self._key_location,
>>>> 'vol_host')
>>>>
>>>> s = Spinner()
>>>>
>>>> log.info("Waiting for instance %s to come up..." %
>>>> self._instance.id)
>>>>
>>>> s.start()
>>>>
>>>> while not self._instance.is_up():
>>>>
>>>> time.sleep(15)
>>>>
>>>> s.stop()
>>>>
>>>> return self._instance
>>>>
>>>>
>>>>
>>>> Apparently the code in line 43
>>>>
>>>>
>>>>
>>>> instance_type='m1.small'
>>>>
>>>>
>>>>
>>>> throws an error if you try to create a volume with a 64-bit
>>>> StarCluster AMI,
>>>> since you can't launch an m1.small with 64 bit images.
>>>>
>>>>
>>>>
>>>> I am in the eu zone, so I can't use the AMI defined in
>>>> image_id=static.BASE_AMI_32.
>>>>
>>>> With the 64-bit image I use instead, an extra parameter could
>>>> prevent this
>>>> error
>>>>
>>>>
>>>>
>>>> After I changed the line into
>>>>
>>>>
>>>>
>>>> instance_type='m1.large'
>>>>
>>>>
>>>>
>>>> the volume was created successfully.
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>> Kind regards,
>>>>
>>>> Joachim
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>
>>> _______________________________________________
>>> Starcluster mailing list
>>> Starcluster_at_mit.edu
>>> http://mailman.mit.edu/mailman/listinfo/starcluster
>>>
>>>
>>
>> _______________________________________________
>> Starcluster mailing list
>> Starcluster_at_mit.edu
>> http://mailman.mit.edu/mailman/listinfo/starcluster
>>
>
>
Received on Thu Jul 08 2010 - 18:32:39 EDT