Re: StarCluster Digest, Vol 44, Issue 5

From: Rajat Banerjee <no email>
Date: Tue, 30 Apr 2013 13:18:47 -0400

Sergio,
I looked at the code that is causing your problems. It's this line:

return datetime.datetime.strptime(str, "%a %b %d %H:%M:%S UTC %Y")

where 'str' is the output of the *remote* call to date. My mac returns this:
rbanerjee:~/starcluster/StarCluster/starcluster/balancers/sge $ date
Tue Apr 30 13:12:44 EDT 2013

Which looks like it would time format OK. One oddity is that your time
format is returning PDT when UTC is expected:

ValueError: time data 'Tue Apr 30 07:01:35 PDT 2013' does not match format
'%a %b %d %H:%M:%S UTC %Y'

Not sure what's causing the problems since it looks mostly right, but feel
free to tweak the time setting in the code in
starcluster/balancers/sge/__init__.py line 466 to make the pattern match
yours. Do you know why your time zone may be set differently than other AWS
instances we've used? Custom images?
Best,
Rajat




On Tue, Apr 30, 2013 at 12:43 PM, <starcluster-request_at_mit.edu> wrote:

> Send StarCluster mailing list submissions to
> starcluster_at_mit.edu
>
> To subscribe or unsubscribe via the World Wide Web, visit
> http://mailman.mit.edu/mailman/listinfo/starcluster
> or, via email, send a message with subject or body 'help' to
> starcluster-request_at_mit.edu
>
> You can reach the person managing the list at
> starcluster-owner_at_mit.edu
>
> When replying, please edit your Subject line so it is more specific
> than "Re: Contents of StarCluster digest..."
>
> Today's Topics:
>
> 1. Unable to mount ebs volume (Jerry Lee, GW/US)
> 2. LoadBalance (Sergio Mafra)
>
>
> ---------- Forwarded message ----------
> From: "Jerry Lee, GW/US" <Jerry.Lee_at_genewiz.com>
> To: <starcluster_at_mit.edu>
> Cc:
> Date: Mon, 15 Apr 2013 17:11:38 -0500
> Subject: [StarCluster] Unable to mount ebs volume
>
> Hi,****
>
> ** **
>
> I am a beginner of using the StarCluster. I created a ebs volme via Amazon
> AWS and configure it on the config file to use it for my cluster, but no
> matter what I do, it doesn't automatically mount the ebs volume onto the
> cluster. Please help.****
>
> ** **
>
> [cluster jerrycluster]****
>
> EXTENDS = smallcluster****
>
> VOLUMES = testdata****
>
> ** **
>
> [volume testdata]****
>
> VOLUME_ID=vol-a0fe24f9****
>
> MOUNT_PATH=/data****
>
> ** **
>
> >>> Waiting for cluster to come up... (updating every 30s)****
>
> >>> Waiting for instances to activate...****
>
> >>> Waiting for all nodes to be in a 'running' state...****
>
> 2/2 ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
> 100%****
>
> >>> Waiting for SSH to come up on all nodes...****
>
> 2/2 ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
> 100%****
>
> >>> Waiting for cluster to come up took 1.847 mins****
>
> >>> The master node is ec2-54-234-229-206.compute-1.amazonaws.com****
>
> >>> Setting up the cluster...****
>
> >>> Configuring hostnames...****
>
> 2/2 ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
> 100%****
>
> >>> Creating cluster user: None (uid: 1001, gid: 1001)****
>
> 2/2 ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
> 100%****
>
> >>> Configuring scratch space for user(s): sgeadmin****
>
> 2/2 ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
> 100%****
>
> >>> Configuring /etc/hosts on each node****
>
> 2/2 ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
> 100%****
>
> >>> Starting NFS server on master****
>
> >>> Configuring NFS exports path(s):****
>
> /home****
>
> >>> Mounting all NFS export path(s) on 1 worker node(s)****
>
> 1/1 ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
> 100%****
>
> >>> Setting up NFS took 0.073 mins****
>
> >>> Configuring passwordless ssh for root****
>
> >>> Configuring passwordless ssh for sgeadmin****
>
> >>> Shutting down threads...****
>
> 20/20 ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
> 100%****
>
> >>> Configuring SGE...****
>
> >>> Configuring NFS exports path(s):****
>
> /opt/sge6****
>
> >>> Mounting all NFS export path(s) on 1 worker node(s)****
>
> 1/1 ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
> 100%****
>
> >>> Setting up NFS took 0.020 mins****
>
> >>> Installing Sun Grid Engine...****
>
> 1/1 ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
> 100%****
>
> >>> Creating SGE parallel environment 'orte'****
>
> 2/2 ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
> 100%****
>
> >>> Adding parallel environment 'orte' to queue 'all.q'****
>
> >>> Shutting down threads...****
>
> 20/20 ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
> 100%****
>
> >>> Configuring cluster took 1.325 mins****
>
> >>> Starting cluster took 3.197 mins****
>
> ** **
>
> ** **
>
> Thanks,****
>
> ** **
>
> Jerry Lee****
>
> ** **
>
> Jerry Lee****
>
> Assistant Manager of Global Infrastructure****
>
> GENEWIZ Inc.****
>
> 40 Cragwood Road. Suite 201****
>
> South Plainfield, NJ 07080****
>
> Phone: 908-222-0711 ext. 3379****
>
> Fax: 908-333-4511 ****
>
> jerry.lee_at_genewiz.com****
>
> www.genewiz.com****
>
> ****
>
> This electronic message, including its attachments, is confidential and
> proprietary and is solely for the intended recipient. If you are not the
> intended recipient, this message was sent to you in error and you are
> hereby advised that any review, disclosure, copying, distribution or use of
> this message or any of the information included in this message by you is
> unauthorized and strictly prohibited. If you have received this message in
> error, please immediately notify the sender by reply to this message and
> permanently delete all copies of this message and its attachments in your
> possession. Thank you for your cooperation.****
>
> ** **
>
>
> ---------- Forwarded message ----------
> From: Sergio Mafra <sergiohmafra_at_gmail.com>
> To: "starcluster_at_mit.edu" <starcluster_at_mit.edu>
> Cc:
> Date: Tue, 30 Apr 2013 11:05:45 -0300
> Subject: [StarCluster] LoadBalance
> Hi fellows,
>
> Iīm testing StarCluster version 0.999 and so far so good.
> one thing that isnīt working is loadbalance. This is what I get:
>
> ubuntu_at_domU-12-31-39-02-19-36:~$ starcluster loadbalance newcam
> StarCluster - (http://star.mit.edu/cluster) (v. 0.9999)
> Software Tools for Academics and Researchers (STAR)
> Please submit bug reports to starcluster_at_mit.edu
>
> >>> Starting load balancer (Use ctrl-c to exit)
> Maximum cluster size: 3
> Minimum cluster size: 1
> Cluster growth rate: 1 nodes/iteration
>
> *** WARNING - Failed to retrieve stats (1/5):
> Traceback (most recent call last):
> File
> "/usr/local/lib/python2.7/dist-packages/StarCluster-0.9999-py2.7.egg/starcluster/balancers/sge/__init__.py",
> line 515, in get_stats
> self.stat = self._get_stats()
> File
> "/usr/local/lib/python2.7/dist-packages/StarCluster-0.9999-py2.7.egg/starcluster/balancers/sge/__init__.py",
> line 487, in _get_stats
> now = self.get_remote_time()
> File
> "/usr/local/lib/python2.7/dist-packages/StarCluster-0.9999-py2.7.egg/starcluster/balancers/sge/__init__.py",
> line 466, in get_remote_time
> return datetime.datetime.strptime(str, "%a %b %d %H:%M:%S UTC %Y")
> File "/usr/lib/python2.7/_strptime.py", line 325, in _strptime
> (data_string, format))
> ValueError: time data 'Tue Apr 30 07:01:35 PDT 2013' does not match format
> '%a %b %d %H:%M:%S UTC %Y'
> *** WARNING - Retrying in 60s
> ^CTraceback (most recent call last):
> File "/usr/local/bin/starcluster", line 9, in <module>
> load_entry_point('StarCluster==0.9999', 'console_scripts',
> 'starcluster')()
> File
> "/usr/local/lib/python2.7/dist-packages/StarCluster-0.9999-py2.7.egg/starcluster/cli.py",
> line 313, in main
> StarClusterCLI().main()
> File
> "/usr/local/lib/python2.7/dist-packages/StarCluster-0.9999-py2.7.egg/starcluster/cli.py",
> line 257, in main
> sc.execute(args)
> File
> "/usr/local/lib/python2.7/dist-packages/StarCluster-0.9999-py2.7.egg/starcluster/commands/loadbalance.py",
> line 90, in execute
> lb.run(cluster)
> File
> "/usr/local/lib/python2.7/dist-packages/StarCluster-0.9999-py2.7.egg/starcluster/balancers/sge/__init__.py",
> line 576, in run
> self.get_stats()
> File "<string>", line 2, in get_stats
> File
> "/usr/local/lib/python2.7/dist-packages/StarCluster-0.9999-py2.7.egg/starcluster/utils.py",
> line 92, in wrap_f
> res = func(*arg, **kargs)
> File
> "/usr/local/lib/python2.7/dist-packages/StarCluster-0.9999-py2.7.egg/starcluster/balancers/sge/__init__.py",
> line 521, in get_stats
> time.sleep(self.polling_interval)
>
> Any ideas?
>
> All Best,
>
> Sergio
>
> _______________________________________________
> StarCluster mailing list
> StarCluster_at_mit.edu
> http://mailman.mit.edu/mailman/listinfo/starcluster
>
>
Received on Tue Apr 30 2013 - 13:19:08 EDT