Using StarCluster with Hudson Continuous Integration

From: Vaughan, Garrison <no email>
Date: Thu, 14 Jun 2012 21:11:50 +0000

Hello,

I'm currently working as an intern with NASA at the Goddard Space Flight Center. We have decided to move
some regression tests we run on our own high performance computer system into the cloud.
I have been assigned the task of setting up a continuous integration server and setting up the scripts which will
instantiate clusters that will be used for the builds/regression test. I was shown StarCluster by a system architect here
and needless to say, it's an extremely impressive tool!

I am currently using a CI server called Hudson which is running smoothly on a micro AMI instance which mimics the OS image we run on our cluster.
The Hudson Server runs under a user (user Hudson) on the instance and can run scripts as that user on the system when triggered.
The point is to, when triggered, create a cluster (using StarCluster) that will mimic the cluster we run here,
and use that cluster to build and do a regression test against a clients code repository, and subsequently tear the
cluster down upon completion.

I have StarCluster installed under user Hudson, and when I shell into user Hudson, StarCluster works without a hitch.
My problem occurs when I try to run any StarCluster command through the Hudson Web Servlet. I have a job going on Hudson
which, when triggered, will execute a shell command (currently "starcluster help") under user Hudson. After setting up the $PATH to StarCluster
through the Hudson UI, the starcluster command is recognized.

But for some reason, which I've been struggling with all day, I get this error as output when I run any starcluster command:


#######Hudson Console Output#######

Started by user admin
[workspace] $ /bin/bash -xe /tmp/hudson6888230102812979297.sh ## [workspace] is a directory under user Hudson. When the same command
                                                                                                                               ## which is in the /tmp/hudsonxxxx.sh script (currently "starcluster help") is
                                                                                                                               ## run from this directory through ssh, it runs just fine.

+ starcluster help ##This is the command in /tmp/hudsonxxxxx.sh. A whoami shows this command running under user Hudson.
!!! ERROR - /root/.starcluster *must* be a directory ##This is what has me baffled, why does anything in /root need to be referenced/exist?
                                                                                                                               ##Again, this isn't the case when this command is run directly under user Hudson (through ssh).
 !!! ERROR - /root/.starcluster *must* be a directory
Traceback (most recent call last):
  File "/usr/local/bin/starcluster", line 9, in <module>
    load_entry_point('StarCluster==0.93.3', 'console_scripts', 'starcluster')()
  File "/usr/local/lib64/python2.6/site-packages/StarCluster-0.93.3-py2.6.egg/starcluster/cli.py", line 310, in main
    logger.configure_sc_logging()
  File "/usr/local/lib64/python2.6/site-packages/StarCluster-0.93.3-py2.6.egg/starcluster/logger.py", line 129, in configure_sc_logging
    backupCount=2)
  File "/usr/lib64/python2.6/logging/handlers.py", line 112, in __init__
    BaseRotatingHandler.__init__(self, filename, mode, encoding, delay)
  File "/usr/lib64/python2.6/logging/handlers.py", line 64, in __init__
    logging.FileHandler.__init__(self, filename, mode, encoding, delay)
  File "/usr/lib64/python2.6/logging/__init__.py", line 827, in __init__
    StreamHandler.__init__(self, self._open())
  File "/usr/lib64/python2.6/logging/__init__.py", line 846, in _open
    stream = open(self.baseFilename, self.mode)
IOError: [Errno 13] Permission denied: '/root/.starcluster/logs/debug.log'
[DEBUG] Skipping watched dependency update for build: Test Build #30 due to result: FAILURE
Finished: FAILURE

.starcluster exists at /var/lib/hudson/.starcluster and is shown as the default path to the config file when "starcluster help" is run on user Hudson (through ssh).
I have passed that directory as a (starcluster -c "path_to_config") argument in the Hudson script, and still get the same error.
Permissions for /var/lib/hudson/.starcluster are rwxr-xr-x.

Any help would be greatly appreciated!

Thanks for the great work,
Garrison
Received on Thu Jun 14 2012 - 17:11:58 EDT