Re: advice about using starcluster with ubuntu 14.04

From: Cedar McKay <no email>
Date: Mon, 16 Feb 2015 09:27:32 -0800

Some people, like Jennifer, have managed to get 14.04 AMIs working. It would be a real service if those people could share them.

Also, does anyone have a reasonable explanation for why the official version is so slow in coming? Is there some sort of issue?

Thanks,
Cedar



> On Jan 29, 2015, at 6:41 AM, Pablo Escobar <pescobar001_at_gmail.com> wrote:
>
> Hi Jennifer
>
> thank you very much for your help. It's really useful.
>
> regards,
> Pablo.
>
> 2015-01-28 19:10 GMT+01:00 Jennifer Staab <jstaab_at_cs.unc.edu>:
> I installed Starcluster on Ubuntu 14.04 LTS and it is working fine for me. I didn't use the guide you mentioned, just a little guidance from emails on mailing list between Jacob and Hugh regarding installation (see here) and some sleuthing to get it to work. Hugh also provides a guideline/cookbook for creating new Starcluster AMI's here on the mailing list and said that just upgrading the existing AMI for 12.04 to 14.04 worked for him. To me that seems like the easiest way to handle this whole issue is to use existing Ubuntu 12.04 Starcluster AMI and just upgrade it to 14.04. I wasn't able to do this, so below is what worked for me. I am sure there are other ways to create a new Ubuntu 14.04 Starcluster AMI.
>
> --------------------------------------------------------------------------------------
> 1) Locate the AMI ID of the latest version of Ubuntu 14.04 for your region (see here). Start up your Ubuntu 14.04 AMI -- login and run update and apply all the security and distribution updates.
>
> 2) Install packages "python-dev" and "python-setuptools" -- You are welcome to install other packages, but those are the only ones needed if to do the Starcluster install. A list of all python packages can be found here .
>
> 3) You need to install NFS server so install package "nfs-kernel-server". Installing this package should install its dependencies (for me that included: nfs-common rpcbind libgssglue1 libnfsidmap2 libtirpc1). One issue I found was that in previous versions the startup script located in /etc/init.d/ was named "nfs" this installed version it's named "nfs-kernel-server". The easiest fix is to create a symbolic link to "/etc/init.d/nfs" (ln -s /etc/init.d/nfs-kernel-server /etc/init.d/nfs). Test by issuing the command "/etc/init.d/nfs restart" if it restarts then all is well.
>
> 4) Install Starcluster with "easy_install Starcluster"
>
> 5) You need to get a copy of the Open Grid Scheduler and place it in "/opt/sge6-fresh/" -- this is what Hugh and Jacob discuss in their emails on the mailing list. The easiest way I was able to do this was to use a running Ubuntu 12.04 Starcluster AMI (ami-765b3e1f) and copy "/opt/sge-fresh" from the 12.04 version to my Ubuntu 14.04 version. There are other ways to do this see their emails here.
>
> 6) Ubuntu 14.04 replaces portmap with rpcbind -- so this was an issue for me because the Starcluster software expects "portmap" to exist. So this is where I had to "edit" the code a bit.There are other ways to deal with the portmap issue, look here for an alternative solution. This is just how I handled the issue: locate the "node.py" portion of the code in the installed StarCluster version, mine was in "/usr/local/lib/python2.7/dist-packages/StarclusterXXX.egg/starcluster/node.py" On line 731 in the function "mount_nfs_shares()" you see line "self.ssh.execute('/etc/init.d/portmap start')". This won't work on Ubuntu 14.04 since "portmap" was replaced with "rpcbind". One way to fix this issue without breaking the code for other OS versions still using portmap is by replacing "self.ssh.execute('/etc/init.d/portmap start')" with a "try/except" statement so that if portmap doesn't work it calls rpcbind command instead; thus, handling the issue with portmap on Ubuntu 14.04. Below is how I changed the code in
node.py:
> FIND line 731:
> self.ssh.execute('/etc/init.d/portmap start')
> REPLACE with:
> try:
> self.ssh.execute('/etc/init.d/portmap start')
> except:
> self.ssh.execute('service rpcbind restart')
>
> 7) Stop this instance and take an AMI of it. Use that AMI for nodes on your Starcluster clusters. If it bothers you to have Starcluster installed on the AMI you are using in your Starcluster clusters skip steps 2 and 4 BUT be certain to edit the version of Starcluster software you are using to create your Ubuntu 14.04 clusters to deal with portmap/rpcbind issue as described in step 6.
>
> A few things to note:
>
> 1) I only use starcluster commands to create clusters, terminate them, addnodes and removenodes (on-demand & spots) -- and this is what has been working for me. I haven't tested the other functionality, so there could be other issues like the portmap one that I don't know about. For the other functionality (like stopping instances, saving AMIs, etc), I use AWS directly to perform those functions.
> 2) The version of StarCluster that I used was 0.95.6 -- as available via easy_install. These issues (or ones similar to them) have been discussed and a few fixed in some of the development branches of Starcluster available on github see here and here.
> 3) I had some issue installing other software, but I am guessing most of that has been resolved now that it has been so long since the initial release of Ubuntu 14.04.
>
> Good Luck,
> Jennifer
>
>
> On 1/27/15 3:59 PM, Pablo Escobar wrote:
>> Hi all,
>>
>> I plan to use starcluster with ubuntu14.04. I will adapt the ami following this guide
>> https://www.youtube.com/watch?v=2RBupgpi_ec
>>
>> Is anyone in the list using starcluster with ubuntu 14.04 in production? Is it working fine or is there any drawback? is there any problem I should be aware before start using starcluster with ubuntu14.04?
>>
>> I know this has been asked before, but is there any plan to release a ubuntu14.04 official ami?
>>
>> thanks in advance for any advice.
>>
>> best regards,
>> Pablo.
>>
>>
>> _______________________________________________
>> StarCluster mailing list
>>
>> StarCluster_at_mit.edu
>> http://mailman.mit.edu/mailman/listinfo/starcluster
>
>
> _______________________________________________
> StarCluster mailing list
> StarCluster_at_mit.edu
> http://mailman.mit.edu/mailman/listinfo/starcluster
Received on Mon Feb 16 2015 - 12:27:39 EST