Re: advice about using starcluster with ubuntu 14.04

From: Pablo Escobar <no email>
Date: Thu, 29 Jan 2015 15:41:22 +0100

Hi Jennifer

thank you very much for your help. It's really useful.

regards,
Pablo.

2015-01-28 19:10 GMT+01:00 Jennifer Staab <jstaab_at_cs.unc.edu>:

> I installed Starcluster on Ubuntu 14.04 LTS and it is working fine for
> me. I didn't use the guide you mentioned, just a little guidance from
> emails on mailing list between Jacob and Hugh regarding installation (see
> here <http://star.mit.edu/cluster/mlarchives/2389.html>) and some
> sleuthing to get it to work. Hugh also provides a guideline/cookbook for
> creating new Starcluster AMI's here
> <http://star.mit.edu/cluster/mlarchives/2353.html> on the mailing list
> and said that just upgrading the existing AMI for 12.04 to 14.04 worked for
> him. To me that seems like the easiest way to handle this whole issue is to
> use existing Ubuntu 12.04 Starcluster AMI and just upgrade it to 14.04. I
> wasn't able to do this, so below is what worked for me. I am sure there are
> other ways to create a new Ubuntu 14.04 Starcluster AMI.
>
>
> --------------------------------------------------------------------------------------
> 1) Locate the AMI ID of the latest version of Ubuntu 14.04 for your region
> (see here <http://cloud-images.ubuntu.com/locator/ec2/>). Start up your
> Ubuntu 14.04 AMI -- login and run update and apply all the security and
> distribution updates.
>
> 2) Install packages "python-dev" and "python-setuptools" -- You are
> welcome to install other packages, but those are the only ones needed if to
> do the Starcluster install. A list of all python packages can be found
> here <http://packages.ubuntu.com/source/trusty/python/> .
>
> 3) You need to install NFS server so install package "nfs-kernel-server".
> Installing this package should install its dependencies (for me that
> included: nfs-common rpcbind libgssglue1 libnfsidmap2 libtirpc1). One issue
> I found was that in previous versions the startup script located in
> /etc/init.d/ was named "nfs" this installed version it's named
> "nfs-kernel-server". The easiest fix is to create a symbolic link to
> "/etc/init.d/nfs" (ln -s /etc/init.d/nfs-kernel-server /etc/init.d/nfs).
> Test by issuing the command "/etc/init.d/nfs restart" if it restarts then
> all is well.
>
> 4) Install Starcluster with "easy_install Starcluster"
>
> 5) You need to get a copy of the Open Grid Scheduler and place it in
> "/opt/sge6-fresh/" -- this is what Hugh and Jacob discuss in their emails
> on the mailing list. The easiest way I was able to do this was to use a
> running Ubuntu 12.04 Starcluster AMI (ami-765b3e1f) and copy
> "/opt/sge-fresh" from the 12.04 version to my Ubuntu 14.04 version. There
> are other ways to do this see their emails here
> <http://star.mit.edu/cluster/mlarchives/2389.html>.
>
> 6) Ubuntu 14.04 replaces portmap with rpcbind -- so this was an issue for
> me because the Starcluster software expects "portmap" to exist. So this is
> where I had to "edit" the code a bit.There are other ways to deal with the
> portmap issue, look here
> <http://star.mit.edu/cluster/mlarchives/2353.html> for an alternative
> solution. This is just how I handled the issue: locate the "node.py"
> portion of the code in the installed StarCluster version, mine was in
> "/usr/local/lib/python2.7/dist-packages/StarclusterXXX.egg/starcluster/node.py"
> On line 731 in the function "mount_nfs_shares()" you see line
> "self.ssh.execute('/etc/init.d/portmap start')". This won't work on Ubuntu
> 14.04 since "portmap" was replaced with "rpcbind". One way to fix this
> issue without breaking the code for other OS versions still using portmap
> is by replacing "self.ssh.execute('/etc/init.d/portmap start')" with a
> "try/except" statement so that if portmap doesn't work it calls rpcbind
> command instead; thus, handling the issue with portmap on Ubuntu 14.04.
> Below is how I changed the code in node.py:
> FIND line 731:
> self.ssh.execute('/etc/init.d/portmap start')
> REPLACE with:
> try:
> self.ssh.execute('/etc/init.d/portmap start')
> except:
> self.ssh.execute('service rpcbind restart')
>
> 7) Stop this instance and take an AMI of it. Use that AMI for nodes on
> your Starcluster clusters. If it bothers you to have Starcluster installed
> on the AMI you are using in your Starcluster clusters skip steps 2 and 4
> BUT be certain to edit the version of Starcluster software you are using to
> create your Ubuntu 14.04 clusters to deal with portmap/rpcbind issue as
> described in step 6.
>
> A few things to note:
>
> 1) I only use starcluster commands to create clusters, terminate them,
> addnodes and removenodes (on-demand & spots) -- and this is what has been
> working for me. I haven't tested the other functionality, so there could be
> other issues like the portmap one that I don't know about. For the other
> functionality (like stopping instances, saving AMIs, etc), I use AWS
> directly to perform those functions.
> 2) The version of StarCluster that I used was 0.95.6 -- as available via
> easy_install. These issues (or ones similar to them) have been discussed
> and a few fixed in some of the development branches of Starcluster
> available on github see here
> <https://github.com/jtriley/StarCluster/issues?q=portmap> and here
> <https://github.com/jtriley/StarCluster/issues?q=ubuntu+14.04>.
> 3) I had some issue installing other software, but I am guessing most of
> that has been resolved now that it has been so long since the initial
> release of Ubuntu 14.04.
>
> Good Luck,
> Jennifer
>
>
> On 1/27/15 3:59 PM, Pablo Escobar wrote:
>
> Hi all,
>
> I plan to use starcluster with ubuntu14.04. I will adapt the ami
> following this guide
> https://www.youtube.com/watch?v=2RBupgpi_ec
>
> Is anyone in the list using starcluster with ubuntu 14.04 in production?
> Is it working fine or is there any drawback? is there any problem I should
> be aware before start using starcluster with ubuntu14.04?
>
> I know this has been asked before, but is there any plan to release a
> ubuntu14.04 official ami?
>
> thanks in advance for any advice.
>
> best regards,
> Pablo.
>
>
> _______________________________________________
> StarCluster mailing listStarCluster_at_mit.eduhttp://mailman.mit.edu/mailman/listinfo/starcluster
>
>
>
Received on Thu Jan 29 2015 - 09:41:45 EST