https://github.com/datacratic/StarCluster/issues/56
My immediate need is ubuntu14 or 16 amis in non-us-east-1 regions, and I'm sure I'm not the only one. The current situation with broken ubuntu13 amis still present in listpublic is definitely suboptimal.
So I'm thinking about introducing a new command into starcluster which will allow it to make it's own amis. This might also allow you to 'bake' an ami after all of your plugin installs have completed, to reduce setup time for new worker nodes.
it would largely follow the steps from
http://star.mit.edu/cluster/mlarchives/2545.html
and would probably have a command line interface similar to
starcluster makeami --baseami ami-ed82e39e
obviously a new command is a more dramatic change than a new plugin. So before I jump in I thought I'd see if others would weigh in.
Is it useful? or perhaps impractical for reasons I'm overlooking?
Michael Cariaso
<mailto:michael.cariaso_at_keygene.com>
Bioinformatician<
http://www.keygene.com>
________________________________
From: starcluster-bounces_at_mit.edu <starcluster-bounces_at_mit.edu> on behalf of Avner May <avnermay_at_cs.columbia.edu>
Sent: Saturday, January 30, 2016 1:58 AM
To: Ed Gray
Cc: starcluster_at_mit.edu; Don Morton
Subject: Re: [StarCluster] Updating public AMIs to have Ubuntu 14.04? Or how to install starcluster on AWS AMI?
I just wanted to give one final request for there being public Ubuntu 14.04 StarCluster AMIs (EBS and HVM-EBS), which are listed in the output of the "starcluster listpublic", so that other people don't go through the painful experience I had to go through.
What is the process for adding a new public AMI to StarCluster?
Sergio Mafra has pointed out to me that ami-38b99850 is available and has StarCluster installed on Ubuntu 14.04.
On Thu, Jan 28, 2016 at 9:34 PM, Ed Gray <gray_ed_at_hotmail.com<mailto:gray_ed_at_hotmail.com>> wrote:
Having worked with Jennifer, I can tell you she's a star.
Sent from my Smartphone; please forgive any spelling and grammar mishaps.
-------- Original message --------
From: Avner May <avnermay_at_cs.columbia.edu<mailto:avnermay_at_cs.columbia.edu>>
Date: 01/28/2016 6:13 PM (GMT-05:00)
To: Don Morton <don.morton_at_borealscicomp.com<mailto:don.morton_at_borealscicomp.com>>, Jennifer Staab <jstaab_at_cs.unc.edu<mailto:jstaab_at_cs.unc.edu>>
Cc: starcluster_at_mit.edu<mailto:starcluster_at_mit.edu>
Subject: Re: [StarCluster] Updating public AMIs to have Ubuntu 14.04? Or how to install starcluster on AWS AMI?
Hi all,
I have carefully followed Jennifer Staab's instructions here<
http://star.mit.edu/cluster/mlarchives/2545.html>, and things are working great (*** THANK YOU SO MUCH JENNIFER!! ***). One small hiccup I ran into is that at that link, one of the commands is long and is split into 2 lines ('apt-get install nfs-kernel-server nfs-common rpcbind libgssglue1 libnfsidmap2 libtirpc1 -y") -- if this is copy-pasted as 2 lines, it will fail quietly when instance is launched, and it eventually leads to an error starting NFS on the master (see error message below).
I also followed the instructions here<
https://github.com/BVLC/caffe/wiki/Install-Caffe-on-EC2-from-scratch-(Ubuntu,-CUDA-7,-cuDNN)> in order to install NVIDIA drivers and CUDA (though I installed the latest versions, not the ones included in that post).
I would recommend including a modified version of Jennifer's instructions directly linked from the primary StarCluster website in order to explain how to set-up a non-StarCluster AMI to work with StarCluster. This is super valuable information.
Now everything is working!!!
Avner
========================
Starting NFS server on master
!!! ERROR - Error occured while running plugin 'starcluster.clustersetup.DefaultClusterSetup':
!!! ERROR - remote command 'source /etc/profile && /etc/init.d/nfs
!!! ERROR - start' failed with status 127:
!!! ERROR - bash: /etc/init.d/nfs: No such file or directory
On Thu, Jan 28, 2016 at 3:01 PM, Don Morton <don.morton_at_borealscicomp.com<mailto:don.morton_at_borealscicomp.com>> wrote:
Hello,
I faced this same issue a few months ago, and somehow succeeded after bumping into lots of walls (most of it based on my own ignorance). I haven't done anything with this since, and these notes are extremely rough (hopefully I didn't cuss too much!), but you are welcome to use what you need. Maybe someone will have the time to make something more formal, but I'm afraid it won't be me for some time now.
All the best,
Don
---
Don Morton, Owner/Manager
Boreal Scientific Computing LLC
Fairbanks, Alaska USA
http://www.borealscicomp.com/
http://www.borealscicomp.com/Miscellaneous/MortonBio/
On Thu, Jan 28, 2016 at 7:42 PM, Avner May <avnermay_at_cs.columbia.edu<mailto:avnermay_at_cs.columbia.edu>> wrote:
I see that this has already been asked several times: http://star.mit.edu/cluster/mlarchives/2690.html, http://star.mit.edu/cluster/mlarchives/2545.html, https://www.youtube.com/watch?v=2RBupgpi_ec, etc.
Which of these guides should I use? Why hasn't an Ubuntu 14.04 AMI been made public by the StarCluster development team yet? Are there plans to do this? Meanwhile, could someone make their AMI public?
It seems there is widespread interest in updated AMIs (EBS and HVM-EBS versions).
Thanks so much,
Avner
On Thu, Jan 28, 2016 at 2:30 PM, Avner May <avnermay_at_cs.columbia.edu<mailto:avnermay_at_cs.columbia.edu>> wrote:
Hi,
My goal is to have a StarCluster AMI with Ubuntu 14.04, CUDA 7.5, and the latest NVIDIA drivers (I want to use g2.2xlarge instance type).
I have noticed that the public StarCluster AMIs are pretty outdated, with the most up-to-date AMIs using Ubuntu 13.04, which has already reached its "end of life<http://fridge.ubuntu.com/2014/01/28/ubuntu-13-04-raring-ringtail-end-of-life-reached-on-january-27-2014/>". Could an Ubuntu 14.04 + 64-bit + HVM-EBS public AMI be created, with CUDA 7.5 and the latest NVIDIA driver<http://www.nvidia.com/download/driverResults.aspx/97645/en-us> installed? This would be incredibly useful. I am having a very hard time upgrading ami-6b211202 for my needs (I have run into issues updating ubuntu, as well as installing NVIDIA drivers and CUDA).
Here<http://tleyden.github.io/blog/2014/10/25/cuda-6-dot-5-on-aws-gpu-instance-running-ubuntu-14-dot-04/> are instructions on how to install NVIDIA drivers and CUDA on Ubuntu 14.04 AMI (ami-9eaa1cf6). So another option which would satisfy my needs would be if you simply gave me instructions on how to install StarCluster on this AMI.
Any advice on the easiest way to get a working StarCluster AMI, with an up to date version of Ubuntu, would be great.
Thanks!
Avner
_______________________________________________
StarCluster mailing list
StarCluster_at_mit.edu<mailto:StarCluster_at_mit.edu>
http://mailman.mit.edu/mailman/listinfo/starcluster
Received on Thu Aug 25 2016 - 00:15:03 EDT