Re: Using miniconda on EBS volume to run ipython notebook with ipcluster plugin on starcluster

From: Dylan Kotliar <no email>
Date: Wed, 7 Dec 2016 10:11:38 -0500

Hi Teddy,

Thanks for the reply. You are spot on about my use case--I am currently
trying out many different python environment configurations and different
cluster users will want to be managing their own environments. Updating
package installation plugins would be one solution but I was trying out
this solution instead. Up until this issue, it worked without a hitch. And
thanks for the tip about appending the miniconda path to /etc/profile.d this
solved the problem!

Best,
Dylan

On Mon, Dec 5, 2016 at 12:24 PM, Teddy Thomas <tthoma24_at_mit.edu> wrote:

> Hi Dylan-
>
> One way you could accomplish what you want would be to write a plugin that
> modifies /etc/profile.d, rather than just the local homedir, on your
> machines to include the EBS volume where you’ve included the software in
> your PATH variable. By default, the profile in /etc/profile.d or
> /etc/bashrc should be being sourced.
>
> I don’t think Jacob’s impression is correct that what you wanted is
> considered “taking a shortcut.” I’m sure you have a use case for doing what
> you want the way you’re trying to. One use case I see for this is having a
> lot of different packages, and people needing to use different version of
> the packages simultaneously. In our use of starcluster, we’ve found that it
> gets painful to need to update plugins to install things with apt on
> starcluster.
>
> -Teddy
>
> On Dec 4, 2016, at 5:43 PM, Dylan Kotliar <dkotliar_at_broadinstitute.org>
> wrote:
>
>
> OK, I have come around to this. Thanks!
>
> On Fri, Dec 2, 2016 at 2:30 AM, Jacob Barhak <jacob.barhak_at_gmail.com>
> wrote:
>
>> So Dylan,
>>
>> It seems you are trying to bypass installation. I suggest avoiding
>> shortcuts.
>>
>> Using code in centralized storage also implies you need to make sure code
>> is accessible.
>>
>> If you install whatever you need on each node upon startup you should be
>> good unless I am missing somethings.
>>
>> In any case, getting back to basics and simplifying things may help.
>>
>> Good luck.
>>
>> Jacob
>>
>> On Dec 2, 2016 1:29 AM, "Dylan Kotliar" <dkotliar_at_broadinstitute.org>
>> wrote:
>>
>>> Hi Jacob,
>>>
>>> Thanks for the response. Since IPython is installed on the EBS volume,
>>> it should be accessible by all nodes. The only issue is that the EBS volume
>>> needs to be on the python path at the time the plugin execute. I have
>>> gotten around this by having a .bashrc file with the lines:
>>>
>>> export PATH="path_to_EBS_volume/bin/miniconda2/bin:$PATH"
>>> source activate python2
>>>
>>> that gets copied to the root directory upon startup. However, this
>>> .bashrc file is not getting sourced prior to running the ipcluster plugin.
>>> I tried to address this by extending the IPCluster class to include the
>>> following in the run method:
>>>
>>> for node in nodes:
>>> node.ssh.execute("source ~/.bashrc")
>>>
>>> However, I still get the "!!! ERROR - IPython is not installed!" error.
>>> If I comment out the call to _check_ipython_installed, it fails further on
>>> with ipython not found.
>>>
>>> Thanks,
>>> Dylan
>>>
>>>
>>>
>>> On Thu, Dec 1, 2016 at 2:38 PM, Jacob Barhak <jacob.barhak_at_gmail.com>
>>> wrote:
>>>
>>>> So Dylan,
>>>>
>>>> Did you install Ipython on all nodes?
>>>>
>>>> It is the first thing that comes to mind.
>>>>
>>>> There are plugins that come with starcluster to install other software
>>>> through pip. Did you use one of there?
>>>>
>>>> The third thing at comes to mind is using the Ipython plugin:
>>>>
>>>> http://star.mit.edu/cluster/docs/latest/plugins/ipython.html
>>>> #ipcluster-plugin
>>>>
>>>> It is hard to fully understand what you do, yet I hope on of these
>>>> paths will be relevant towards a solution.
>>>>
>>>> Jacob
>>>>
>>>> On Nov 30, 2016 5:06 PM, "Dylan Kotliar" <dkotliar_at_broadinstitute.org>
>>>> wrote:
>>>>
>>>>> Hello all,
>>>>>
>>>>> I am currently using Starcluster by mounting an EBS volume containing
>>>>> much of the software I want to use including python as version controlled
>>>>> by miniconda. I have written a plugin that copies necessary hidden files
>>>>> (e.g. .bashrc) to the home directory so that the environment is configured
>>>>> when I ssh to the master node and the python stored on the EBS volume is
>>>>> used by default.
>>>>>
>>>>> Now I would like to use the ipcluster plugin to open ipython notebooks
>>>>> in the cluster. However, I get the error message:
>>>>>
>>>>> Running plugin starcluster.plugins.ipcluster.IPCluster !!! ERROR -
>>>>> Error occured while running plugin 'starcluster.plugins.ipcluster.IPCluster':
>>>>> !!! ERROR - IPython is not installed!
>>>>>
>>>>> This must be because the ipcluster plugin is using the default version
>>>>> of python rather than my version controlled version. This is despite the
>>>>> fact that when I ssh to the cluster, the python in the environment is the
>>>>> correct one (the one on the EBS volume). Any suggestions on how to
>>>>> circumvent this?
>>>>>
>>>>> Thanks!
>>>>>
>>>>> _______________________________________________
>>>>> StarCluster mailing list
>>>>> StarCluster_at_mit.edu
>>>>> http://mailman.mit.edu/mailman/listinfo/starcluster
>>>>>
>>>>>
>>>
>>>
>>> --
>>> Dylan Kotliar,
>>> Harvard Medical School (HST) / Systems Biology PhD Program
>>> MD/PhD candidate
>>> Phone: 732-379-0215 <(732)%20379-0215>
>>>
>>
>
>
> --
> Dylan Kotliar,
> Harvard Medical School (HST) / Systems Biology PhD Program
> MD/PhD candidate
> Phone: 732-379-0215 <(732)%20379-0215>
> _______________________________________________
> StarCluster mailing list
> StarCluster_at_mit.edu
> http://mailman.mit.edu/mailman/listinfo/starcluster
>
>
>

-- 
Dylan Kotliar,
Harvard Medical School (HST) / Systems Biology PhD Program
MD/PhD candidate
Phone: 732-379-0215

Received on Wed Dec 07 2016 - 10:12:11 EST