Re: Using miniconda on EBS volume to run ipython notebook with ipcluster plugin on starcluster

From: Dylan Kotliar <no email>
Date: Fri, 2 Dec 2016 01:29:21 -0500

Hi Jacob,

Thanks for the response. Since IPython is installed on the EBS volume, it
should be accessible by all nodes. The only issue is that the EBS volume
needs to be on the python path at the time the plugin execute. I have
gotten around this by having a .bashrc file with the lines:

export PATH="path_to_EBS_volume/bin/miniconda2/bin:$PATH"
source activate python2

that gets copied to the root directory upon startup. However, this .bashrc
file is not getting sourced prior to running the ipcluster plugin. I tried
to address this by extending the IPCluster class to include the following
in the run method:

    for node in nodes:
        node.ssh.execute("source ~/.bashrc")

However, I still get the "!!! ERROR - IPython is not installed!" error. If
I comment out the call to _check_ipython_installed, it fails further on
with ipython not found.

Thanks,
Dylan



On Thu, Dec 1, 2016 at 2:38 PM, Jacob Barhak <jacob.barhak_at_gmail.com> wrote:

> So Dylan,
>
> Did you install Ipython on all nodes?
>
> It is the first thing that comes to mind.
>
> There are plugins that come with starcluster to install other software
> through pip. Did you use one of there?
>
> The third thing at comes to mind is using the Ipython plugin:
>
> http://star.mit.edu/cluster/docs/latest/plugins/ipython.
> html#ipcluster-plugin
>
> It is hard to fully understand what you do, yet I hope on of these paths
> will be relevant towards a solution.
>
> Jacob
>
> On Nov 30, 2016 5:06 PM, "Dylan Kotliar" <dkotliar_at_broadinstitute.org>
> wrote:
>
>> Hello all,
>>
>> I am currently using Starcluster by mounting an EBS volume containing
>> much of the software I want to use including python as version controlled
>> by miniconda. I have written a plugin that copies necessary hidden files
>> (e.g. .bashrc) to the home directory so that the environment is configured
>> when I ssh to the master node and the python stored on the EBS volume is
>> used by default.
>>
>> Now I would like to use the ipcluster plugin to open ipython notebooks in
>> the cluster. However, I get the error message:
>>
>> Running plugin starcluster.plugins.ipcluster.IPCluster !!! ERROR - Error
>> occured while running plugin 'starcluster.plugins.ipcluster.IPCluster':
>> !!! ERROR - IPython is not installed!
>>
>> This must be because the ipcluster plugin is using the default version of
>> python rather than my version controlled version. This is despite the fact
>> that when I ssh to the cluster, the python in the environment is the
>> correct one (the one on the EBS volume). Any suggestions on how to
>> circumvent this?
>>
>> Thanks!
>>
>> _______________________________________________
>> StarCluster mailing list
>> StarCluster_at_mit.edu
>> http://mailman.mit.edu/mailman/listinfo/starcluster
>>
>>

-- 
Dylan Kotliar,
Harvard Medical School (HST) / Systems Biology PhD Program
MD/PhD candidate
Phone: 732-379-0215

Received on Fri Dec 02 2016 - 01:29:53 EST