Re: Using miniconda on EBS volume to run ipython notebook with ipcluster plugin on starcluster

From: Dylan Kotliar <no email>
Date: Sun, 4 Dec 2016 17:43:15 -0500

OK, I have come around to this. Thanks!

On Fri, Dec 2, 2016 at 2:30 AM, Jacob Barhak <jacob.barhak_at_gmail.com> wrote:

> So Dylan,
>
> It seems you are trying to bypass installation. I suggest avoiding
> shortcuts.
>
> Using code in centralized storage also implies you need to make sure code
> is accessible.
>
> If you install whatever you need on each node upon startup you should be
> good unless I am missing somethings.
>
> In any case, getting back to basics and simplifying things may help.
>
> Good luck.
>
> Jacob
>
> On Dec 2, 2016 1:29 AM, "Dylan Kotliar" <dkotliar_at_broadinstitute.org>
> wrote:
>
>> Hi Jacob,
>>
>> Thanks for the response. Since IPython is installed on the EBS volume, it
>> should be accessible by all nodes. The only issue is that the EBS volume
>> needs to be on the python path at the time the plugin execute. I have
>> gotten around this by having a .bashrc file with the lines:
>>
>> export PATH="path_to_EBS_volume/bin/miniconda2/bin:$PATH"
>> source activate python2
>>
>> that gets copied to the root directory upon startup. However, this
>> .bashrc file is not getting sourced prior to running the ipcluster plugin.
>> I tried to address this by extending the IPCluster class to include the
>> following in the run method:
>>
>> for node in nodes:
>> node.ssh.execute("source ~/.bashrc")
>>
>> However, I still get the "!!! ERROR - IPython is not installed!" error.
>> If I comment out the call to _check_ipython_installed, it fails further on
>> with ipython not found.
>>
>> Thanks,
>> Dylan
>>
>>
>>
>> On Thu, Dec 1, 2016 at 2:38 PM, Jacob Barhak <jacob.barhak_at_gmail.com>
>> wrote:
>>
>>> So Dylan,
>>>
>>> Did you install Ipython on all nodes?
>>>
>>> It is the first thing that comes to mind.
>>>
>>> There are plugins that come with starcluster to install other software
>>> through pip. Did you use one of there?
>>>
>>> The third thing at comes to mind is using the Ipython plugin:
>>>
>>> http://star.mit.edu/cluster/docs/latest/plugins/ipython.html
>>> #ipcluster-plugin
>>>
>>> It is hard to fully understand what you do, yet I hope on of these paths
>>> will be relevant towards a solution.
>>>
>>> Jacob
>>>
>>> On Nov 30, 2016 5:06 PM, "Dylan Kotliar" <dkotliar_at_broadinstitute.org>
>>> wrote:
>>>
>>>> Hello all,
>>>>
>>>> I am currently using Starcluster by mounting an EBS volume containing
>>>> much of the software I want to use including python as version controlled
>>>> by miniconda. I have written a plugin that copies necessary hidden files
>>>> (e.g. .bashrc) to the home directory so that the environment is configured
>>>> when I ssh to the master node and the python stored on the EBS volume is
>>>> used by default.
>>>>
>>>> Now I would like to use the ipcluster plugin to open ipython notebooks
>>>> in the cluster. However, I get the error message:
>>>>
>>>> Running plugin starcluster.plugins.ipcluster.IPCluster !!! ERROR -
>>>> Error occured while running plugin 'starcluster.plugins.ipcluster.IPCluster':
>>>> !!! ERROR - IPython is not installed!
>>>>
>>>> This must be because the ipcluster plugin is using the default version
>>>> of python rather than my version controlled version. This is despite the
>>>> fact that when I ssh to the cluster, the python in the environment is the
>>>> correct one (the one on the EBS volume). Any suggestions on how to
>>>> circumvent this?
>>>>
>>>> Thanks!
>>>>
>>>> _______________________________________________
>>>> StarCluster mailing list
>>>> StarCluster_at_mit.edu
>>>> http://mailman.mit.edu/mailman/listinfo/starcluster
>>>>
>>>>
>>
>>
>> --
>> Dylan Kotliar,
>> Harvard Medical School (HST) / Systems Biology PhD Program
>> MD/PhD candidate
>> Phone: 732-379-0215 <(732)%20379-0215>
>>
>

-- 
Dylan Kotliar,
Harvard Medical School (HST) / Systems Biology PhD Program
MD/PhD candidate
Phone: 732-379-0215

Received on Sun Dec 04 2016 - 17:43:48 EST