OK,
Farther along.
I made an additional patch to cluster.py, in load_plugins():
old code:
for plugin in plugins:
new code:
for key in plugins:
plugin = plugins[key]
That gets me past that trouble spot.
Then, though, when I try and run my plugin from the shell, I get this:
In [1]: cluster.run_plugin("setup","myfirstcluster",cfg)
>>> Using private key /Users/dante/.ec2/bioc-keypair.pem.rsa (rsa)
---------------------------------------------------------------------------
AttributeError Traceback (most recent call last)
/Library/Python/2.6/site-packages/StarCluster-0.91.2-py2.6.egg/starcluster/cli.pyc
in <module>()
----> 1
2
3
4
5
/Library/Python/2.6/site-packages/StarCluster-0.91.2-py2.6.egg/starcluster/cluster.pyc
in run_plugin(plugin_name, cluster_tag, cfg)
178 master = cl.master_node
179 for p in plugins:
--> 180 p.run(cl.nodes, cl.master_node, cl.cluster_user,
cl.cluster_shell,
181 volumes)
182
AttributeError: 'tuple' object has no attribute 'run'
In [2]:
It looks like "p" is a tuple but it needs to be an instance of my plugin.
Incidentally I am trying this with the testplugins from the StarCluster
distribution.
Can someone suggest a patch for this?
Thanks!
Dan
On Sat, Oct 30, 2010 at 12:40 PM, Dan Tenenbaum <dtenenba_at_fhcrc.org> wrote:
> OK, so I've gone back to vanilla StarCluster with the patch Marc suggests
> above (adding an argument to the call to get_cluster() in run_plugins().
>
> Then I get the weird:
> AttributeError: 'str' object has no attribute 'get'
>
> I have put some print statements in to try and clarify things and here is
> the output:
>
> before load_plugins, plugins is
> {'shortreadplugin': {'setup_class': 'bioconductor.ShortReadPlugin', 'arg1':
> 'foo', '__name__': 'shortreadplugin'}}
> plugin:
> shortreadplugin
>
> So this tells me that I am passing a hash to load_plugins(), but
> load_plugins() is receiving a string! Why? Is this some python thing?
>
> Sorry for spamming the list so much with this but I'm sort of under the gun
> on this project....
>
> Thanks
> Dan
>
>
> On Sat, Oct 30, 2010 at 12:16 PM, Dan Tenenbaum <dtenenba_at_fhcrc.org>wrote:
>
>> Something else is going on....the shell doesn't seem to recognize any of
>> the methods in "cluster" as belonging to ClusterManager. I'm not enough of a
>> python developer to know why this might be.
>> Dan
>>
>>
>>
>> On Sat, Oct 30, 2010 at 12:05 PM, Marc Resnick <mresnick_at_mit.edu> wrote:
>>
>>> Got me stumped. If I remember correctly, that function worked for me
>>> the last time I tried it...
>>>
>>> On Sat, Oct 30, 2010 at 2:57 PM, Dan Tenenbaum <dtenenba_at_fhcrc.org>
>>> wrote:
>>> > Looks like there are a number of files in this fork that don't exist in
>>> the
>>> > 0.91 that I downloaded.
>>> > So just downloaded your whole fork and built that, then got this in the
>>> > shell:
>>> > In [1]: cluster.run_plugin("shortreadplugin","myfirstcluster",cfg)
>>> >
>>> ---------------------------------------------------------------------------
>>> > AttributeError Traceback (most recent call
>>> last)
>>> >
>>> /Library/Python/2.6/site-packages/StarCluster-0.9999-py2.6.egg/starcluster/commands/shell.pyc
>>> > in <module>()
>>> > ----> 1
>>> > 2
>>> > 3
>>> > 4
>>> > 5
>>> > AttributeError: 'module' object has no attribute 'run_plugin'
>>> > In [2]:
>>> > Dan
>>> >
>>> > On Sat, Oct 30, 2010 at 11:45 AM, Dan Tenenbaum <dtenenba_at_fhcrc.org>
>>> wrote:
>>> >>
>>> >> Keeping this on the list.
>>> >> BTW, the new cluster.py has no syntax errors, so something else must
>>> be
>>> >> happening.
>>> >> Dan
>>> >>
>>> >> On Sat, Oct 30, 2010 at 11:44 AM, Dan Tenenbaum <dtenenba_at_fhcrc.org>
>>> >> wrote:
>>> >>>
>>> >>> Thanks. That gave me a different error, below.
>>> >>> The first time this happened, I did:
>>> >>> sudo easy_install iptools, but it is still happening.
>>> >>> Dan
>>> >>> $ starcluster shell
>>> >>> Leopard libedit detected.
>>> >>> ERROR: An unexpected error occurred while tokenizing input
>>> >>> The following traceback may be corrupted or invalid
>>> >>> The error message is: ('EOF in multi-line statement', (405, 0))
>>> >>>
>>> >>>
>>> ---------------------------------------------------------------------------
>>> >>> ImportError Traceback (most recent call
>>> >>> last)
>>> >>>
>>> >>>
>>> /Library/Python/2.6/site-packages/StarCluster-0.91-py2.6.egg/EGG-INFO/scripts/starcluster
>>> >>> in <module>()
>>> >>> 3 __requires__ = 'StarCluster==0.91'
>>> >>> 4 import pkg_resources
>>> >>> ----> 5 pkg_resources.run_script('StarCluster==0.91', 'starcluster')
>>> >>> 6
>>> >>> 7
>>> >>>
>>> >>>
>>> /System/Library/Frameworks/Python.framework/Versions/2.6/Extras/lib/python/pkg_resources.pyc
>>> >>> in run_script(self, requires, script_name)
>>> >>> 440 ns.clear()
>>> >>> 441 ns['__name__'] = name
>>> >>> --> 442 self.require(requires)[0].run_script(script_name, ns)
>>> >>> 443
>>> >>> 444
>>> >>>
>>> >>>
>>> /System/Library/Frameworks/Python.framework/Versions/2.6/Extras/lib/python/pkg_resources.pyc
>>> >>> in run_script(self, script_name, namespace)
>>> >>> 1165 )
>>> >>> 1166 script_code =
>>> >>> compile(script_text,script_filename,'exec')
>>> >>> -> 1167 exec script_code in namespace, namespace
>>> >>> 1168
>>> >>> 1169 def _has(self, path):
>>> >>>
>>> >>>
>>> /Library/Python/2.6/site-packages/StarCluster-0.91-py2.6.egg/EGG-INFO/scripts/starcluster
>>> >>> in <module>()
>>> >>> 1
>>> >>> ----> 2 #!/usr/bin/python
>>> >>> 3 # EASY-INSTALL-SCRIPT: 'StarCluster==0.91','starcluster'
>>> >>> 4 __requires__ = 'StarCluster==0.91'
>>> >>> 5 import pkg_resources
>>> >>> 6 pkg_resources.run_script('StarCluster==0.91', 'starcluster')
>>> >>>
>>> >>>
>>> /Users/dante/.starcluster/build/bdist.macosx-10.6-universal/egg/starcluster/cli.py
>>> >>> in <module>()
>>> >>> 36
>>> >>> 37
>>> >>> ---> 38
>>> >>> 39
>>> >>> 40
>>> >>>
>>> >>>
>>> /Users/dante/.starcluster/build/bdist.macosx-10.6-universal/egg/starcluster/cluster.py
>>> >>> in <module>()
>>> >>> 10
>>> >>> 11
>>> >>> ---> 12
>>> >>> 13
>>> >>> 14
>>> >>> ImportError: cannot import name iptools
>>> >>>
>>> >>> On Sat, Oct 30, 2010 at 11:38 AM, Marc Resnick <mresnick_at_mit.edu>
>>> wrote:
>>> >>>>
>>> >>>> The version here
>>> >>>>
>>> >>>>
>>> http://github.com/jtriley/StarCluster/blob/experimental/starcluster/cluster.py
>>> >>>> may have those bugs fixed, as I remember that one as well.
>>> >>>>
>>> >>>>
>>> >>>> Marc
>>> >>>>
>>> >>>> On Sat, Oct 30, 2010 at 2:31 PM, Dan Tenenbaum <dtenenba_at_fhcrc.org>
>>> >>>> wrote:
>>> >>>> > I already did that. ;)
>>> >>>> > Now I'm getting another error:
>>> >>>> > In [1]: cluster.run_plugin("shortreadplugin","myfirstcluster",cfg)
>>> >>>> >>>> Using private key /Users/dante/.ec2/bioc-keypair.pem.rsa (rsa)
>>> >>>> >
>>> >>>> >
>>> ---------------------------------------------------------------------------
>>> >>>> > AttributeError Traceback (most recent
>>> call
>>> >>>> > last)
>>> >>>> >
>>> >>>> >
>>> /Library/Python/2.6/site-packages/StarCluster-0.91-py2.6.egg/starcluster/cli.pyc
>>> >>>> > in <module>()
>>> >>>> > ----> 1
>>> >>>> > 2
>>> >>>> > 3
>>> >>>> > 4
>>> >>>> > 5
>>> >>>> >
>>> >>>> >
>>> /Library/Python/2.6/site-packages/StarCluster-0.91-py2.6.egg/starcluster/cluster.pyc
>>> >>>> > in run_plugin(plugin_name, cluster_tag, cfg)
>>> >>>> > 175 plugins = {}
>>> >>>> > 176 plugins[plugin_name] = plug
>>> >>>> > --> 177 plugins = cl.load_plugins(plugins)
>>> >>>> > 178 master = cl.master_node
>>> >>>> > 179 for p in plugins:
>>> >>>> >
>>> >>>> >
>>> /Library/Python/2.6/site-packages/StarCluster-0.91-py2.6.egg/starcluster/cluster.pyc
>>> >>>> > in load_plugins(self, plugins)
>>> >>>> > 314 plugs = []
>>> >>>> > 315 for plugin in plugins:
>>> >>>> > --> 316 setup_class = plugin.get('setup_class')
>>> >>>> > 317 plugin_name = plugin.get('__name__')
>>> >>>> > 318 mod_name =
>>> '.'.join(setup_class.split('.')[:-1])
>>> >>>> > AttributeError: 'str' object has no attribute 'get'
>>> >>>> > Hmm....
>>> >>>> > Dan
>>> >>>> >
>>> >>>> > On Sat, Oct 30, 2010 at 11:27 AM, Marc Resnick <mresnick_at_mit.edu>
>>> >>>> > wrote:
>>> >>>> >>
>>> >>>> >> Yes, that's the bug I told Justin about this Summer... I think I
>>> >>>> >> patched it, but he may have forgotten to pull from my fork.
>>> >>>> >>
>>> >>>> >> You can fix it by editing the run_plugin method, and replacing
>>> >>>> >>
>>> >>>> >> cl = get_cluster(cluster_tag)
>>> >>>> >>
>>> >>>> >> with
>>> >>>> >>
>>> >>>> >> cl = get_cluster(cluster_tag, cfg)
>>> >>>> >>
>>> >>>> >>
>>> >>>> >>
>>> >>>> >> Marc
>>> >>>> >>
>>> >>>> >>
>>> >>>> >> On Sat, Oct 30, 2010 at 2:20 PM, Dan Tenenbaum <
>>> dtenenba_at_fhcrc.org>
>>> >>>> >> wrote:
>>> >>>> >> > Hi Marc,
>>> >>>> >> > I am running 0.91.
>>> >>>> >> > I was doing a couple things wrong. I figured them out but I am
>>> >>>> >> > still
>>> >>>> >> > having
>>> >>>> >> > this problem.
>>> >>>> >> > One thing I did wrong was have the
>>> >>>> >> > plugins=shortreadplugin
>>> >>>> >> > line under the global section; it needed to be under the
>>> >>>> >> > [cluster smallcluster]
>>> >>>> >> > section.
>>> >>>> >> > When I start the cluster now, I can see the output from my
>>> plugin:
>>> >>>> >> >>>> Running plugin shortreadplugin
>>> >>>> >> >>>> about to echo
>>> >>>> >> > But, when I run the shell and put in this command:
>>> >>>> >> > cluster.run_plugin("shortreadplugin","myfirstcluster",cfg)
>>> >>>> >> > I get the same error described below.
>>> >>>> >> > Dan
>>> >>>> >> >
>>> >>>> >> > On Sat, Oct 30, 2010 at 11:11 AM, Marc Resnick <
>>> mresnick_at_mit.edu>
>>> >>>> >> > wrote:
>>> >>>> >> >>
>>> >>>> >> >> Hey Dan,
>>> >>>> >> >>
>>> >>>> >> >> Are you running the latest version? I saw that bug a few
>>> months
>>> >>>> >> >> ago, I
>>> >>>> >> >> thought I remember Justin fixing it.
>>> >>>> >> >>
>>> >>>> >> >>
>>> >>>> >> >> Marc
>>> >>>> >> >>
>>> >>>> >> >> On Sat, Oct 30, 2010 at 2:00 PM, Dan Tenenbaum
>>> >>>> >> >> <dtenenba_at_fhcrc.org>
>>> >>>> >> >> wrote:
>>> >>>> >> >> > Hi,
>>> >>>> >> >> > I am not much of a python programmer and I'm trying to
>>> figure
>>> >>>> >> >> > out the
>>> >>>> >> >> > plugin
>>> >>>> >> >> > system.
>>> >>>> >> >> > I created a ~/.starcluster/plugins directory and put a file
>>> in
>>> >>>> >> >> > it
>>> >>>> >> >> > called
>>> >>>> >> >> > shortread.py.
>>> >>>> >> >> > Here are its contents:
>>> >>>> >> >> >
>>> >>>> >> >> > from starcluster.clustersetup import ClusterSetup
>>> >>>> >> >> > from starcluster.logger import log
>>> >>>> >> >> > class ShortReadPlugin(ClusterSetup):
>>> >>>> >> >> > def __init__(self, arg1):
>>> >>>> >> >> > self.arg1 = arg1
>>> >>>> >> >> > log.info('arg1 = %s' % arg1)
>>> >>>> >> >> > def run(self, nodes, master, user, user_shell,
>>> volumes):
>>> >>>> >> >> > log.info("about to echo")
>>> >>>> >> >> > master.ssh.execute("echo 'hey'")
>>> >>>> >> >> > As you can see, it's just a test.
>>> >>>> >> >> > In my config, I put this at the bottom:
>>> >>>> >> >> > [plugin shortreadplugin]
>>> >>>> >> >> > SETUP_CLASS = ShortReadPlugin
>>> >>>> >> >> > arg1=foo
>>> >>>> >> >> > And above in the [global] section I put:
>>> >>>> >> >> > plugins=shortreadplugin
>>> >>>> >> >> > I'm a bit confused about whether this is right because in
>>> your
>>> >>>> >> >> > example:
>>> >>>> >> >> > http://web.mit.edu/stardev/cluster/docs/plugins.html
>>> >>>> >> >> > ...you have the SETUP_CLASS as ubuntu.PackageInstaller. I'm
>>> not
>>> >>>> >> >> > sure
>>> >>>> >> >> > where
>>> >>>> >> >> > the "ubuntu" fits into this. Is it like java where I need to
>>> >>>> >> >> > have a
>>> >>>> >> >> > folder
>>> >>>> >> >> > called ubuntu where the class resides?
>>> >>>> >> >> > Anyway, I fired up my cluster and went to test this with
>>> >>>> >> >> > "starcluster
>>> >>>> >> >> > shell"
>>> >>>> >> >> > and got the following output:
>>> >>>> >> >> > In [5]: cluster.run_plugin('shortreadplugin',
>>> 'myfirstcluster',
>>> >>>> >> >> > cfg)
>>> >>>> >> >> >
>>> >>>> >> >> >
>>> >>>> >> >> >
>>> >>>> >> >> >
>>> ---------------------------------------------------------------------------
>>> >>>> >> >> > TypeError Traceback (most
>>> recent
>>> >>>> >> >> > call
>>> >>>> >> >> > last)
>>> >>>> >> >> >
>>> >>>> >> >> >
>>> >>>> >> >> >
>>> >>>> >> >> >
>>> /Library/Python/2.6/site-packages/StarCluster-0.91-py2.6.egg/starcluster/cli.pyc
>>> >>>> >> >> > in <module>()
>>> >>>> >> >> > ----> 1
>>> >>>> >> >> > 2
>>> >>>> >> >> > 3
>>> >>>> >> >> > 4
>>> >>>> >> >> > 5
>>> >>>> >> >> >
>>> >>>> >> >> >
>>> >>>> >> >> >
>>> >>>> >> >> >
>>> /Library/Python/2.6/site-packages/StarCluster-0.91-py2.6.egg/starcluster/cluster.pyc
>>> >>>> >> >> > in run_plugin(plugin_name, cluster_tag, cfg)
>>> >>>> >> >> > 170 def run_plugin(plugin_name, cluster_tag, cfg):
>>> >>>> >> >> > 171 ec2 = cfg.get_easy_ec2()
>>> >>>> >> >> > --> 172 cl = get_cluster(cluster_tag)
>>> >>>> >> >> > 173 cl.load_receipt()
>>> >>>> >> >> > 174 plug = cfg.get_plugin(plugin_name)
>>> >>>> >> >> > TypeError: get_cluster() takes exactly 2 arguments (1 given)
>>> >>>> >> >> >
>>> >>>> >> >> > It's hard for me to understand what's happening here. I of
>>> >>>> >> >> > course am
>>> >>>> >> >> > not
>>> >>>> >> >> > calling get_cluster directly. Is there a bug in run_plugin()
>>> or
>>> >>>> >> >> > is
>>> >>>> >> >> > there
>>> >>>> >> >> > something wrong in my setup, or both?
>>> >>>> >> >> > Thanks
>>> >>>> >> >> > Dan
>>> >>>> >> >> >
>>> >>>> >> >> > _______________________________________________
>>> >>>> >> >> > StarCluster mailing list
>>> >>>> >> >> > StarCluster_at_mit.edu
>>> >>>> >> >> > http://mailman.mit.edu/mailman/listinfo/starcluster
>>> >>>> >> >> >
>>> >>>> >> >> >
>>> >>>> >> >>
>>> >>>> >> >
>>> >>>> >> >
>>> >>>> >>
>>> >>>> >
>>> >>>> >
>>> >>>>
>>> >>>
>>> >>
>>> >
>>> >
>>>
>>>
>>
>
Received on Sat Oct 30 2010 - 17:44:23 EDT