Re: plugin shell help

From: Dan Tenenbaum <no email>
Date: Sat, 30 Oct 2010 11:57:27 -0700

Looks like there are a number of files in this fork that don't exist in the
0.91 that I downloaded.

So just downloaded your whole fork and built that, then got this in the
shell:

In [1]: cluster.run_plugin("shortreadplugin","myfirstcluster",cfg)
---------------------------------------------------------------------------
AttributeError Traceback (most recent call last)

/Library/Python/2.6/site-packages/StarCluster-0.9999-py2.6.egg/starcluster/commands/shell.pyc
in <module>()
----> 1
      2
      3
      4
      5

AttributeError: 'module' object has no attribute 'run_plugin'

In [2]:

Dan


On Sat, Oct 30, 2010 at 11:45 AM, Dan Tenenbaum <dtenenba_at_fhcrc.org> wrote:

> Keeping this on the list.
> BTW, the new cluster.py has no syntax errors, so something else must be
> happening.
> Dan
>
>
> On Sat, Oct 30, 2010 at 11:44 AM, Dan Tenenbaum <dtenenba_at_fhcrc.org>wrote:
>
>> Thanks. That gave me a different error, below.
>> The first time this happened, I did:
>> sudo easy_install iptools, but it is still happening.
>>
>> Dan
>>
>> $ starcluster shell
>> Leopard libedit detected.
>> ERROR: An unexpected error occurred while tokenizing input
>> The following traceback may be corrupted or invalid
>> The error message is: ('EOF in multi-line statement', (405, 0))
>>
>>
>> ---------------------------------------------------------------------------
>> ImportError Traceback (most recent call
>> last)
>>
>> /Library/Python/2.6/site-packages/StarCluster-0.91-py2.6.egg/EGG-INFO/scripts/starcluster
>> in <module>()
>> 3 __requires__ = 'StarCluster==0.91'
>> 4 import pkg_resources
>> ----> 5 pkg_resources.run_script('StarCluster==0.91', 'starcluster')
>> 6
>> 7
>>
>> /System/Library/Frameworks/Python.framework/Versions/2.6/Extras/lib/python/pkg_resources.pyc
>> in run_script(self, requires, script_name)
>> 440 ns.clear()
>> 441 ns['__name__'] = name
>> --> 442 self.require(requires)[0].run_script(script_name, ns)
>> 443
>> 444
>>
>> /System/Library/Frameworks/Python.framework/Versions/2.6/Extras/lib/python/pkg_resources.pyc
>> in run_script(self, script_name, namespace)
>> 1165 )
>> 1166 script_code =
>> compile(script_text,script_filename,'exec')
>> -> 1167 exec script_code in namespace, namespace
>> 1168
>> 1169 def _has(self, path):
>>
>> /Library/Python/2.6/site-packages/StarCluster-0.91-py2.6.egg/EGG-INFO/scripts/starcluster
>> in <module>()
>> 1
>> ----> 2 #!/usr/bin/python
>>
>> 3 # EASY-INSTALL-SCRIPT: 'StarCluster==0.91','starcluster'
>>
>> 4 __requires__ = 'StarCluster==0.91'
>> 5 import pkg_resources
>> 6 pkg_resources.run_script('StarCluster==0.91', 'starcluster')
>>
>> /Users/dante/.starcluster/build/bdist.macosx-10.6-universal/egg/starcluster/cli.py
>> in <module>()
>> 36
>> 37
>> ---> 38
>> 39
>> 40
>>
>> /Users/dante/.starcluster/build/bdist.macosx-10.6-universal/egg/starcluster/cluster.py
>> in <module>()
>> 10
>> 11
>> ---> 12
>> 13
>> 14
>>
>> ImportError: cannot import name iptools
>>
>>
>> On Sat, Oct 30, 2010 at 11:38 AM, Marc Resnick <mresnick_at_mit.edu> wrote:
>>
>>> The version here
>>>
>>> http://github.com/jtriley/StarCluster/blob/experimental/starcluster/cluster.py
>>> may have those bugs fixed, as I remember that one as well.
>>>
>>>
>>> Marc
>>>
>>> On Sat, Oct 30, 2010 at 2:31 PM, Dan Tenenbaum <dtenenba_at_fhcrc.org>
>>> wrote:
>>> > I already did that. ;)
>>> > Now I'm getting another error:
>>> > In [1]: cluster.run_plugin("shortreadplugin","myfirstcluster",cfg)
>>> >>>> Using private key /Users/dante/.ec2/bioc-keypair.pem.rsa (rsa)
>>> >
>>> ---------------------------------------------------------------------------
>>> > AttributeError Traceback (most recent call
>>> last)
>>> >
>>> /Library/Python/2.6/site-packages/StarCluster-0.91-py2.6.egg/starcluster/cli.pyc
>>> > in <module>()
>>> > ----> 1
>>> > 2
>>> > 3
>>> > 4
>>> > 5
>>> >
>>> /Library/Python/2.6/site-packages/StarCluster-0.91-py2.6.egg/starcluster/cluster.pyc
>>> > in run_plugin(plugin_name, cluster_tag, cfg)
>>> > 175 plugins = {}
>>> > 176 plugins[plugin_name] = plug
>>> > --> 177 plugins = cl.load_plugins(plugins)
>>> > 178 master = cl.master_node
>>> > 179 for p in plugins:
>>> >
>>> /Library/Python/2.6/site-packages/StarCluster-0.91-py2.6.egg/starcluster/cluster.pyc
>>> > in load_plugins(self, plugins)
>>> > 314 plugs = []
>>> > 315 for plugin in plugins:
>>> > --> 316 setup_class = plugin.get('setup_class')
>>> > 317 plugin_name = plugin.get('__name__')
>>> > 318 mod_name = '.'.join(setup_class.split('.')[:-1])
>>> > AttributeError: 'str' object has no attribute 'get'
>>> > Hmm....
>>> > Dan
>>> >
>>> > On Sat, Oct 30, 2010 at 11:27 AM, Marc Resnick <mresnick_at_mit.edu>
>>> wrote:
>>> >>
>>> >> Yes, that's the bug I told Justin about this Summer... I think I
>>> >> patched it, but he may have forgotten to pull from my fork.
>>> >>
>>> >> You can fix it by editing the run_plugin method, and replacing
>>> >>
>>> >> cl = get_cluster(cluster_tag)
>>> >>
>>> >> with
>>> >>
>>> >> cl = get_cluster(cluster_tag, cfg)
>>> >>
>>> >>
>>> >>
>>> >> Marc
>>> >>
>>> >>
>>> >> On Sat, Oct 30, 2010 at 2:20 PM, Dan Tenenbaum <dtenenba_at_fhcrc.org>
>>> wrote:
>>> >> > Hi Marc,
>>> >> > I am running 0.91.
>>> >> > I was doing a couple things wrong. I figured them out but I am still
>>> >> > having
>>> >> > this problem.
>>> >> > One thing I did wrong was have the
>>> >> > plugins=shortreadplugin
>>> >> > line under the global section; it needed to be under the
>>> >> > [cluster smallcluster]
>>> >> > section.
>>> >> > When I start the cluster now, I can see the output from my plugin:
>>> >> >>>> Running plugin shortreadplugin
>>> >> >>>> about to echo
>>> >> > But, when I run the shell and put in this command:
>>> >> > cluster.run_plugin("shortreadplugin","myfirstcluster",cfg)
>>> >> > I get the same error described below.
>>> >> > Dan
>>> >> >
>>> >> > On Sat, Oct 30, 2010 at 11:11 AM, Marc Resnick <mresnick_at_mit.edu>
>>> wrote:
>>> >> >>
>>> >> >> Hey Dan,
>>> >> >>
>>> >> >> Are you running the latest version? I saw that bug a few months
>>> ago, I
>>> >> >> thought I remember Justin fixing it.
>>> >> >>
>>> >> >>
>>> >> >> Marc
>>> >> >>
>>> >> >> On Sat, Oct 30, 2010 at 2:00 PM, Dan Tenenbaum <dtenenba_at_fhcrc.org
>>> >
>>> >> >> wrote:
>>> >> >> > Hi,
>>> >> >> > I am not much of a python programmer and I'm trying to figure out
>>> the
>>> >> >> > plugin
>>> >> >> > system.
>>> >> >> > I created a ~/.starcluster/plugins directory and put a file in it
>>> >> >> > called
>>> >> >> > shortread.py.
>>> >> >> > Here are its contents:
>>> >> >> >
>>> >> >> > from starcluster.clustersetup import ClusterSetup
>>> >> >> > from starcluster.logger import log
>>> >> >> > class ShortReadPlugin(ClusterSetup):
>>> >> >> > def __init__(self, arg1):
>>> >> >> > self.arg1 = arg1
>>> >> >> > log.info('arg1 = %s' % arg1)
>>> >> >> > def run(self, nodes, master, user, user_shell, volumes):
>>> >> >> > log.info("about to echo")
>>> >> >> > master.ssh.execute("echo 'hey'")
>>> >> >> > As you can see, it's just a test.
>>> >> >> > In my config, I put this at the bottom:
>>> >> >> > [plugin shortreadplugin]
>>> >> >> > SETUP_CLASS = ShortReadPlugin
>>> >> >> > arg1=foo
>>> >> >> > And above in the [global] section I put:
>>> >> >> > plugins=shortreadplugin
>>> >> >> > I'm a bit confused about whether this is right because in your
>>> >> >> > example:
>>> >> >> > http://web.mit.edu/stardev/cluster/docs/plugins.html
>>> >> >> > ...you have the SETUP_CLASS as ubuntu.PackageInstaller. I'm not
>>> sure
>>> >> >> > where
>>> >> >> > the "ubuntu" fits into this. Is it like java where I need to have
>>> a
>>> >> >> > folder
>>> >> >> > called ubuntu where the class resides?
>>> >> >> > Anyway, I fired up my cluster and went to test this with
>>> "starcluster
>>> >> >> > shell"
>>> >> >> > and got the following output:
>>> >> >> > In [5]: cluster.run_plugin('shortreadplugin', 'myfirstcluster',
>>> cfg)
>>> >> >> >
>>> >> >> >
>>> >> >> >
>>> ---------------------------------------------------------------------------
>>> >> >> > TypeError Traceback (most recent
>>> call
>>> >> >> > last)
>>> >> >> >
>>> >> >> >
>>> >> >> >
>>> /Library/Python/2.6/site-packages/StarCluster-0.91-py2.6.egg/starcluster/cli.pyc
>>> >> >> > in <module>()
>>> >> >> > ----> 1
>>> >> >> > 2
>>> >> >> > 3
>>> >> >> > 4
>>> >> >> > 5
>>> >> >> >
>>> >> >> >
>>> >> >> >
>>> /Library/Python/2.6/site-packages/StarCluster-0.91-py2.6.egg/starcluster/cluster.pyc
>>> >> >> > in run_plugin(plugin_name, cluster_tag, cfg)
>>> >> >> > 170 def run_plugin(plugin_name, cluster_tag, cfg):
>>> >> >> > 171 ec2 = cfg.get_easy_ec2()
>>> >> >> > --> 172 cl = get_cluster(cluster_tag)
>>> >> >> > 173 cl.load_receipt()
>>> >> >> > 174 plug = cfg.get_plugin(plugin_name)
>>> >> >> > TypeError: get_cluster() takes exactly 2 arguments (1 given)
>>> >> >> >
>>> >> >> > It's hard for me to understand what's happening here. I of course
>>> am
>>> >> >> > not
>>> >> >> > calling get_cluster directly. Is there a bug in run_plugin() or
>>> is
>>> >> >> > there
>>> >> >> > something wrong in my setup, or both?
>>> >> >> > Thanks
>>> >> >> > Dan
>>> >> >> >
>>> >> >> > _______________________________________________
>>> >> >> > StarCluster mailing list
>>> >> >> > StarCluster_at_mit.edu
>>> >> >> > http://mailman.mit.edu/mailman/listinfo/starcluster
>>> >> >> >
>>> >> >> >
>>> >> >>
>>> >> >
>>> >> >
>>> >>
>>> >
>>> >
>>>
>>>
>>
>
Received on Sat Oct 30 2010 - 14:57:31 EDT