Re: [Starcluster] NumPy and SciPy tests fail on Ubuntu 9.10 Karmic

From: Damian Eads <no email>
Date: Sat, 17 Apr 2010 18:07:14 -0400

I meant to include the list in my reply. I fixed my problem. The
ptcblas error when running the NumPy and SciPy regression suites was a
result of installing liblapack-dev, an optional dependency of one of
my computer vision package. By not installing it this time, I no
longer get the same error.

Cheers,

Damian

On Sat, Apr 17, 2010 at 4:24 PM, Damian Eads <eads_at_soe.ucsc.edu> wrote:
> Hi Justin,
>
> As we discussed last night, I may have built my AMI off of yours
> incorrectly by starting a cluster and building with
> ec2-bundle-instance without excluding NFS-mounted directories
> (whoops). I decided to rebuild my AMI off of yours again and the first
> thing I did was run the NumPy and SciPy regression tests. This time,
> I've launched your AMI using the EC2 web interface. For
> reproducibility sake, the architecture I'm trying is c1.large.
>
> $ nosetests numpy
> ...
> ----------------------------------------------------------------------
> Ran 2036 tests in 3.687s
>
> FAILED (SKIP=11, errors=7)
>
> $ nosetests scipy
> ----------------------------------------------------------------------
> Ran 4117 tests in 385.762s
>
> FAILED (SKIP=17, errors=10)
>
> Seven and ten failures respectively for NumPy and SciPy. Not bad. As
> for the ptcblas error, it may have been a result of one of my computer
> vision packages needing liblapack-dev. This may explain the complete
> failure of the regressions as I explained in my earlier e-mail.
>
> Damian
>
>
> On Fri, Apr 16, 2010 at 11:11 PM, Justin Riley <jtriley_at_mit.edu> wrote:
>> Hi Damian,
>>
>> I just launched the 32bit version on m1.small and ran the scipy/numpy tests in
>> ipython:
>>
>> ~> numpy.test()
>> ......
>> .......
>> OK (KNOWNFAIL=1, SKIP=11)
>> Out[3]: <nose.result.TextTestResult run=2030 errors=0 failures=0>
>>
>>
>> ~> scipy.test()
>> ...
>> ....
>> OK (KNOWNFAIL=4, SKIP=17)
>> Out[5]: <nose.result.TextTestResult run=3490 errors=0 failures=0>
>>
>> However when I run using "nosetests numpy scipy" like you I get a summary
>> saying things failed. However, I don't see any issues with ptcblas and the
>> likes so I'm not sure what's up.
>>
>> I've always used numpy/scipy.test() and numpy/scipy.show_config() to validate
>> the installation (eg proper detection of atlas and company) and I get a test
>> pass when using them as you see above. So now I'm confused at which is the
>> preferred method and why they produce different results.
>>
>> Regardless, the numpy/scipy that comes with ubuntu did not pass the tests that
>> pass above in ipython using our custom compiled atlas/blas/lapack/numpy/scipy
>> on the AMI. I compiled atlas on the largest instance in order to optimize for
>> the c1.xlarge given that the atlas support from ubuntu is not optimized. Also,
>> I may be incorrect here, but I seem to recall that ubuntu's atlas was also not
>> threaded (ie missing the libpt*.so's).
>>
>> BTW, here are the (pseudo) steps I used to create the AMI which includes the
>> numpy/scipy/lapack/atlas setup:
>>
>> http://starcluster.scripts.mit.edu/~starcluster/wiki/index.php?title=StarCluster_AMI_Cookbook
>>
>> I'll try the numpy/scipy.test()'s with the 64bit ami and c1.xlarge instance.
>>
>> ~Justin
>>
>> On Friday 16 April 2010 7:13:08 pm Damian Eads wrote:
>>> Hi Justin,
>>>
>>> I decided to run my usual computer vision regressions but before I got
>>> there, I ran the NumPy and SciPy regression suite.
>>>
>>> root_at_domU-12-31-38-01-58-B1:/data# nosetests numpy scipy
>>> EE
>>> ======================================================================
>>> ERROR: Failure: ImportError (/usr/lib/libptcblas.so: undefined symbol:
>>> ATL_cpttrsm)
>>> ----------------------------------------------------------------------
>>> Traceback (most recent call last):
>>>   File "/usr/local/lib/python2.6/dist-packages/nose/loader.py", line
>>> 368, in loadTestsFromName
>>>     module = resolve_name(addr.module)
>>>   File "/usr/local/lib/python2.6/dist-packages/nose/util.py", line
>>> 334, in resolve_name
>>>     module = __import__('.'.join(parts_copy))
>>>   File "/usr/local/lib/python2.6/dist-packages/numpy/__init__.py",
>>> line 130, in <module>
>>>     import add_newdocs
>>>   File "/usr/local/lib/python2.6/dist-packages/numpy/add_newdocs.py",
>>> line 9, in <module>
>>>     from lib import add_newdoc
>>>   File "/usr/local/lib/python2.6/dist-packages/numpy/lib/__init__.py",
>>> line 13, in <module>
>>>     from polynomial import *
>>>   File "/usr/local/lib/python2.6/dist-packages/numpy/lib/polynomial.py",
>>> line 18, in <module>
>>>     from numpy.linalg import eigvals, lstsq
>>>   File "/usr/local/lib/python2.6/dist-packages/numpy/linalg/__init__.py",
>>> line 47, in <module>
>>>     from linalg import *
>>>   File "/usr/local/lib/python2.6/dist-packages/numpy/linalg/linalg.py",
>>> line 22, in <module>
>>>     from numpy.linalg import lapack_lite
>>> ImportError: /usr/lib/libptcblas.so: undefined symbol: ATL_cpttrsm
>>>
>>> ======================================================================
>>> ERROR: Failure: AttributeError ('module' object has no attribute 'core')
>>> ----------------------------------------------------------------------
>>> Traceback (most recent call last):
>>>   File "/usr/local/lib/python2.6/dist-packages/nose/loader.py", line
>>> 368, in loadTestsFromName
>>>     module = resolve_name(addr.module)
>>>   File "/usr/local/lib/python2.6/dist-packages/nose/util.py", line
>>> 334, in resolve_name
>>>     module = __import__('.'.join(parts_copy))
>>>   File "/usr/local/lib/python2.6/dist-packages/scipy/__init__.py",
>>> line 11, in <module>
>>>     from numpy import show_config as show_numpy_config
>>>   File "/usr/local/lib/python2.6/dist-packages/numpy/__init__.py",
>>> line 130, in <module>
>>>     import add_newdocs
>>>   File "/usr/local/lib/python2.6/dist-packages/numpy/add_newdocs.py",
>>> line 9, in <module>
>>>     from lib import add_newdoc
>>>   File "/usr/local/lib/python2.6/dist-packages/numpy/lib/__init__.py",
>>> line 13, in <module>
>>>     from polynomial import *
>>>   File "/usr/local/lib/python2.6/dist-packages/numpy/lib/polynomial.py",
>>> line 11, in <module>
>>>     import numpy.core.numeric as NX
>>> AttributeError: 'module' object has no attribute 'core'
>>>
>>> ----------------------------------------------------------------------
>>> Ran 2 tests in 0.001s
>>>
>>> FAILED (errors=2)
>>> root_at_domU-12-31-38-01-58-B1:/data#
>>>
>>> Is there a reason for not using the Ubuntu packages for numpy and scipy?
>>>
>>> Just curious.
>>>
>>> Thanks,
>>>
>>> Damian
>>>
>>>
>>>
>>> -----------------------------------------------------
>>> Damian Eads                           Ph.D. Candidate
>>> University of California             Computer Science
>>> 1156 High Street         Machine Learning Lab, E2-489
>>> Santa Cruz, CA 95064    http://www.soe.ucsc.edu/~eads
>>> _______________________________________________
>>> Starcluster mailing list
>>> Starcluster_at_mit.edu
>>> http://mailman.mit.edu/mailman/listinfo/starcluster
Received on Sat Apr 17 2010 - 18:07:16 EDT