conda-forge build test failing on multiprocessing.cpu_count() with Python 2.7

alex.pitchford's Avatar

alex.pitchford

16 Mar, 2017 04:28 PM

Our library test includes a call to

import scipy.misc
which in turn makes a call to
multiprocessing.cpu_count()
which is currently throwing an exception for the win py27 combination.

See
https://github.com/conda-forge/qutip-feedstock/pull/18
for more information

  1. Support Staff 1 Posted by Feodor Fitsner on 16 Mar, 2017 05:43 PM

    Feodor Fitsner's Avatar

    I've only found mentioning of multiprocessing.cpu_count() and it worked fine in AppVeyor environment: https://ci.appveyor.com/project/FeodorFitsner/python-multi-proc/bui...

    This SO topic also suggests multiprocessing.cpu_count(): http://stackoverflow.com/questions/1006289/how-to-find-out-the-numb...

    multiprocessing.num_cpus() didn't worked as the function unknown: https://ci.appveyor.com/project/FeodorFitsner/python-multi-proc/bui...

  2. 2 Posted by alex.pitchford on 17 Mar, 2017 10:39 AM

    alex.pitchford's Avatar

    Thanks for testing. Sorry, my mistake. I should have written `cpu_count` in
    the message.
    You can see that it is this function that is (was) raising the exception
    from within scipy.misc in the link I included.
    Our build was passing one day, then failed the next. So perhaps it is worth
    trying again today.

  3. 3 Posted by alex.pitchford on 17 Mar, 2017 02:28 PM

    alex.pitchford's Avatar

    I tried again today and still find that the build test fails for Python
    2.7.

    The stack trace is as follows:

    TEST START: C:\bld\win-64\qutip-4.1.0-np112py27_1.tar.bz2
    Deleting work directory, C:\bld\qutip_1489758249158\work\qutip-4.1.0
    The following packages will be downloaded:
        package                    |            build
        ---------------------------|-----------------
        qutip-4.1.0                |      np112py27_1         2.2 MB  local
    The following NEW packages will be INSTALLED:
        certifi:        2017.1.23-py27_0   conda-forge
        cython:         0.25.2-py27_1      conda-forge
        mingwpy:        0.1.0-py27_2       conda-forge
        mkl:            2017.0.1-0         defaults
        numpy:          1.12.0-py27_0      defaults
        python:         2.7.12-2           conda-forge
        qutip:          4.1.0-np112py27_1  local
        scipy:          0.19.0-np112py27_0 defaults
        setuptools:     33.1.0-py27_0      conda-forge
        vs2008_runtime: 9.00.30729.5054-0  defaults
        wincertstore:   0.2-py27_0         conda-forge
    WARNING: conda-build appears to be out of date. You have version 2.1.6 but
    the
    latest version is 2.1.7. Run
    conda update -n root conda-build
    to get the latest version.
    C:\bld\qutip_1489758249158\test_tmp>call
    "C:\Miniconda-x64\Scripts\activate.bat" "C:\bld\qutip_1489758249158_t_env"
     1>nul 2>&1
    (C:\bld\qutip_1489758249158_t_env) C:\bld\qutip_1489758249158\test_tmp>if
    errorlevel 1 exit 1
    (C:\bld\qutip_1489758249158_t_env)
    C:\bld\qutip_1489758249158\test_tmp>"C:\bld\qutip_1489758249158_t_env\python.exe"
    -s "C:\bld\qutip_1489758249158\test_tmp\run_test.py"
    import: u'qutip'
    C:\bld\qutip_1489758249158_t_env\lib\site-packages\qutip_init.py:161:
    UserWarning: matplotlib not found: Graphics will not work.
      warnings.warn("matplotlib not found: Graphics will not work.")
    Traceback (most recent call last):
      File "C:\bld\qutip_1489758249158\test_tmp\run_test.py", line 2, in
    <module>
        import qutip
      File
    "C:\bld\qutip_1489758249158_t_env\lib\site-packages\qutip__init.py",
    line 183, in <module>
        from qutip.visualization import *
      File
    "C:\bld\qutip_1489758249158_t_env\lib\site-packages\qutip\visualization.py",
    line 62, in <module>
        from qutip.wigner import wigner
      File
    "C:\bld\qutip_1489758249158_t_env\lib\site-packages\qutip\wigner.py", line
    51, in <module>
        from scipy.misc import factorial
      File
    "C:\bld\qutip_1489758249158_t_env\lib\site-packages\scipy\misc__init.py",
    line 53, in <module>
        from scipy.interpolate._pade import pade
      File
    "C:\bld\qutip_1489758249158_t_env\lib\site-packages\scipy\interpolate__init.py",
    line 176, in <module>
        from .interpolate import *
      File
    "C:\bld\qutip_1489758249158_t_env\lib\site-packages\scipy\interpolate\interpolate.py",
    line 32, in <module>
        from .interpnd import _ndim_coords_from_arrays
      File "scipy/interpolate/interpnd.pyx", line 1, in init
    scipy.interpolate.interpnd (scipy\interpolate\interpnd.c:25150)
      File
    "C:\bld\qutip_1489758249158_t_env\lib\site-packages\scipy\spatial__init.py",
    line 94, in <module>
        from .ckdtree import *
      File "scipy/spatial/ckdtree.pyx", line 29, in init scipy.spatial.ckdtree
    (scipy\spatial\ckdtree.cxx:19224)
      File "C:\bld\qutip_1489758249158_t_env\lib\multiprocessing__init_.py",
    line 136, in cpu_count
        raise NotImplementedError('cannot determine number of cpus')
    NotImplementedError: cannot determine number of cpus
    (C:\bld\qutip_1489758249158_t_env) C:\bld\qutip_1489758249158\test_tmp>if
    errorlevel 1 exit 1
    TESTS FAILED: qutip-4.1.0-np112py27_1
    Command exited with code 1
    

    The multiprocessing.cpu_count() function looks like this in Python 2.7

    def cpu_count():
        '''
        Returns the number of CPUs in the system
        '''
        if sys.platform == 'win32':
            try:
                num = int(os.environ['NUMBER_OF_PROCESSORS'])
            except (ValueError, KeyError):
                num = 0
        elif 'bsd' in sys.platform or sys.platform == 'darwin':
            comm = '/sbin/sysctl -n hw.ncpu'
            if sys.platform == 'darwin':
                comm = '/usr' + comm
            try:
                with os.popen(comm) as p:
                    num = int(p.read())
            except ValueError:
                num = 0
        else:
            try:
                num = os.sysconf('SC_NPROCESSORS_ONLN')
            except (ValueError, OSError, AttributeError):
                num = 0
    
        if num >= 1:
            return num
        else:
            raise NotImplementedError('cannot determine number of cpus')
    

    Is it possible that the NUMBER_OF_PROCESSORS env var is not set?

    The build is successful for Python 3.6. The multiprocessing.cpu_count()
    function looks like this in Python 3.6

        def cpu_count(self):
            '''Returns the number of CPUs in the system'''
            num = os.cpu_count()
            if num is None:
                raise NotImplementedError('cannot determine number of cpus')
            else:
                return num
    
  4. Support Staff 4 Posted by Feodor Fitsner on 17 Mar, 2017 02:35 PM

    Feodor Fitsner's Avatar

    Can you drop a link to that build please?

    -Feodor Fitsner, AppVeyor

    ________________________________

  5. 5 Posted by alex.pitchford on 17 Mar, 2017 02:55 PM

    alex.pitchford's Avatar
  6. Support Staff 6 Posted by Feodor Fitsner on 17 Mar, 2017 06:51 PM

    Feodor Fitsner's Avatar

    It worked with both 32- and 64-bit Miniconda as well: https://ci.appveyor.com/project/FeodorFitsner/python-multi-proc/bui...

    If it takes the value from environment variable could it be that something during the build breaks that variable?..

  7. 7 Posted by alex.pitchford on 19 Mar, 2017 09:45 PM

    alex.pitchford's Avatar

    It turned out to be some issue with conda-build. Sorry to trouble you. This
    can be closed now.

Reply to this discussion

Internal reply

Formatting help / Preview (switch to plain text) No formatting (switch to Markdown)

Attaching KB article:

»

Attached Files

You can attach files up to 10MB

If you don't have an account yet, we need to confirm you're human and not a machine trying to post spam.

Keyboard shortcuts

Generic

? Show this help
ESC Blurs the current field

Comment Form

r Focus the comment reply box
^ + ↩ Submit the comment

You can use Command ⌘ instead of Control ^ on Mac

Recent Discussions

18 Oct, 2017 08:54 AM
18 Oct, 2017 08:21 AM
18 Oct, 2017 07:42 AM
17 Oct, 2017 07:36 PM
17 Oct, 2017 05:39 PM

 

17 Oct, 2017 04:29 PM
17 Oct, 2017 07:49 AM
16 Oct, 2017 11:38 PM
16 Oct, 2017 10:01 PM
16 Oct, 2017 08:25 AM
16 Oct, 2017 07:58 AM