Memory allocation errors on 32 bit instances

jonathan.underwood's Avatar

jonathan.underwood

06 Dec, 2017 05:16 PM

I am running a lot of tests for a project (python-lz4). The test harness I am using, pytest, runs all of the tests (numbering 800+) in a single python process. The tests are using a extension module which calls the LZ4 library, and the LZ4 library makes a lot of memory allocations and deallocations (outside of the Python managed heap, but within the process heap i.e. using malloc/calloc/free). What I am seeing is that at a certain point the mallocs begin to fail. It looks like this is because the memory is fragmented. This doesn't affect 64 bit builds. I've looked at possibly breaking the tests up to run in multiple (sequential) processes, but so far haven't found a straightforward way to do that. Has anyone else encountered this problem on Appveyor? Any suggestions for a work around?

  1. Support Staff 1 Posted by Ilya Finkelshte... on 06 Dec, 2017 06:42 PM

    Ilya Finkelshteyn's Avatar

    Maybe you could look at parallel testing. It is actually different builds (on different VMs), very similar to the build matrix. For free accounts it will run sequentially, but you will get full isolation between groups of tests.

  2. 2 Posted by jonathan.underw... on 06 Dec, 2017 08:20 PM

    jonathan.underwood's Avatar

    Thanks Ilya, I'll have a look at those.

    In the meantime, is there an easy way to specify that the test_script
    section should only be ran for some members of the build matrix?

  3. Support Staff 3 Posted by Ilya Finkelshte... on 06 Dec, 2017 09:03 PM

    Ilya Finkelshteyn's Avatar

    I think I just answered that here but you can play with environment variables any way you want. For example you can check for specific values of that variable, and do it inside `build.cmd, it is up to you.

Reply to this discussion

Internal reply

Formatting help / Preview (switch to plain text) No formatting (switch to Markdown)

Attaching KB article:

»

Attached Files

You can attach files up to 10MB

If you don't have an account yet, we need to confirm you're human and not a machine trying to post spam.

Keyboard shortcuts

Generic

? Show this help
ESC Blurs the current field

Comment Form

r Focus the comment reply box
^ + ↩ Submit the comment

You can use Command ⌘ instead of Control ^ on Mac