Unit tests failing only in AppVeyor

Eric J. Smith's Avatar

Eric J. Smith

21 Feb, 2015 10:04 PM

I have some unit tests that are using redis that work great locally, but fail on AppVeyor. I have tried everything to try and reproduce this issue locally. Trying to run the tests in the exact same way by using the same command line parameters that you guys are using, but I am having no luck reproducing the issue. Do you have any ideas? Any tips for tracking this issue down?

  1. 1 Posted by Eric J. Smith on 21 Feb, 2015 10:08 PM

    Eric J. Smith's Avatar

    Also, these are Xunit tests and I saw that you are passing the /appveyor command line parameter to xunit, but when I try to do that locally it says that it's not a valid parameter. Are you guys using a custom version?

  2. Support Staff 2 Posted by Feodor Fitsner on 21 Feb, 2015 10:27 PM

    Feodor Fitsner's Avatar

    Hi Eric,

    Yep, we use modified xunit 1.9.2 runners with built-in logging. To see if xunit is causing the issue try running your tests with standard xunit runners. They can be downloaded from nuget.org and pushed to PATH:

    nuget install xunit.runners -excludeversion
    set PATH=%APPVEYOR_BUILD_FOLDER%\xunit.runners\tools;%PATH%
  3. 3 Posted by Eric J. Smith on 21 Feb, 2015 10:57 PM

    Eric J. Smith's Avatar

    I changed my appveyor.yml file to run my tests manually with the normal xunit runner and they still fail in appveyor, but not locally. This is driving me nuts since I can't reproduce it locally and I have no way to try and debug what is going on in the build server.

  4. Support Staff 4 Posted by Feodor Fitsner on 21 Feb, 2015 11:00 PM

    Feodor Fitsner's Avatar

    What error are you getting? How Redis is deployed?

    Btw, are you running these tests on Pro environment?

  5. Support Staff 5 Posted by Feodor Fitsner on 21 Feb, 2015 11:34 PM

    Feodor Fitsner's Avatar

    Another idea - try doing a simple standalone console app with just that test and run it on AppVeyor.

  6. 6 Posted by Eric J. Smith on 22 Feb, 2015 02:30 AM

    Eric J. Smith's Avatar

    I found the powershell script to let me connect to the console and that gave me a couple hints. I don't know why it doesn't happen locally, but I think some background threads that are being spawned in the classes that I am testing are not being released quickly enough and that is causing the appdomain to stay alive for a little bit and then I end up with multiple tests running concurrently and stealing each others data which is causing my asserts to fail.

  7. Support Staff 7 Posted by Feodor Fitsner on 22 Feb, 2015 03:28 AM

    Feodor Fitsner's Avatar

    Cool, thanks for the update!

  8. 8 Posted by Eric J. Smith on 22 Feb, 2015 04:07 AM

    Eric J. Smith's Avatar

    I thought I had it narrowed down, but I have created all kinds of safe guards and logging and still the tests fail in appveyor but not locally. I am sure it's going to be something stupid in the end, but right now I'm getting so dang frustrated.

    It's like the redis pub/sub subscription is randomly not sending the messages out in the appveyor environment.

    I am installing redis via chocolatey in the appveyor.yml file here:

    I am on a trial pro plan right now, but we are definitely going to pay for a subscription soon... assuming I can figure out why these tests don't work.

  9. 9 Posted by Eric J. Smith on 22 Feb, 2015 04:30 AM

    Eric J. Smith's Avatar

    Logged into the box and installed redis manually to see if it was a bad version of redis, but that didn't help either. This is sooo weird.

  10. Support Staff 10 Posted by Feodor Fitsner on 22 Feb, 2015 05:00 AM

    Feodor Fitsner's Avatar

    Have you tried running it on Azure environment? Select, say, "unstable" OS  on Environment tab to force using Azure.

    Btw, are you able to localize this test in a single method? We've been using Redis for Appveyor. I'd like to see if your test works locally.


  11. 11 Posted by Eric J. Smith on 22 Feb, 2015 07:14 AM

    Eric J. Smith's Avatar

    I tried on the unstable OS environment and got the same results. :-(

  12. 12 Posted by Eric J. Smith on 22 Feb, 2015 08:38 AM

    Eric J. Smith's Avatar

    Well, it always boils down to something stupid in the end and this time was no exception. I had a set of redis tests that were not sharing a ConnectionMultiplexer like they should have and this somehow resulted in me randomly not receiving pub/sub messages. Man, that was a fun use of a day! :-)

  13. Support Staff 13 Posted by Feodor Fitsner on 22 Feb, 2015 06:16 PM

    Feodor Fitsner's Avatar

    Nice! :) Glad you managed to fix that.

  14. 14 Posted by Eric J. Smith on 22 Feb, 2015 07:20 PM

    Eric J. Smith's Avatar

    Still no idea why I could run it locally on multiple machines, but it would only fail on the build server. Guessing that maybe it was resource / performance related.

Reply to this discussion

Internal reply

Formatting help / Preview (switch to plain text) No formatting (switch to Markdown)

Attaching KB article:


Attached Files

You can attach files up to 10MB

If you don't have an account yet, we need to confirm you're human and not a machine trying to post spam.

Keyboard shortcuts


? Show this help
ESC Blurs the current field

Comment Form

r Focus the comment reply box
^ + ↩ Submit the comment

You can use Command ⌘ instead of Control ^ on Mac