NUnit + ApprovalTests tests started failing

russkie's Avatar

russkie

03 Aug, 2019 02:26 PM

It looks some sort of an environmental issue - a commit that built successfully few days ago (https://ci.appveyor.com/project/gitextensions/gitextensions/builds/...) now fails to build (https://ci.appveyor.com/project/gitextensions/gitextensions/builds/...)

Other commits also started failing on the build server (https://ci.appveyor.com/project/gitextensions/gitextensions/history) yet all of them are passing locally.

Help is appreciated.
Thanks, Igor

  1. 1 Posted by russkie on 03 Aug, 2019 08:37 PM

    russkie's Avatar

    I have added attributes as suggested in the error message and my builds got fixed.
    Then they started randomly failing with some other tests failing....

    The same commit:

    Note, the failed builds fail in different tests.
    The successful build would have failed with a time out - I logged in the box and dismissed a popup - it was breaking in some other random place...

    It is definitely something to do with the environments, and it started in the past few days.

  2. Support Staff 2 Posted by Ilya Finkelshte... on 03 Aug, 2019 11:42 PM

    Ilya Finkelshteyn's Avatar

    This can be indeed related to the environment change, specifically August 1st images update. Please try to set image to the Previous Visual Studio 2017 and see if this helps.

    But please come back to us in any case. Even if switching to "previous" helps, it can be treated only as a temporary workaround, valid only till next update. So we will be needed to root cause it together.

  3. 3 Posted by russkie on 23 Aug, 2019 08:15 PM

    russkie's Avatar

    It appears I have never received a notification about your reply.
    I will try to test it in coming days (I'm flat out busy, but I'll try to carve some time) and will come back.

    We've had tests that previously were stable now failing at random. Re-running builds fixes them back...

  4. 4 Posted by russkie on 03 Sep, 2019 12:51 PM

    russkie's Avatar

    I have done some more testing and I tend to believe there may be a regression in the current VS 2017 image.

    One thing definitely not working correctly is opencover.
    VS 2017 - fails to submit test results:
    https://ci.appveyor.com/project/gitextensions/gitextensions/builds/...

    Previous VS 2017 - all good:
    https://ci.appveyor.com/project/gitextensions/gitextensions/builds/...

    The only difference between the two commits is the targeted image.

  5. 5 Posted by russkie on 03 Sep, 2019 12:55 PM

    russkie's Avatar

    The current VS 2017 image appears very unstable.

    It is a total gamble whether the same commit would fail or build sucessfully:
    - good https://ci.appveyor.com/project/gitextensions/gitextensions/builds/... - failed https://ci.appveyor.com/project/gitextensions/gitextensions/builds/...

    In the failed build one of the C++ projects builds just silently failed (https://ci.appveyor.com/project/gitextensions/gitextensions/builds/...) whereas in the subsequent build it built successfully (https://ci.appveyor.com/project/gitextensions/gitextensions/builds/...)

  6. 6 Posted by russkie on 03 Sep, 2019 01:04 PM

    russkie's Avatar

    We have also tried to build against VS 2019 image.
    A number of tests started failing out of the blue - https://ci.appveyor.com/project/gitextensions/gitextensions/builds/..., opencover also didn't work

  7. 7 Posted by russkie on 03 Sep, 2019 01:22 PM

    russkie's Avatar

    To fix opencover issue it looks like it has to be run with -register:administrator as opposed to -register:user that we historically had.
    https://ci.appveyor.com/project/gitextensions/gitextensions/builds/...

    I'm not sure what changed, but I can speculate it may be some sort of permissions issue.

  8. Support Staff 8 Posted by Ilya Finkelshte... on 04 Sep, 2019 06:58 PM

    Ilya Finkelshteyn's Avatar

    I took a look at one specific failure you sent, this one. It fails because The system cannot find the file '..\GitExtensionsShellEx\Release\GitExtensionsShellEx32.dll'. However before it builds only 64-bit version of that file: C:\projects\gitextensions\GitExtensionsShellEx\Release-x64\GitExtensionsShellEx64.dll. On the good build it builds both 32- and 64-bit version of GitExtensionsShellEx. Also it seems that "good" build rise a warning warning C4996: 'GetVersion': was declared deprecated while "bad" not. Does it rings any bells?

  9. 9 Posted by russkie on 04 Sep, 2019 07:24 PM

    russkie's Avatar

    Yes, that's precisely what was happening and the expectation that both builds produce near identical output (including warnings).
    Somehow on the bad build the 32-bit build silently fails/terminates.

  10. Support Staff 10 Posted by Ilya Finkelshte... on 05 Sep, 2019 01:56 AM

    Ilya Finkelshteyn's Avatar

    I would add catch here and write the exception. Something like catch {$_} should work. Without catch it might hide real error.

Reply to this discussion

Internal reply

Formatting help / Preview (switch to plain text) No formatting (switch to Markdown)

Attaching KB article:

»

Attached Files

You can attach files up to 10MB

If you don't have an account yet, we need to confirm you're human and not a machine trying to post spam.

Keyboard shortcuts

Generic

? Show this help
ESC Blurs the current field

Comment Form

r Focus the comment reply box
^ + ↩ Submit the comment

You can use Command ⌘ instead of Control ^ on Mac