NUnit + ApprovalTests tests started failing

russkie's Avatar

russkie

03 Aug, 2019 02:26 PM

It looks some sort of an environmental issue - a commit that built successfully few days ago (https://ci.appveyor.com/project/gitextensions/gitextensions/builds/...) now fails to build (https://ci.appveyor.com/project/gitextensions/gitextensions/builds/...)

Other commits also started failing on the build server (https://ci.appveyor.com/project/gitextensions/gitextensions/history) yet all of them are passing locally.

Help is appreciated.
Thanks, Igor

  1. 1 Posted by russkie on 03 Aug, 2019 08:37 PM

    russkie's Avatar

    I have added attributes as suggested in the error message and my builds got fixed.
    Then they started randomly failing with some other tests failing....

    The same commit:

    Note, the failed builds fail in different tests.
    The successful build would have failed with a time out - I logged in the box and dismissed a popup - it was breaking in some other random place...

    It is definitely something to do with the environments, and it started in the past few days.

  2. 2 Posted by Ilya Finkelshte... on 03 Aug, 2019 11:42 PM

    Ilya Finkelshteyn's Avatar

    This can be indeed related to the environment change, specifically August 1st images update. Please try to set image to the Previous Visual Studio 2017 and see if this helps.

    But please come back to us in any case. Even if switching to "previous" helps, it can be treated only as a temporary workaround, valid only till next update. So we will be needed to root cause it together.

  3. 3 Posted by russkie on 23 Aug, 2019 08:15 PM

    russkie's Avatar

    It appears I have never received a notification about your reply.
    I will try to test it in coming days (I'm flat out busy, but I'll try to carve some time) and will come back.

    We've had tests that previously were stable now failing at random. Re-running builds fixes them back...

  4. 4 Posted by russkie on 03 Sep, 2019 12:51 PM

    russkie's Avatar

    I have done some more testing and I tend to believe there may be a regression in the current VS 2017 image.

    One thing definitely not working correctly is opencover.
    VS 2017 - fails to submit test results:
    https://ci.appveyor.com/project/gitextensions/gitextensions/builds/...

    Previous VS 2017 - all good:
    https://ci.appveyor.com/project/gitextensions/gitextensions/builds/...

    The only difference between the two commits is the targeted image.

  5. 5 Posted by russkie on 03 Sep, 2019 12:55 PM

    russkie's Avatar

    The current VS 2017 image appears very unstable.

    It is a total gamble whether the same commit would fail or build sucessfully:
    - good https://ci.appveyor.com/project/gitextensions/gitextensions/builds/... - failed https://ci.appveyor.com/project/gitextensions/gitextensions/builds/...

    In the failed build one of the C++ projects builds just silently failed (https://ci.appveyor.com/project/gitextensions/gitextensions/builds/...) whereas in the subsequent build it built successfully (https://ci.appveyor.com/project/gitextensions/gitextensions/builds/...)

  6. 6 Posted by russkie on 03 Sep, 2019 01:04 PM

    russkie's Avatar

    We have also tried to build against VS 2019 image.
    A number of tests started failing out of the blue - https://ci.appveyor.com/project/gitextensions/gitextensions/builds/..., opencover also didn't work

  7. 7 Posted by russkie on 03 Sep, 2019 01:22 PM

    russkie's Avatar

    To fix opencover issue it looks like it has to be run with -register:administrator as opposed to -register:user that we historically had.
    https://ci.appveyor.com/project/gitextensions/gitextensions/builds/...

    I'm not sure what changed, but I can speculate it may be some sort of permissions issue.

  8. 8 Posted by Ilya Finkelshte... on 04 Sep, 2019 06:58 PM

    Ilya Finkelshteyn's Avatar

    I took a look at one specific failure you sent, this one. It fails because The system cannot find the file '..\GitExtensionsShellEx\Release\GitExtensionsShellEx32.dll'. However before it builds only 64-bit version of that file: C:\projects\gitextensions\GitExtensionsShellEx\Release-x64\GitExtensionsShellEx64.dll. On the good build it builds both 32- and 64-bit version of GitExtensionsShellEx. Also it seems that "good" build rise a warning warning C4996: 'GetVersion': was declared deprecated while "bad" not. Does it rings any bells?

  9. 9 Posted by russkie on 04 Sep, 2019 07:24 PM

    russkie's Avatar

    Yes, that's precisely what was happening and the expectation that both builds produce near identical output (including warnings).
    Somehow on the bad build the 32-bit build silently fails/terminates.

  10. 10 Posted by Ilya Finkelshte... on 05 Sep, 2019 01:56 AM

    Ilya Finkelshteyn's Avatar

    I would add catch here and write the exception. Something like catch {$_} should work. Without catch it might hide real error.

  11. Ilya Finkelshteyn closed this discussion on 05 Nov, 2019 09:01 PM.

Comments are currently closed for this discussion. You can start a new one.

Keyboard shortcuts

Generic

? Show this help
ESC Blurs the current field

Comment Form

r Focus the comment reply box
^ + ↩ Submit the comment

You can use Command ⌘ instead of Control ^ on Mac

Recent Discussions

07 Jul, 2020 06:21 PM
07 Jul, 2020 03:15 PM
05 Jul, 2020 02:33 AM
03 Jul, 2020 07:29 PM
03 Jul, 2020 03:53 AM

 

02 Jul, 2020 09:09 PM
02 Jul, 2020 03:24 PM
01 Jul, 2020 01:12 PM
30 Jun, 2020 04:26 PM
25 Jun, 2020 05:54 PM
24 Jun, 2020 08:11 AM