tag:help.appveyor.com,2012-11-13:/discussions/problems/24667-nunit-approvaltests-tests-started-failingAppVeyor: Discussion 2019-11-05T21:01:29Ztag:help.appveyor.com,2012-11-13:Comment/474970782019-08-03T20:37:19Z2019-08-03T20:37:44ZNUnit + ApprovalTests tests started failing<div><p>I have added attributes as suggested in the error message and my builds got fixed.<br>
Then they started randomly failing with some other tests failing....</p>
<p>The same commit:</p>
<ul>
<li><a href="https://ci.appveyor.com/project/gitextensions/gitextensions/builds/26450646">https://ci.appveyor.com/project/gitextensions/gitextensions/builds/...</a> - success</li>
<li><a href="https://ci.appveyor.com/project/gitextensions/gitextensions/builds/26450538">https://ci.appveyor.com/project/gitextensions/gitextensions/builds/...</a> - fail</li>
<li><a href="https://ci.appveyor.com/project/gitextensions/gitextensions/builds/26450395">https://ci.appveyor.com/project/gitextensions/gitextensions/builds/...</a> - fail</li>
</ul>
<p>Note, the failed builds fail in different tests.<br>
The successful build would have failed with a time out - I logged in the box and dismissed a popup - it was breaking in some other random place...</p>
<p>It is definitely something to do with the environments, and it started in the past few days.</p></div>russkietag:help.appveyor.com,2012-11-13:Comment/474970782019-08-03T23:42:24Z2019-08-03T23:42:24ZNUnit + ApprovalTests tests started failing<div><p>This can be indeed related to the environment change, specifically <a href="https://www.appveyor.com/updates/2019/08/01/">August 1st images update</a>. Please try to set image to the <code>Previous Visual Studio 2017</code> and see if this helps.</p>
<p>But please come back to us in any case. Even if switching to "previous" helps, it can be treated only as a temporary workaround, valid only till next update. So we will be needed to root cause it together.</p></div>Ilya Finkelshteyntag:help.appveyor.com,2012-11-13:Comment/474970782019-08-23T20:15:25Z2019-08-23T20:15:25ZNUnit + ApprovalTests tests started failing<div><p>It appears I have never received a notification about your reply.<br>
I will try to test it in coming days (I'm flat out busy, but I'll try to carve some time) and will come back.</p>
<p>We've had tests that previously were stable now failing at random. Re-running builds fixes them back...</p></div>russkietag:help.appveyor.com,2012-11-13:Comment/474970782019-09-03T12:51:34Z2019-09-03T12:51:34ZNUnit + ApprovalTests tests started failing<div><p>I have done some more testing and I tend to believe there may be a regression in the current VS 2017 image.</p>
<p>One thing definitely not working correctly is opencover.<br>
VS 2017 - fails to submit test results:<br>
<a href="https://ci.appveyor.com/project/gitextensions/gitextensions/builds/27133479#L196">https://ci.appveyor.com/project/gitextensions/gitextensions/builds/...</a></p>
<p>Previous VS 2017 - all good:<br>
<a href="https://ci.appveyor.com/project/gitextensions/gitextensions/builds/27141204#L203">https://ci.appveyor.com/project/gitextensions/gitextensions/builds/...</a></p>
<p>The only difference between the two commits is the targeted image.</p></div>russkietag:help.appveyor.com,2012-11-13:Comment/474970782019-09-03T12:55:36Z2019-09-03T12:55:36ZNUnit + ApprovalTests tests started failing<div><p>The current VS 2017 image appears very unstable.</p>
<p>It is a total gamble whether the same commit would fail or build sucessfully:<br>
- good <a href="https://ci.appveyor.com/project/gitextensions/gitextensions/builds/27133479">https://ci.appveyor.com/project/gitextensions/gitextensions/builds/...</a> - failed <a href="https://ci.appveyor.com/project/gitextensions/gitextensions/builds/27133350">https://ci.appveyor.com/project/gitextensions/gitextensions/builds/...</a></p>
<p>In the failed build one of the C++ projects builds just silently failed (<a href="https://ci.appveyor.com/project/gitextensions/gitextensions/builds/27133350#L281">https://ci.appveyor.com/project/gitextensions/gitextensions/builds/...</a>) whereas in the subsequent build it built successfully (<a href="https://ci.appveyor.com/project/gitextensions/gitextensions/builds/27133479#L255">https://ci.appveyor.com/project/gitextensions/gitextensions/builds/...</a>)</p></div>russkietag:help.appveyor.com,2012-11-13:Comment/474970782019-09-03T13:04:45Z2019-09-03T13:04:45ZNUnit + ApprovalTests tests started failing<div><p>We have also tried to build against VS 2019 image.<br>
A number of tests started failing out of the blue - <a href="https://ci.appveyor.com/project/gitextensions/gitextensions/builds/27108153#L178">https://ci.appveyor.com/project/gitextensions/gitextensions/builds/...</a>, opencover also didn't work</p></div>russkietag:help.appveyor.com,2012-11-13:Comment/474970782019-09-03T13:22:42Z2019-09-03T13:22:42ZNUnit + ApprovalTests tests started failing<div><p>To fix opencover issue it looks like it has to be run with <code>-register:administrator</code> as opposed to <code>-register:user</code> that we historically had.<br>
<a href="https://ci.appveyor.com/project/gitextensions/gitextensions/builds/27146847">https://ci.appveyor.com/project/gitextensions/gitextensions/builds/...</a></p>
<p>I'm not sure what changed, but I can speculate it may be some sort of permissions issue.</p></div>russkietag:help.appveyor.com,2012-11-13:Comment/474970782019-09-04T18:58:33Z2019-09-04T18:58:33ZNUnit + ApprovalTests tests started failing<div><p>I took a look at one specific failure you sent, <a href="https://ci.appveyor.com/project/gitextensions/gitextensions/builds/27133350">this</a> one. It fails because <code>The system cannot find the file '..\GitExtensionsShellEx\Release\GitExtensionsShellEx32.dll'</code>. However <a href="https://ci.appveyor.com/project/gitextensions/gitextensions/builds/27133350#L322">before</a> it builds only 64-bit version of that file: <code>C:\projects\gitextensions\GitExtensionsShellEx\Release-x64\GitExtensionsShellEx64.dll</code>. On the <a href="https://ci.appveyor.com/project/gitextensions/gitextensions/builds/27133479">good build</a> it builds both 32- and 64-bit version of <code>GitExtensionsShellEx</code>. Also it seems that "good" build rise a warning <code>warning C4996: 'GetVersion': was declared deprecated</code> while "bad" not. Does it rings any bells?</p></div>Ilya Finkelshteyntag:help.appveyor.com,2012-11-13:Comment/474970782019-09-04T19:24:57Z2019-09-04T19:24:57ZNUnit + ApprovalTests tests started failing<div><p>Yes, that's precisely what was happening and the expectation that both builds produce near identical output (including warnings).<br>
Somehow on the bad build the 32-bit build silently fails/terminates.</p></div>russkietag:help.appveyor.com,2012-11-13:Comment/474970782019-09-05T01:56:46Z2019-09-05T01:56:46ZNUnit + ApprovalTests tests started failing<div><p>I would add <code>catch</code> <a href="https://github.com/gitextensions/gitextensions/blob/release/3.2/Build/Prepare-Artifacts.ps1#L55-L67">here</a> and write the exception. Something like <code>catch {$_}</code> should work. Without <code>catch</code> it might hide real error.</p></div>Ilya Finkelshteyn