VS2017 RC: tests unstable

Christian Soltenborn's Avatar

Christian Soltenborn

01 Mar, 2017 07:24 AM

Hi,

we are developing a Visual Studio extension allowing to execute tests written with the C++ framework Google Test within Visual Studio. Part of our build are end-to-end tests which use Visual Studio's vstest.console.exe to run some test executables and compare the resulting, normalized test output file (trx file) with the according reference file (golden file approach).

Now, these tests have proven to be rather stable on the VS2015 image, but they became really unstable with the VS2017 RC image. We have recently added the older VS2015 image to our build matrix, and the tests are still stable on that image, but not on VS2017RC.

I don't think that this is only a VS issue, since we do not see that behavior on our local machines, where the end-to-end tests run just fine.

Here's a couple of examples: three build, same commit, three different failing outcome (and note that the VS2015 build always succeeded):

https://ci.appveyor.com/project/csoltenborn/googletestadapter-u1cxh...
https://ci.appveyor.com/project/csoltenborn/googletestadapter-u1cxh...
https://ci.appveyor.com/project/csoltenborn/googletestadapter-u1cxh...

There's one test which fails in all three builds, but even that test sometimes succeeds, see e.g. these builds:
failing: https://ci.appveyor.com/project/csoltenborn/googletestadapter-u1cxh...
passing: https://ci.appveyor.com/project/csoltenborn/googletestadapter-u1cxh...

The other failing tests seem to not produce a test result file for reasons we don't know.

Do you have any idea on what the cause could be? Can we assist in helping to find and fix the issue? Let me know if you need any more information...

All the best
Christian

  1. 1 Posted by Christian Solte... on 01 Mar, 2017 07:46 AM

    Christian Soltenborn's Avatar

    One side note: I just saw this comment on the "VS 2017 RC" GitHub issue:

    https://github.com/appveyor/ci/issues/1179#issuecomment-262628758

    There, the AppVeyor vstest.console logger seems to not have been found. This could be related to our problems, since the mostly failing test just checks whether our adapter is found by using vstest.console's /ListDiscoverers option, and it would also be consistent with the other failing tests which do not produce a trx file at all.

  2. Support Staff 2 Posted by Feodor Fitsner on 02 Mar, 2017 04:56 AM

    Feodor Fitsner's Avatar

    Will take a look.

  3. Support Staff 3 Posted by Feodor Fitsner on 02 Mar, 2017 05:29 AM

    Feodor Fitsner's Avatar

    I'm getting an error while trying to list discoverers:

    C:\windows\system32>vstest.console /ListDiscoverers
    Microsoft (R) Test Execution Command Line Tool Version 15.0.0.0
    Copyright (c) Microsoft Corporation.  All rights reserved.
    
    Error: The test source file "/ListDiscoverers" provided was not found.
    

    What's the correct syntax?

  4. 4 Posted by Christian Solte... on 02 Mar, 2017 07:00 AM

    Christian Soltenborn's Avatar

    That's weird... This is the parameters we use in that end-to-end test:

    /Logger:Trx /TestAdapterPath:C:\Users\chris\git\GoogleTestAdapter\GoogleTestAdapter\TestAdapter\bin\Debug /ListDiscoverers

    And this is what I get on the "Developer Command Prompt for VS2017 RC":

    C:\Program Files (x86)\Microsoft Visual Studio\2017\Community>vstest.console /ListDiscoverers /UseVsixExtensions:true
    Microsoft (R)-Testausführungs-Befehlszeilentool Version 15.0.26206.0
    Copyright (c) Microsoft Corporation.  Alle Rechte vorbehalten.
    
    Folgende Testerkennungs-Add-Ins sind verfügbar:
    Microsoft.VisualStudio.TestPlatform.Extensions.GenericTestAdapter.GenericTestDiscoverer
                    Standard-Executor-URI: executor://generictestadapter/v1
                    Unterstützte Dateitypen: .generictest
    Microsoft.VisualStudio.TestPlatform.Extensions.MSAppContainerAdapter.MSAppContainerTestDiscoverer
                    Standard-Executor-URI: executor://msappcontainertestadapter/v1
                    Unterstützte Dateitypen: .appx, .dll, .exe
    Microsoft.VisualStudio.TestPlatform.Extensions.OrderedTestAdapter.OrderedTestDiscoverer
                    Standard-Executor-URI: executor://orderedtestadapter/v1
                    Unterstützte Dateitypen: .orderedtest
    Microsoft.VisualStudio.TestPlatform.Extensions.VSTestIntegration.MSTestDiscoverer
                    Standard-Executor-URI: executor://mstestadapter/v1
                    Unterstützte Dateitypen: .dll, .exe
    Microsoft.VisualStudio.TestTools.CppUnitTestFramework.CppUnitTestExtension.TestDiscoverer
                    Standard-Executor-URI: executor://cppunittestexecutor/v1
                    Unterstützte Dateitypen: .dll, .exe
    PowerShellTools.TestAdapter.PowerShellTestDiscoverer
                    Standard-Executor-URI: executor://powershelltestexecutor/v1
                    Unterstützte Dateitypen: .ps1
    GoogleTestAdapter.TestAdapter.TestDiscoverer
                    Standard-Executor-URI: executor://googletestrunner/v1
                    Unterstützte Dateitypen: .exe
    

    Note that the test adapters can be provided to vstest.console in two ways:
    * using UseVsixExtensions:true makes vstest.console use the test adapters installed into VS * using /TestAdapterPath<path>:makes vstest.console use the adapter available in the specified directory

    Currently, we are only using the latter for our end-to-end tests (we used to have tests for both, but the (VS internal) API to install VSIX extensions has changed since VS2017, and we didn't have time to figure out the new one...).

    Using the latter option on the Developer Command prompt gives the same result::

    C:\Program Files (x86)\Microsoft Visual Studio\2017\Community>vstest.console /Logger:Trx /TestAdapterPath:C:\Users\chris\git\GoogleTestAdapter\GoogleTestAdapter\TestAdapter\bin\Debug /ListDiscoverers
    Microsoft (R)-Testausführungs-Befehlszeilentool Version 15.0.26206.0
    Copyright (c) Microsoft Corporation.  Alle Rechte vorbehalten.
    
    Warnung: Die bei einer VSIX-Installation installierten Testerkennungs-Add-Ins werden ignoriert. Verwenden Sie den Parameter "/UseVsixExtensions", um sie einzuschließen, wenn Ihre Installation "vsix"-Extensions unterstützt. Beispiel: vstest.console.exe /ListDiscoverers /UseVsixExtensions:true
    
    Folgende Testerkennungs-Add-Ins sind verfügbar:
    Microsoft.VisualStudio.TestPlatform.Extensions.GenericTestAdapter.GenericTestDiscoverer
                    Standard-Executor-URI: executor://generictestadapter/v1
                    Unterstützte Dateitypen: .generictest
    Microsoft.VisualStudio.TestPlatform.Extensions.MSAppContainerAdapter.MSAppContainerTestDiscoverer
                    Standard-Executor-URI: executor://msappcontainertestadapter/v1
                    Unterstützte Dateitypen: .appx, .dll, .exe
    Microsoft.VisualStudio.TestPlatform.Extensions.OrderedTestAdapter.OrderedTestDiscoverer
                    Standard-Executor-URI: executor://orderedtestadapter/v1
                    Unterstützte Dateitypen: .orderedtest
    Microsoft.VisualStudio.TestPlatform.Extensions.VSTestIntegration.MSTestDiscoverer
                    Standard-Executor-URI: executor://mstestadapter/v1
                    Unterstützte Dateitypen: .dll, .exe
    Microsoft.VisualStudio.TestTools.CppUnitTestFramework.CppUnitTestExtension.TestDiscoverer
                    Standard-Executor-URI: executor://cppunittestexecutor/v1
                    Unterstützte Dateitypen: .dll, .exe
    GoogleTestAdapter.TestAdapter.TestDiscoverer
                    Standard-Executor-URI: executor://googletestrunner/v1
                    Unterstützte Dateitypen: .exe
    

    Finally, your command works, but doesn't list our test adapter as expected:

    C:\Program Files (x86)\Microsoft Visual Studio\2017\Community>vstest.console /ListDiscoverers
    Microsoft (R)-Testausführungs-Befehlszeilentool Version 15.0.26206.0
    Copyright (c) Microsoft Corporation.  Alle Rechte vorbehalten.
    
    Warnung: Die bei einer VSIX-Installation installierten Testerkennungs-Add-Ins werden ignoriert. Verwenden Sie den Parameter "/UseVsixExtensions", um sie einzuschließen, wenn Ihre Installation "vsix"-Extensions unterstützt. Beispiel: vstest.console.exe /ListDiscoverers /UseVsixExtensions:true
    
    Folgende Testerkennungs-Add-Ins sind verfügbar:
    Microsoft.VisualStudio.TestPlatform.Extensions.GenericTestAdapter.GenericTestDiscoverer
                    Standard-Executor-URI: executor://generictestadapter/v1
                    Unterstützte Dateitypen: .generictest
    Microsoft.VisualStudio.TestPlatform.Extensions.MSAppContainerAdapter.MSAppContainerTestDiscoverer
                    Standard-Executor-URI: executor://msappcontainertestadapter/v1
                    Unterstützte Dateitypen: .appx, .dll, .exe
    Microsoft.VisualStudio.TestPlatform.Extensions.OrderedTestAdapter.OrderedTestDiscoverer
                    Standard-Executor-URI: executor://orderedtestadapter/v1
                    Unterstützte Dateitypen: .orderedtest
    Microsoft.VisualStudio.TestPlatform.Extensions.VSTestIntegration.MSTestDiscoverer
                    Standard-Executor-URI: executor://mstestadapter/v1
                    Unterstützte Dateitypen: .dll, .exe
    Microsoft.VisualStudio.TestTools.CppUnitTestFramework.CppUnitTestExtension.TestDiscoverer
                    Standard-Executor-URI: executor://cppunittestexecutor/v1
                    Unterstützte Dateitypen: .dll, .exe
    
  5. 5 Posted by Christian Solte... on 02 Mar, 2017 07:31 AM

    Christian Soltenborn's Avatar

    Two more remarks: First, note that the second and third end-to-end tests pass almost always, and the only difference to the failing test which only sometimes passes is that we use the /ListExecutors and /ListSettingsProviders options instead of the /ListDiscoverers option.

    Second, I just ran your command on our build machine via Remote Desktop - screenshot attached, the /ListDiscoverers option is recognized just fine.

  6. Support Staff 6 Posted by Feodor Fitsner on 03 Mar, 2017 06:18 AM

    Feodor Fitsner's Avatar

    OK, thanks for the screenshot. I've got some idea.

    Can you please add the following commands to install section of your appveyor.yml:

    install:
    - copy "C:\Program Files (x86)\Microsoft Visual Studio\2017\Community\Common7\IDE\Extensions\as2g30ei.2oj\Extensions\Appveyor.BuildAgent.Api.dll" "C:\Program Files (x86)\Microsoft Visual Studio\2017\Community\Common7\IDE\CommonExtensions\Microsoft\TestWindow\Extensions"
    - copy "C:\Program Files (x86)\Microsoft Visual Studio\2017\Community\Common7\IDE\Extensions\as2g30ei.2oj\Extensions\Appveyor.MSTestLogger.VS2017.dll" "C:\Program Files (x86)\Microsoft Visual Studio\2017\Community\Common7\IDE\CommonExtensions\Microsoft\TestWindow\Extensions"
    - set PATH=C:\Program Files (x86)\Microsoft Visual Studio\2017\Community\Common7\IDE\CommonExtensions\Microsoft\TestWindow;%PATH%
    

    Let me know if that fixed the build.

  7. 7 Posted by Christian Solte... on 03 Mar, 2017 08:45 AM

    Christian Soltenborn's Avatar

    Thanks - that indeed seemed to fix the VS2017 build :-) (3 consecutive builds of the same commit were fine)...

    Obviously, the changes broke the VS2015 build, so I'd like to execute your "patch" only if building with OS VS2017. I've checked the AppVeyor environment variables, but the OS seems to not be accessible - can you please give me a hint on how to figure out which OS the build is running on? Or shall I just go with existence of (e.g.) the VS2017 directory?

  8. Support Staff 8 Posted by Feodor Fitsner on 03 Mar, 2017 08:50 PM

    Feodor Fitsner's Avatar

    No need to add that script to your build anymore. We've updated VS 2017 image - it should be fine now without workaround. Let me know how that worked.

  9. 9 Posted by Christian Solte... on 06 Mar, 2017 06:34 AM

    Christian Soltenborn's Avatar

    Build runs fine without "patch" for three consecutive runs. Thanks once more for the awesome support!

    Is there a best practice to check os version anyways? Our VS2015 job's purpose is to guarantee that we do not break project compatibility for VS2015 (to make contributions easier). The VS2017 job produces the actual artifacts - thus, I'd like to do a couple of things only for os VS2017 (upload artifacts via test_script (in case a test fails) and artifacts, uploading code coverage).

  10. Support Staff 10 Posted by Feodor Fitsner on 06 Mar, 2017 07:24 PM

    Feodor Fitsner's Avatar

    I think the best way is to check installed VS version, kind of:

    init:
    - ps: $env:vs2017 = (test-path 'C:\Program Files (x86)\Microsoft Visual Studio\2017\Community')
    

    and then somewhere during the build:

    - IF DEFINED vs2017 do-this.cmd
    
  11. Ilya Finkelshteyn closed this discussion on 25 Aug, 2018 02:15 AM.

Comments are currently closed for this discussion. You can start a new one.

Keyboard shortcuts

Generic

? Show this help
ESC Blurs the current field

Comment Form

r Focus the comment reply box
^ + ↩ Submit the comment

You can use Command ⌘ instead of Control ^ on Mac