on 19 Jun, 2017 07:32 AM
Thanks for quick response.
I analyzed the logs and it looks like the problem is not in the tests, but in the PowerShell scripts that are used to execute them. Each test program/script is run in a subprocess / PS job. The execution time of each job is printed in the log and in most cases it's below 1s. But when you compare it with the timestamp of the log line, it appears that spawning this subprocess/job may take a couple of seconds. It used to be ~3s (average), now it is 9s avg. (not less than 4 sec.!!!).
I have no idea whether this change it is related to the PowerShell itself or some other Windows updates.
on 27 Jun, 2017 09:48 AM
Sorry for late response.
Eventually I found some time to write a simple script that reproduces the issue. It's a simplified version of the actual script we use - stripped out from everything that is not relevant.
See: src/test/RUNTESTS.ps1 + src/test/TEST.ps1
The comments in these files should explain how it works.
Note that the actual test time is about 10s (10 iterations x 1s), but since the overhead of spawning new job is about 1s (that's very long, but still acceptable), then with additional 10s and it gives us 20s in total. (that's how it used to be)
Now the total time is ~50s, so the overhead is 40s (4s per job), which is veeeery bad.
Based on your results and on the fact we still observe sporadic timeouts when using old image, I suspect the problem may be related not to the image update, but rather some other factor - server machines, build cloud, etc...
Apparently update 4014507 causes this. We hope that installing latest updates on top of it could fix that, but they did not. We are not sure if it is safe to uninstall it globally from image. For now workaround is to uninstall it at init stage as I did here. It takes about 40 seconds (including reboot), which maybe tolerable for you, in comparison to performance degrade you observe.
We will look what else we can do, but for now this is workaround we propose.
Also sorry for confusion with Previous Visual Studio 2015 image -- it worked the same as new one sometimes because it was actually new one. We are in process of adding more Hyper-V hosts to our datacenter and during this process there are short periods when older images still not deployed to new host.