Build/test timeouts

Krzysztof Czurylo's Avatar

Krzysztof Czurylo

18 Jun, 2017 11:53 AM

Hi,

For a last few days, we have seen the prolonged build/test time on pmem/nvml project (https://ci.appveyor.com/project/pmem/nvml), which results in build time out/cancellation after 1 hour.
It looks like test execution time is almost twice as long as usual. It used to take about 30 mins. to execute all the unit tests for one configuration, so including the build time, it was less than 40 mins. per job. Now, all the jobs timeouts at about 50% of work done.
At first, I thought it was a regression on our side, but i couldn't find any reason for that, so I decided to re-build the last successful build and it also failed (time out). See below.
Could it be related to the environment update on 16.06?

last successful (6 days ago) - about 37 min.
https://ci.appveyor.com/project/pmem/nvml/build/1.0.329

same commit rebuilt on 17.06.2017 - canceled after 1hr
https://ci.appveyor.com/project/pmem/nvml/build/1.0.357

Best regards,
K.

  1. 1 Posted by Krzysztof Czury... on 18 Jun, 2017 11:55 AM

    Krzysztof Czurylo's Avatar

    Ooops,

    Of course, I meant the update on Jul 12th.
    https://www.appveyor.com/updates/

    K.

  2. Support Staff 2 Posted by Feodor Fitsner on 18 Jun, 2017 06:19 PM

    Feodor Fitsner's Avatar

    Hi Krzysztof,

    Indeed, I've tested the last commit on both "previous" and current images and noticed the regression: https://ci.appveyor.com/project/FeodorFitsner/nvml/history

    Among other things installed on June 12th update were latest Windows updates and most probably something could be related to that. Will be taking a look.

    Can you give me some hint what part of the test could become slow - is it something IO related or perhaps some chore PS task before/after test, etc.?

  3. Support Staff 3 Posted by Feodor Fitsner on 19 Jun, 2017 02:56 AM

    Feodor Fitsner's Avatar

    Oh, forgot to mention - to unblock the builds please run on image: Previous Visual Studio 2015 for now.

  4. 4 Posted by Krzysztof Czury... on 19 Jun, 2017 07:32 AM

    Krzysztof Czurylo's Avatar

    Hi Feodor,

    Thanks for quick response.
    I analyzed the logs and it looks like the problem is not in the tests, but in the PowerShell scripts that are used to execute them. Each test program/script is run in a subprocess / PS job. The execution time of each job is printed in the log and in most cases it's below 1s. But when you compare it with the timestamp of the log line, it appears that spawning this subprocess/job may take a couple of seconds. It used to be ~3s (average), now it is 9s avg. (not less than 4 sec.!!!).
    I have no idea whether this change it is related to the PowerShell itself or some other Windows updates.

  5. Support Staff 5 Posted by Feodor Fitsner on 19 Jun, 2017 01:51 PM

    Feodor Fitsner's Avatar

    How do you invoke PS scripts with tests? I'd like to to a reproduce to use in regression testing.

  6. 6 Posted by Krzysztof Czury... on 20 Jun, 2017 03:32 PM

    Krzysztof Czurylo's Avatar

    It's something like this:

    $sb = {
        cd $args[0]
        Invoke-Expression $args[1]
    }
    $j1 = Start-Job -Name $name -ScriptBlock $sb -ArgumentList (pwd).Path, ".\$runscript"
    ...
    Receive-Job -Job $j1

    If needed, I can write a simple script that would illustrate that.

    K.

  7. Support Staff 7 Posted by Feodor Fitsner on 20 Jun, 2017 04:05 PM

    Feodor Fitsner's Avatar

    Yes, please, an example appveyor build in public repository would be great help. Maybe with a loop to see real delay. Thank you!

Reply to this discussion

Internal reply

Formatting help / Preview (switch to plain text) No formatting (switch to Markdown)

Attaching KB article:

»

Attached Files

You can attach files up to 10MB

If you don't have an account yet, we need to confirm you're human and not a machine trying to post spam.

Keyboard shortcuts

Generic

? Show this help
ESC Blurs the current field

Comment Form

r Focus the comment reply box
^ + ↩ Submit the comment

You can use Command ⌘ instead of Control ^ on Mac