VM stops processing tests

oliver's Avatar

oliver

29 May, 2018 09:17 AM

I have been experiencing this problem quite often recently:

A job hangs (without processing any more tests) until it is canceled after 1 hours.
My config:

environment:
    PSGalleryAPIKey:
        secure: ....
    access_token:
        secure: ....
    matrix:
        - APPVEYOR_BUILD_WORKER_IMAGE: Visual Studio 2017
        - APPVEYOR_BUILD_WORKER_IMAGE: Ubuntu
          PowershellVersion: "6.0.0"
        - APPVEYOR_BUILD_WORKER_IMAGE: Visual Studio 2017
          PowershellVersion: "6.0.0"


matrix: fast_finish: true


version: 0.1.{build}


skip_tags: true


skip_commits: message: /skip-tests/ files: - .github/ - .vscode/ - README.md - .gitattributes - .gitignore - .env*


pull_requests: do_not_increment_build_number: true


init: - ps: | if (-not ($env:PowershellVersion)) {$env:PowershellVersion = "5.1"} if (([version]$env:PowershellVersion).Major -ge 6) {$env:posh = "pwsh"} else {$env:posh = "powershell"} Add-Content (Join-Path $Home ".git-credentials") "https://$($env:access_token):x-oauth-basic@github.com`n" - git config --global credential.helper "store --file ~/.git-credentials" - git config --global user.email "[email blocked]" - git config --global user.name "AtlassianPS automation"


install: - ps: '& $env:posh -File "./Tools/setup.ps1"'


build_script: - ps: '& $env:posh -Command "Invoke-Build"'


############# on_failure: ################# - ps: $blockRdp = $true; iex ((new-object net.webclient).DownloadString('https://raw.githubusercontent.com/appveyor/ci/master/scripts/enable-rdp.ps1'))

most common is for it to hang on the Ubuntu job... but it also happens on the 1st (Visual Studio 2017).
But I did not see this happen to the 3rd job.

Here are some examples:
https://ci.appveyor.com/project/AtlassianPS/jiraps/build/2.7.2-ulxv...
https://ci.appveyor.com/project/AtlassianPS/jiraps/build/2.7.2-xbyq...

  1. Support Staff 1 Posted by Ilya Finkelshte... on 29 May, 2018 10:51 PM

    Ilya Finkelshteyn's Avatar

    From what I see, I wold first try to check if this is related to certain tests, specifically Get-JiraIssue.Tests or/and Get-JiraGroupMember.Tests.ps1. Can you try to remove them both and run build several times in a row?

  2. 2 Posted by oliver on 30 May, 2018 07:56 PM

    oliver's Avatar

    I did not remove it yet, but here are some more builds that failed:

    https://ci.appveyor.com/project/AtlassianPS/jiraps/build/2.7.2-dlut...

    https://ci.appveyor.com/project/AtlassianPS/jiraps/build/2.7.2-krle... <-- this one is particularly interesting

  3. Support Staff 3 Posted by Ilya Finkelshte... on 31 May, 2018 04:52 AM

    Ilya Finkelshteyn's Avatar

    Sorry I cannot find second (particularly interesting) one. Also can you please point the one which hung on Visual Studio 2017?

  4. 4 Posted by oliver on 03 Jun, 2018 09:10 AM

    oliver's Avatar

    Hi Ilya.

    Here are some that failed on Visual Studio 2017:

    and here are some more (on ubuntu):

    one thing that baffles me, is that it appears as the result of Invoke-Pester is uploaded to the build as artefacts.

    here is how I found the links mentioned above:

    $token = 'IAmNotTelling'


    $numOfBuilds = 100 $headers = @{ "Authorization" = "Bearer $token" "Content-type" = "application/json" }


    $invokeRestMethodSplat = @{ Method = 'Get' Headers = $headers Uri = "https://ci.appveyor.com/api/projects/AtlassianPS/jiraps/history?recordsNumber=$numOfBuilds" }


    $failures = (Invoke-RestMethod @invokeRestMethodSplat).builds | ? status -eq "failed"


    $result = New-Object -TypeName System.Collections.ArrayList


    foreach ($fail in $failures) {

    $relevant = $false
    
    $invokeRestMethodSplat = @{
        Method = &#39;Get&#39;
        Headers = $headers
        Uri = &quot;https://ci.appveyor.com/api/projects/AtlassianPS/jiraps/build/$($fail.version)&quot;
    }
    $fail = (Invoke-RestMethod @invokeRestMethodSplat).build
    
    foreach ($job in $fail.jobs) {
        if (([datetime]$job.finished - [datetime]$job.created) -gt [timespan]&quot;00:59:00&quot;) {
            $relevant = $true
        }
    }
    
    if ($relevant) {
        $result.Add($fail)
    }
    
    
    
    
    }
    $result | % { "https://ci.appveyor.com/project/AtlassianPS/jiraps/build/$($_.version)" }
  5. 5 Posted by oliver on 03 Jun, 2018 09:42 AM

    oliver's Avatar

    The one build I said was interesting was deleted. I am sorry for that.
    But here is a new one with exactly the same behavior:

    https://ci.appveyor.com/project/AtlassianPS/jiraps/build/job/3fi862...

  6. 6 Posted by oliver on 05 Jun, 2018 05:57 PM

    oliver's Avatar

    Bump... Sorry

  7. Support Staff 7 Posted by Ilya Finkelshte... on 05 Jun, 2018 06:24 PM

    Ilya Finkelshteyn's Avatar

    Sorry for delay. Will look into this today.
    It is difficult to say anything in this situation without putting hands on a repro. Should we be able to reproduce the problem by forking you repository and creating AppVeyor project?

    I am asking because I see you have some secure variables in your YAML. They obviously will not be decoded in our fork build, and I do not know if this will affect our ability to get a repro.

  8. 8 Posted by oliver on 06 Jun, 2018 04:01 PM

    oliver's Avatar

    The secret env vars are not relevant for the parts that are failing.

    If the problem is code, you should be able to repo.
    If the problem is VMs, you will have to trigger a lot of builds to see some failing
    If the problem is in my account, you will not be able to repro

  9. Support Staff 9 Posted by Owen McDonnell on 06 Jun, 2018 09:48 PM

    Owen McDonnell's Avatar

    I was able to stabilize the build for ubuntu by using our google compute engine build cloud. You can see the builds here starting from build 2.7.14 and on every build is the same - failing due to the same 5 tests.

    This leads me to believe that perhaps your build was non-determinisically exhausting some resources of the build machine. I have enabled google compute engine cloud for your account. Add this environment variable to your yaml and see if it makes a difference for you.

    appveyor_build_worker_cloud: gce
    
    Note that we don't keep google compute engine worker 'pre-heated' in the same way we do other build machines so there will be a longer lag for the build to kick off.

Reply to this discussion

Internal reply

Formatting help / Preview (switch to plain text) No formatting (switch to Markdown)

Attaching KB article:

»

Attached Files

You can attach files up to 10MB

If you don't have an account yet, we need to confirm you're human and not a machine trying to post spam.

Keyboard shortcuts

Generic

? Show this help
ESC Blurs the current field

Comment Form

r Focus the comment reply box
^ + ↩ Submit the comment

You can use Command ⌘ instead of Control ^ on Mac