BYOC macOS clouds go offline for no reason

Oliver Collyer's Avatar

Oliver Collyer

06 Mar, 2020 08:54 AM

I've noticed this on two Macs now. Initially I thought it was because the machines themselves were sleeping but I have verified this is not the case.

Last night all was building fine. This morning, while the Mac is still accessible across the network via ping and VNC, it shows as "Offline" in the AppVeyor server build environment section. Running a build fails as it cannot find a Mac to build on.

Nothing has changed overnight. Both server and Mac are online. I've also verified a ping succeeds from the Mac to the server.

How can I diagnose this issue? Are there logs? Is there some sort of idle timeout?

  1. 1 Posted by Oliver Collyer on 06 Mar, 2020 08:58 AM

    Oliver Collyer's Avatar

    Also, here is confirmation the service is still running, after typing brew services list:

    Name Status User Plist
    appveyor-host-agent started oliver /Users/oliver/Library/LaunchAgents/homebrew.mxcl.appveyor-host-agent.plist

  2. 2 Posted by Oliver Collyer on 06 Mar, 2020 09:08 AM

    Oliver Collyer's Avatar

    And confirmation from netstat that the Mac is still listening on port 5020:

    appveyor- 440 oliver 229u IPv4 0x9baeb5b4fa55a4a1 0t0 TCP localhost:5020 (LISTEN)
    appveyor- 440 oliver 230u IPv6 0x9baeb5b4f2c67bc9 0t0 TCP localhost:5020 (LISTEN)

    The AppVeyor server can also reach the Mac via ping.

    So from what I can tell there is nothing network-related that is causing the cloud to show up as offline.

    I suspect that restarting the service on the Mac would solve it, but I would like to understand the cause as I need this to be reliable.

  3. 3 Posted by Oliver Collyer on 06 Mar, 2020 09:41 AM

    Oliver Collyer's Avatar

    Ok, I've. been trying to upload/paste/link to the logs but your security systems won't allow me.

    Bit from the logs it appears to lose connection at some point. (I can't see what time, as the logs aren't timestamped, but this may coincide with increased network activity over my LAN during overnight backups.

    The stdout log ends at the connection failure.

    Is the agent missing some reconnection logic? After all, transitory network events do occur, but judging from the logs, it didn't try and reconnect and just gave up.

  4. 4 Posted by Oliver Collyer on 06 Mar, 2020 10:26 AM

    Oliver Collyer's Avatar

    Ok, my work around is to run this script every 60s:

    if grep "Error connecting Host Agent to AppVeyor" /usr/local/var/appveyor/host-agent/host-agent.stdout.log ; then
    brew services stop appveyor-host-agent rm /usr/local/var/appveyor/host-agent/*.log brew services start appveyor-host-agent fi

  5. 5 Posted by Oliver Collyer on 28 Apr, 2020 06:46 AM

    Oliver Collyer's Avatar

    So I've found that this also happens on Windows too.

    It's necessary to schedule the following batch file every 60s to workaround it:

    wevtutil qe "AppVeyor" | findstr /C:"Error connecting Host Agent to AppVeyor" || exit 0
    PowerShell -NoProfile -ExecutionPolicy Bypass -Command "& Stop-Service Appveyor.HostAgent"
    wevtutil cl "AppVeyor"
    PowerShell -NoProfile -ExecutionPolicy Bypass -Command "& Start-Service Appveyor.HostAgent"

    This can be created as a task in task scheduler, and needs to be given admin privileges.

    Perhaps the service applet can be improved so that it periodically attempts to reconnect instead of giving up forever? Otherwise, it seems to me that as soon as there is a network outage of a sufficiently long period of time the whole thing just stops without these workarounds.

    Hope these workarounds help someone.

Reply to this discussion

Internal reply

Formatting help / Preview (switch to plain text) No formatting (switch to Markdown)

Attaching KB article:

»

Attached Files

You can attach files up to 10MB

If you don't have an account yet, we need to confirm you're human and not a machine trying to post spam.

Keyboard shortcuts

Generic

? Show this help
ESC Blurs the current field

Comment Form

r Focus the comment reply box
^ + ↩ Submit the comment

You can use Command ⌘ instead of Control ^ on Mac