tag:help.appveyor.com,2012-11-13:/discussions/problems/26494-byoc-macos-clouds-go-offline-for-no-reasonAppVeyor: Discussion 2022-09-09T11:53:19Ztag:help.appveyor.com,2012-11-13:Comment/481403052020-03-06T08:58:36Z2020-03-06T08:58:37ZBYOC macOS clouds go offline for no reason<div><p>Also, here is confirmation the service is still running, after typing brew services list:</p>
<p>Name Status User Plist<br>
appveyor-host-agent started oliver /Users/oliver/Library/LaunchAgents/homebrew.mxcl.appveyor-host-agent.plist</p></div>Oliver Collyertag:help.appveyor.com,2012-11-13:Comment/481403052020-03-06T09:08:26Z2020-03-06T09:08:27ZBYOC macOS clouds go offline for no reason<div><p>And confirmation from netstat that the Mac is still listening on port 5020:</p>
<p>appveyor- 440 oliver 229u IPv4 0x9baeb5b4fa55a4a1 0t0 TCP localhost:5020 (LISTEN)<br>
appveyor- 440 oliver 230u IPv6 0x9baeb5b4f2c67bc9 0t0 TCP localhost:5020 (LISTEN)</p>
<p>The AppVeyor server can also reach the Mac via ping.</p>
<p>So from what I can tell there is nothing network-related that is causing the cloud to show up as offline.</p>
<p>I suspect that restarting the service on the Mac would solve it, but I would like to understand the cause as I need this to be reliable.</p></div>Oliver Collyertag:help.appveyor.com,2012-11-13:Comment/481403052020-03-06T09:41:41Z2020-03-06T09:41:42ZBYOC macOS clouds go offline for no reason<div><p>Ok, I've. been trying to upload/paste/link to the logs but your security systems won't allow me.</p>
<p>Bit from the logs it appears to lose connection at some point. (I can't see what time, as the logs aren't timestamped, but this may coincide with increased network activity over my LAN during overnight backups.</p>
<p>The stdout log ends at the connection failure.</p>
<p>Is the agent missing some reconnection logic? After all, transitory network events do occur, but judging from the logs, it didn't try and reconnect and just gave up.</p></div>Oliver Collyertag:help.appveyor.com,2012-11-13:Comment/481403052020-03-06T10:26:19Z2020-03-06T10:26:20ZBYOC macOS clouds go offline for no reason<div><p>Ok, my work around is to run this script every 60s:</p>
<p>if grep "Error connecting Host Agent to AppVeyor" /usr/local/var/appveyor/host-agent/host-agent.stdout.log ; then<br>
brew services stop appveyor-host-agent rm /usr/local/var/appveyor/host-agent/*.log brew services start appveyor-host-agent fi</p></div>Oliver Collyertag:help.appveyor.com,2012-11-13:Comment/481403052020-04-28T06:46:47Z2020-04-28T06:46:48ZBYOC macOS clouds go offline for no reason<div><p>So I've found that this also happens on Windows too.</p>
<p>It's necessary to schedule the following batch file every 60s to workaround it:</p>
<p>wevtutil qe "AppVeyor" | findstr /C:"Error connecting Host Agent to AppVeyor" || exit 0<br>
PowerShell -NoProfile -ExecutionPolicy Bypass -Command "& Stop-Service Appveyor.HostAgent"<br>
wevtutil cl "AppVeyor"<br>
PowerShell -NoProfile -ExecutionPolicy Bypass -Command "& Start-Service Appveyor.HostAgent"</p>
<p>This can be created as a task in task scheduler, and needs to be given admin privileges.</p>
<p>Perhaps the service applet can be improved so that it periodically attempts to reconnect instead of giving up forever? Otherwise, it seems to me that as soon as there is a network outage of a sufficiently long period of time the whole thing just stops without these workarounds.</p>
<p>Hope these workarounds help someone.</p></div>Oliver Collyertag:help.appveyor.com,2012-11-13:Comment/481403052022-09-09T11:53:17Z2022-09-09T11:53:19ZBYOC macOS clouds go offline for no reason<div><p>I wonder if there can be improvements to the cmdlet for this?</p>
<p>Here is how things currently are, 2.5 years later. This is the output from the host agent on a Mac when I stop the appveyor server service (Windows) for a minute or so, and then start it again.</p>
<p>As you can see, the host agen, never recovers, it stays forever on "Stopping Host Agent connection...".</p>
<p>One has to manually stop/start the service, to enable it to reconnect. Clearly you have put in some disconnection detection and reconnection logic, but it would appear that it isn't working in this case.</p>
<p>info: Appveyor.HostAgent.Services.HostAgentHubClientCore[0]<br>
Connected to HostAgentHub info: Appveyor.HostAgent.AppHost[0]<br>
Registering Host Agent info: Appveyor.HostAgent.AppHost[0]<br>
Host Agent registered info: Appveyor.HostAgent.AppHost[0]<br>
Registering all clouds received from AppVeyor info: Appveyor.HostAgent.BuildAgentProcess.WorkerCloud[0]<br>
Configuring cloud 13 info: Appveyor.HostAgent.BuildAgentProcess.WorkerCloud[0]<br>
[worker-13-001] Start info: Appveyor.HostAgent.BuildAgentProcess.WorkerCloud[0]<br>
[worker-13-001] Run info: Appveyor.HostAgent.BuildAgentProcess.WorkerCloud[0]<br>
[worker-13-001] Ready info: Appveyor.HostAgent.AppHost[0]<br>
Updating cloud availability (buildCloudId=13): 20 worker(s) available, 0 worker(s) busy info: Appveyor.HostAgent.BuildAgentProcess.WorkerCloud[0]<br>
[worker-13-001] Started info: Appveyor.HostAgent.BuildAgentProcess.WorkerCloud[0]<br>
[worker-13-013] Start info: Appveyor.HostAgent.BuildAgentProcess.WorkerCloud[0]<br>
[worker-13-013] Run info: Appveyor.HostAgent.BuildAgentProcess.WorkerCloud[0]<br>
[worker-13-013] Ready info: Appveyor.HostAgent.AppHost[0]<br>
Updating cloud availability (buildCloudId=13): 20 worker(s) available, 0 worker(s) busy info: Appveyor.HostAgent.BuildAgentProcess.WorkerCloud[0]<br>
[worker-13-013] Started info: Appveyor.HostAgent.BuildAgentProcess.WorkerCloud[0]<br>
[worker-13-005] Start info: Appveyor.HostAgent.BuildAgentProcess.WorkerCloud[0]<br>
[worker-13-005] Run info: Appveyor.HostAgent.BuildAgentProcess.WorkerCloud[0]<br>
[worker-13-005] Ready info: Appveyor.HostAgent.AppHost[0]<br>
Updating cloud availability (buildCloudId=13): 20 worker(s) available, 0 worker(s) busy info: Appveyor.HostAgent.BuildAgentProcess.WorkerCloud[0]<br>
[worker-13-005] Started info: Appveyor.HostAgent.BuildAgentProcess.WorkerCloud[0]<br>
[worker-13-004] Start info: Appveyor.HostAgent.BuildAgentProcess.WorkerCloud[0]<br>
[worker-13-004] Run info: Appveyor.HostAgent.BuildAgentProcess.WorkerCloud[0]<br>
[worker-13-004] Ready info: Appveyor.HostAgent.AppHost[0]<br>
Updating cloud availability (buildCloudId=13): 20 worker(s) available, 0 worker(s) busy info: Appveyor.HostAgent.BuildAgentProcess.WorkerCloud[0]<br>
[worker-13-004] Started info: Appveyor.HostAgent.BuildAgentProcess.WorkerCloud[0]<br>
[worker-13-019] Start info: Appveyor.HostAgent.BuildAgentProcess.WorkerCloud[0]<br>
[worker-13-019] Run info: Appveyor.HostAgent.BuildAgentProcess.WorkerCloud[0]<br>
[worker-13-019] Ready info: Appveyor.HostAgent.AppHost[0]<br>
Updating cloud availability (buildCloudId=13): 20 worker(s) available, 0 worker(s) busy info: Appveyor.HostAgent.BuildAgentProcess.WorkerCloud[0]<br>
[worker-13-019] Started info: Appveyor.HostAgent.BuildAgentProcess.WorkerCloud[0]<br>
[worker-13-006] Start info: Appveyor.HostAgent.BuildAgentProcess.WorkerCloud[0]<br>
[worker-13-006] Run info: Appveyor.HostAgent.BuildAgentProcess.WorkerCloud[0]<br>
[worker-13-006] Ready info: Appveyor.HostAgent.AppHost[0]<br>
Updating cloud availability (buildCloudId=13): 20 worker(s) available, 0 worker(s) busy info: Appveyor.HostAgent.BuildAgentProcess.WorkerCloud[0]<br>
[worker-13-006] Started info: Appveyor.HostAgent.BuildAgentProcess.WorkerCloud[0]<br>
[worker-13-017] Start info: Appveyor.HostAgent.BuildAgentProcess.WorkerCloud[0]<br>
[worker-13-017] Run info: Appveyor.HostAgent.BuildAgentProcess.WorkerCloud[0]<br>
[worker-13-017] Ready info: Appveyor.HostAgent.AppHost[0]<br>
Updating cloud availability (buildCloudId=13): 20 worker(s) available, 0 worker(s) busy info: Appveyor.HostAgent.BuildAgentProcess.WorkerCloud[0]<br>
[worker-13-017] Started info: Appveyor.HostAgent.BuildAgentProcess.WorkerCloud[0]<br>
[worker-13-003] Start info: Appveyor.HostAgent.BuildAgentProcess.WorkerCloud[0]<br>
[worker-13-003] Run info: Appveyor.HostAgent.BuildAgentProcess.WorkerCloud[0]<br>
[worker-13-003] Ready info: Appveyor.HostAgent.AppHost[0]<br>
Updating cloud availability (buildCloudId=13): 20 worker(s) available, 0 worker(s) busy info: Appveyor.HostAgent.BuildAgentProcess.WorkerCloud[0]<br>
[worker-13-003] Started info: Appveyor.HostAgent.BuildAgentProcess.WorkerCloud[0]<br>
[worker-13-014] Start info: Appveyor.HostAgent.BuildAgentProcess.WorkerCloud[0]<br>
[worker-13-014] Run info: Appveyor.HostAgent.BuildAgentProcess.WorkerCloud[0]<br>
[worker-13-014] Ready info: Appveyor.HostAgent.AppHost[0]<br>
Updating cloud availability (buildCloudId=13): 20 worker(s) available, 0 worker(s) busy info: Appveyor.HostAgent.BuildAgentProcess.WorkerCloud[0]<br>
[worker-13-014] Started info: Appveyor.HostAgent.BuildAgentProcess.WorkerCloud[0]<br>
[worker-13-012] Start info: Appveyor.HostAgent.BuildAgentProcess.WorkerCloud[0]<br>
[worker-13-012] Run info: Appveyor.HostAgent.BuildAgentProcess.WorkerCloud[0]<br>
[worker-13-012] Ready info: Appveyor.HostAgent.AppHost[0]<br>
Updating cloud availability (buildCloudId=13): 20 worker(s) available, 0 worker(s) busy info: Appveyor.HostAgent.BuildAgentProcess.WorkerCloud[0]<br>
[worker-13-012] Started info: Appveyor.HostAgent.BuildAgentProcess.WorkerCloud[0]<br>
[worker-13-016] Start info: Appveyor.HostAgent.BuildAgentProcess.WorkerCloud[0]<br>
[worker-13-016] Run info: Appveyor.HostAgent.BuildAgentProcess.WorkerCloud[0]<br>
[worker-13-016] Ready info: Appveyor.HostAgent.AppHost[0]<br>
Updating cloud availability (buildCloudId=13): 20 worker(s) available, 0 worker(s) busy info: Appveyor.HostAgent.BuildAgentProcess.WorkerCloud[0]<br>
[worker-13-016] Started info: Appveyor.HostAgent.BuildAgentProcess.WorkerCloud[0]<br>
[worker-13-002] Start info: Appveyor.HostAgent.BuildAgentProcess.WorkerCloud[0]<br>
[worker-13-002] Run info: Appveyor.HostAgent.BuildAgentProcess.WorkerCloud[0]<br>
[worker-13-002] Ready info: Appveyor.HostAgent.AppHost[0]<br>
Updating cloud availability (buildCloudId=13): 20 worker(s) available, 0 worker(s) busy info: Appveyor.HostAgent.BuildAgentProcess.WorkerCloud[0]<br>
[worker-13-002] Started info: Appveyor.HostAgent.BuildAgentProcess.WorkerCloud[0]<br>
[worker-13-008] Start info: Appveyor.HostAgent.BuildAgentProcess.WorkerCloud[0]<br>
[worker-13-008] Run info: Appveyor.HostAgent.BuildAgentProcess.WorkerCloud[0]<br>
[worker-13-008] Ready info: Appveyor.HostAgent.AppHost[0]<br>
Updating cloud availability (buildCloudId=13): 20 worker(s) available, 0 worker(s) busy info: Appveyor.HostAgent.BuildAgentProcess.WorkerCloud[0]<br>
[worker-13-008] Started info: Appveyor.HostAgent.BuildAgentProcess.WorkerCloud[0]<br>
[worker-13-009] Start info: Appveyor.HostAgent.BuildAgentProcess.WorkerCloud[0]<br>
[worker-13-009] Run info: Appveyor.HostAgent.BuildAgentProcess.WorkerCloud[0]<br>
[worker-13-009] Ready info: Appveyor.HostAgent.AppHost[0]<br>
Updating cloud availability (buildCloudId=13): 20 worker(s) available, 0 worker(s) busy info: Appveyor.HostAgent.BuildAgentProcess.WorkerCloud[0]<br>
[worker-13-009] Started info: Appveyor.HostAgent.BuildAgentProcess.WorkerCloud[0]<br>
[worker-13-018] Start info: Appveyor.HostAgent.BuildAgentProcess.WorkerCloud[0]<br>
[worker-13-018] Run info: Appveyor.HostAgent.BuildAgentProcess.WorkerCloud[0]<br>
[worker-13-018] Ready info: Appveyor.HostAgent.AppHost[0]<br>
Updating cloud availability (buildCloudId=13): 20 worker(s) available, 0 worker(s) busy info: Appveyor.HostAgent.BuildAgentProcess.WorkerCloud[0]<br>
[worker-13-018] Started info: Appveyor.HostAgent.BuildAgentProcess.WorkerCloud[0]<br>
[worker-13-015] Start info: Appveyor.HostAgent.BuildAgentProcess.WorkerCloud[0]<br>
[worker-13-015] Run info: Appveyor.HostAgent.BuildAgentProcess.WorkerCloud[0]<br>
[worker-13-015] Ready info: Appveyor.HostAgent.AppHost[0]<br>
Updating cloud availability (buildCloudId=13): 20 worker(s) available, 0 worker(s) busy info: Appveyor.HostAgent.BuildAgentProcess.WorkerCloud[0]<br>
[worker-13-015] Started info: Appveyor.HostAgent.BuildAgentProcess.WorkerCloud[0]<br>
[worker-13-020] Start info: Appveyor.HostAgent.BuildAgentProcess.WorkerCloud[0]<br>
[worker-13-020] Run info: Appveyor.HostAgent.BuildAgentProcess.WorkerCloud[0]<br>
[worker-13-020] Ready info: Appveyor.HostAgent.AppHost[0]<br>
Updating cloud availability (buildCloudId=13): 20 worker(s) available, 0 worker(s) busy info: Appveyor.HostAgent.BuildAgentProcess.WorkerCloud[0]<br>
[worker-13-020] Started info: Appveyor.HostAgent.BuildAgentProcess.WorkerCloud[0]<br>
[worker-13-007] Start info: Appveyor.HostAgent.BuildAgentProcess.WorkerCloud[0]<br>
[worker-13-007] Run info: Appveyor.HostAgent.BuildAgentProcess.WorkerCloud[0]<br>
[worker-13-007] Ready info: Appveyor.HostAgent.AppHost[0]<br>
Updating cloud availability (buildCloudId=13): 20 worker(s) available, 0 worker(s) busy info: Appveyor.HostAgent.BuildAgentProcess.WorkerCloud[0]<br>
[worker-13-007] Started info: Appveyor.HostAgent.BuildAgentProcess.WorkerCloud[0]<br>
[worker-13-010] Start info: Appveyor.HostAgent.BuildAgentProcess.WorkerCloud[0]<br>
[worker-13-010] Run info: Appveyor.HostAgent.BuildAgentProcess.WorkerCloud[0]<br>
[worker-13-010] Ready info: Appveyor.HostAgent.AppHost[0]<br>
Updating cloud availability (buildCloudId=13): 20 worker(s) available, 0 worker(s) busy info: Appveyor.HostAgent.BuildAgentProcess.WorkerCloud[0]<br>
[worker-13-010] Started info: Appveyor.HostAgent.BuildAgentProcess.WorkerCloud[0]<br>
[worker-13-011] Start info: Appveyor.HostAgent.BuildAgentProcess.WorkerCloud[0]<br>
[worker-13-011] Run info: Appveyor.HostAgent.BuildAgentProcess.WorkerCloud[0]<br>
[worker-13-011] Ready info: Appveyor.HostAgent.AppHost[0]<br>
Updating cloud availability (buildCloudId=13): 20 worker(s) available, 0 worker(s) busy info: Appveyor.HostAgent.BuildAgentProcess.WorkerCloud[0]<br>
[worker-13-011] Started info: Appveyor.HostAgent.AppHost[0]<br>
Checking Host Agent connection: ba8e8de6-0a4a-48df-944e-07405eb97e16 info: Appveyor.HostAgent.AppHost[0]<br>
Host Agent connection is alive warn: Appveyor.HostAgent.Services.HostAgentHubClientCore[0]<br>
Host Agent has been disconnected from AppVeyor info: Appveyor.HostAgent.Services.HostAgentHubClientCore[0]<br>
Host Agent will try to reconnect in 10 seconds. info: Appveyor.HostAgent.Services.HostAgentHubClientCore[0]<br>
Connecting to HostAgentHub... fail: Appveyor.HostAgent.Services.HostAgentHubClientCore[0]<br>
Error connecting Host Agent to AppVeyor: Operation timed out info: Appveyor.HostAgent.AppHost[0]<br>
Checking Host Agent connection: 4112a4b9-5376-4646-9967-0637e6ac331e warn: Appveyor.HostAgent.AppHost[0]<br>
Ping callback has not received in 20 seconds: 4112a4b9-5376-4646-9967-0637e6ac331e warn: Appveyor.HostAgent.AppHost[0]<br>
Host Agent connection is not responding on AppVeyor events info: Appveyor.HostAgent.AppHost[0]<br>
Stopping Host Agent connection...</p></div>Oliver Collyer