BYOC: process runner leaves behind workdir on cancelled jobs

Jan Vesely's Avatar

Jan Vesely

31 May, 2020 01:45 AM

Using process runners running in a podman container (since podman runner is not available and docker doesn' run easily on recent rhel/fedora).
If a job is canceled, either by explicit cancel action or automatically when a more recent commit is pushed, the worker leaves behind a full workdir.
The next job of the project creates a new dir named $PROJECT-$JOB_HASH_ID.
These directories accumulate leading to a lot of wasted space on the runner.

  1. 1 Posted by Jan Vesely on 10 Jun, 2020 04:14 AM

    Jan Vesely's Avatar

    unlike https://help.appveyor.com/discussions/problems/27520-byoc-process-r... , there's no attempt to delete the build dir in the logs:

    Jun 10 04:08:07 46eca2fdaf2d appveyor-host-agent[325]: info: Appveyor.HostAgent.BuildAgentProcess.WorkerCloud[0]
    Jun 10 04:08:07 46eca2fdaf2d appveyor-host-agent[325]:       [worker-968-001] Worker 5e52362c4237430cb83c3f4399be2194 has been decommissioned. Reverting worker...
    Jun 10 04:08:07 46eca2fdaf2d appveyor-host-agent[325]: info: Appveyor.HostAgent.BuildAgentProcess.WorkerCloud[0]
    Jun 10 04:08:07 46eca2fdaf2d appveyor-host-agent[325]:       Set status: Recreate
    Jun 10 04:08:07 46eca2fdaf2d appveyor-host-agent[325]: info: Appveyor.HostAgent.BuildAgentProcess.WorkerCloud[0]
    Jun 10 04:08:07 46eca2fdaf2d appveyor-host-agent[325]:       [worker-968-001] Terminate worker process
    Jun 10 04:08:07 46eca2fdaf2d appveyor-host-agent[325]: info: Appveyor.HostAgent.BuildAgentProcess.WorkerCloud[0]
    Jun 10 04:08:07 46eca2fdaf2d appveyor-host-agent[325]:       ProcessWorker.Decommission - Killing worker process 5e52362c4237430cb83c3f4399be2194 with process ID 17537
    Jun 10 04:08:07 46eca2fdaf2d appveyor-host-agent[325]: info: Appveyor.HostAgent.BuildAgentProcess.WorkerCloud[0]
    Jun 10 04:08:07 46eca2fdaf2d appveyor-host-agent[325]:       Set status: Create
    Jun 10 04:08:07 46eca2fdaf2d appveyor-host-agent[325]: info: Appveyor.HostAgent.BuildAgentProcess.WorkerCloud[0]
    Jun 10 04:08:07 46eca2fdaf2d appveyor-host-agent[325]:       [worker-968-001] Create worker
    Jun 10 04:08:07 46eca2fdaf2d appveyor-host-agent[325]: info: Appveyor.HostAgent.BuildAgentProcess.WorkerCloud[0]
    Jun 10 04:08:07 46eca2fdaf2d appveyor-host-agent[325]:       Set status: Ready
    Jun 10 04:08:07 46eca2fdaf2d appveyor-host-agent[325]: info: Appveyor.HostAgent.AppHost[0]
    Jun 10 04:08:07 46eca2fdaf2d appveyor-host-agent[325]:       Updating cloud availability (buildCloudId=968): 5 worker(s) available, 0 worker(s) busy
    Jun 10 04:08:08 46eca2fdaf2d appveyor-host-agent[325]: info: Appveyor.HostAgent.BuildAgentProcess.WorkerCloud[0]
    Jun 10 04:08:08 46eca2fdaf2d appveyor-host-agent[325]:       [worker-968-001] Ready
    Jun 10 04:08:08 46eca2fdaf2d appveyor-host-agent[325]: info: Appveyor.HostAgent.AppHost[0]
    Jun 10 04:08:08 46eca2fdaf2d appveyor-host-agent[325]:       Updating cloud availability (buildCloudId=968): 5 worker(s) available, 0 worker(s) busy
    Jun 10 04:08:13 46eca2fdaf2d appveyor-host-agent[325]: info: Appveyor.HostAgent.BuildAgentProcess.WorkerCloud[0]
    Jun 10 04:08:13 46eca2fdaf2d appveyor-host-agent[325]:       Deleting build 'bin' directory: /tmp/av-build-33427213
    Jun 10 04:08:13 46eca2fdaf2d appveyor-host-agent[325]: appveyor-host-agent Information: 0 : sudo rm -r "/tmp/av-build-33427213"
    Jun 10 04:08:13 46eca2fdaf2d appveyor-host-agent[325]: appveyor-host-agent Information: 0 :
    Jun 10 04:08:13 46eca2fdaf2d appveyor-host-agent[325]: sudo rm -r "/tmp/av-build-33427213"
    Jun 10 04:08:13 46eca2fdaf2d sudo[18354]: appveyor : TTY=unknown ; PWD=/opt/appveyor/host-agent ; USER=root ; COMMAND=/bin/rm -r /tmp/av-build-33427213
    Jun 10 04:08:13 46eca2fdaf2d sudo[18354]: pam_unix(sudo:session): session opened for user root by (uid=0)
    Jun 10 04:08:13 46eca2fdaf2d sudo[18354]: pam_unix(sudo:session): session closed for user root
    Jun 10 04:08:25 46eca2fdaf2d appveyor-host-agent[325]: info: Appveyor.HostAgent.AppHost[0]
    Jun 10 04:08:25 46eca2fdaf2d appveyor-host-agent[325]:       Checking Host Agent connection: d8457800-7c9f-4563-ac1f-920924b16b56
    

Reply to this discussion

Internal reply

Formatting help / Preview (switch to plain text) No formatting (switch to Markdown)

Attaching KB article:

»

Attached Files

You can attach files up to 10MB

If you don't have an account yet, we need to confirm you're human and not a machine trying to post spam.

Keyboard shortcuts

Generic

? Show this help
ESC Blurs the current field

Comment Form

r Focus the comment reply box
^ + ↩ Submit the comment

You can use Command ⌘ instead of Control ^ on Mac