Based on our experience (we use Azure as a backup cloud for VS 2019 image), compared to other clouds, Azure could be absolutely horrible in locking resources! That's why we implemented heavy retry logic specifically for Azure cloud provider. You can see the number of retries and interval between retries under Provisioning settings section on Azure cloud settings. Try increasing the number of retries and the interval between them, say 20 mins. Although periodic checks of resource group with AppVeyor builds still won't hurt.
Also, enable "Use managed disk" option - it will be provisioning Azure managed disks instead of copying blobs which could be also more stable.
While you are observing the storage with new settings in place if you notice something is not removed please tell us VM name and build date to trace the logs and see what happened. It's absolutely possible that our removal logic is not ideal and could be improved!