Much slower builds this week

Bede's Avatar

Bede

20 Dec, 2018 04:44 PM

Hi,

I'm seeing substantial sustained build speed degradation this week. Builds that reliably took between 9-10 minutes early last week are now taking upwards of 13-14 minutes, and seem to be never lower than 12 minutes. AppVeyor staff can look at the history here: https://ci.appveyor.com/project/BedeJordan/shelf/history

Is this a known and understood systemic problem that has just recently gotten worse? It seems there's a known AppVeyor capacity problem, as it is detailed in this thread from 1 month ago by AppVeyor support staff:
https://help.appveyor.com/discussions/questions/29832-did-recent-changes-apply-to-possibly-slow-down-builds

This kind of random slowdown is a big problem for our workflow. It would help if you could:
1. Acknowledge that this slowdown is occurring this week.
2. Give a firm ETA for a fix. The previous discussion I linked seems to imply the problem would be fixed in a couple of weeks, but I don't think that's true.

Note: please do not suggest I implement a build cache - this wasn't slow last week without a build cache. I also see degradation in both the CPU and I/O bound parts of my build workload, so I don't believe this will solve the issue. I've tried implementing build caches several times and they generally do not actually improve things for me - the unzip and checksum steps are so slow that they negate the savings and actually make the build slower.

Thanks in advance,
Bede

  1. Support Staff 1 Posted by Ilya Finkelshte... on 20 Dec, 2018 06:23 PM

    Ilya Finkelshteyn's Avatar

    Hi Bede,

    Sorry for the trouble. Performance issue discussed in that ticket was mitigated. I think that your issues are related to the recent image update. I see that you are using Previous Visual Studio 2017 image, probably also because of performance issues happened after this update. When image update happens, current image becomes "previous" so you are right now using the image you probably tried to avoid. Latest Visual Studio 2017 update contains a lot of fixes so, please switch to Visual Studio 2017 and let us know how it goes.

    Also please subscribe to Technical updates at https://ci.appveyor.com/notifications to avoid this kind of surprises when image change happens.

    Please let us know how it goes. And ping us as soon as you see something wrong, no need to reach the point when it becomes a big problem.

    Regarding cache, we do not actually suggest it as a way to speed up the build (only in rare special cases when dependencies need to be compiled and then cached). It is mostly used as a safety net in case nuget/npm/chocolatey are not available.

  2. Support Staff 2 Posted by Ilya Finkelshte... on 20 Dec, 2018 10:55 PM

    Ilya Finkelshteyn's Avatar

    Me again. I checked your builds and see that though you used Visual Studio 2017 image couple of times, it was still 13-14 minutes. Can you please try to open some old 9-minute build in the build history and run RE-BUILD COMMIT?

  3. 3 Posted by Bede Jordan on 20 Dec, 2018 11:55 PM

    Bede Jordan's Avatar

    Hi Ilya,

    Thanks for looking into this. There is a bunch of churn happening in our
    dependencies that is also causing some increase in time. But I just did as
    you suggested, and reran an old "fast" job. Here's the before and after:

    Before: 9:11
    https://ci.appveyor.com/project/BedeJordan/shelf/builds/20804638

    After: 10:42
    https://ci.appveyor.com/project/BedeJordan/shelf/builds/21162207

    Did a build image update occur between these two, is that the suspected
    culprit?

    Regards,
    Bede

  4. 4 Posted by Bede on 21 Dec, 2018 12:27 AM

    Bede's Avatar

    I also switched that branch to use the latest image (i.e. from Previous Visual Studio 2017 -> Visual Studio 2017).
    So, assuming that a single image update has occurred between today and 15 days ago, and we call the current image N, here are the results:

    Executed 15 days ago, N-2: 9m11s: https://ci.appveyor.com/project/BedeJordan/shelf/builds/20804638

    Executed Today, N-1: 10m42s:
    https://ci.appveyor.com/project/BedeJordan/shelf/builds/21162207

    Executed Today, N: 1 failure at 9m44s (which I've seen before when AppVeyor is overloaded/slow, Azure storage emulator fails to start, if it had succeeded there are still several minutes to run after that), 1 build complete at 13m58s!
    https://ci.appveyor.com/project/BedeJordan/shelf/history

    Not a good trend. N seems even worse than N-1 was. N-2 is no longer available as far as I know, so there's no way to try that, to control for AppVeyor system load changes between now and 15 days ago.

    Thanks,
    Bede

  5. Support Staff 5 Posted by Ilya Finkelshte... on 21 Dec, 2018 12:49 AM

    Ilya Finkelshteyn's Avatar

    Let us try to isolate environment issues by running your builds on Google cloud VMs. For that please set APPVEYOR_BUILD_WORKER_CLOUD to GCE. Note that build start will be longer -- about 3 minutes (time needed to provision GCE VM). Let us know what you observe.

  6. Ilya Finkelshteyn closed this discussion on 20 Feb, 2019 09:00 PM.

Comments are currently closed for this discussion. You can start a new one.

Keyboard shortcuts

Generic

? Show this help
ESC Blurs the current field

Comment Form

r Focus the comment reply box
^ + ↩ Submit the comment

You can use Command ⌘ instead of Control ^ on Mac

Recent Discussions

21 Aug, 2019 11:06 PM
21 Aug, 2019 02:08 PM
21 Aug, 2019 01:37 PM
21 Aug, 2019 10:21 AM
21 Aug, 2019 09:10 AM

 

21 Aug, 2019 05:56 AM
21 Aug, 2019 03:08 AM
20 Aug, 2019 11:33 PM
20 Aug, 2019 08:07 PM
20 Aug, 2019 05:50 PM
20 Aug, 2019 12:35 PM