MSYS build on VS 2019 image is twice as slow as on VS 2017 one

belegdol's Avatar

belegdol

17 Oct, 2019 05:37 AM

Hello,
as stated in topic, the MSYS builds on VS2019 image are almost twice as slow as VS2017:
https://ci.appveyor.com/project/belegdol/mame/builds/28163993
VS2017: 35 min, including huge pacman update
VS2019: 57 min
VS2019 builds are in fact so slow that they regularly go over the default 60 minutes. While the timeout for my fork was bumped to 90 minutes [1], this issue still affects upstream (startaq/mame).
Is such a huge performance downgrade expected here?

[1] https://help.appveyor.com/discussions/questions/41921-appveyor-builds-more-than-twice-as-slow-as-travis

  1. Support Staff 1 Posted by Feodor Fitsner on 17 Oct, 2019 04:32 PM

    Feodor Fitsner's Avatar

    My guess would be MSYS version difference between VS 2017 and VS 2019 images. Currently, VS 2019 image is being built from scratch with always the latest MSYS pulled at the moment of building. MSYS on VS 2017 though wasn't updated for some time now. Could it be the case?

  2. 2 Posted by belegdol on 17 Oct, 2019 04:52 PM

    belegdol's Avatar

    I don't think it is down to MSYS version difference. While VS 2017 starts with much older MSYS, I have two
    bash -lc "pacman -Syu --noconfirm"
    calls in my .appveyor.yml, which effectively updates everything to the latest status.

  3. Support Staff 3 Posted by Feodor Fitsner on 17 Oct, 2019 04:54 PM

    Feodor Fitsner's Avatar

    OK, could base OS be a problem? VS 2017 is on Windows Server 2016 while VS 2019 image is on Windows Server 2019.

  4. 4 Posted by belegdol on 17 Oct, 2019 04:58 PM

    belegdol's Avatar

    The OS version difference could very well be the reason. Is this something you might check from appveyor end? Nearly 50% performance drop is quite something.

  5. Support Staff 5 Posted by Feodor Fitsner on 17 Oct, 2019 05:01 PM

    Feodor Fitsner's Avatar

    Would be great having some sort of an easy repro, kind of something doing in a loop X times, if that possible at all? I don't think VS 2019 VMs are generally slower than VS 2016 as it's the same hardware, but there could be some particular operation that is slower.

  6. 6 Posted by belegdol on 17 Oct, 2019 05:07 PM

    belegdol's Avatar

    You can repro it by building mame (MINGW variant) from belegdol/mame appveyor-debug branch, but this does take a while. The builds are consistently slower. If you look at
    https://ci.appveyor.com/project/startaq/mame/history
    almost none of the builds over the last few days succeeded due to hitting 60 minutes.
    On the other hand, the few builds from the branch still carrying the old .appveyor.yml all finished within 35 minutes or so:
    https://ci.appveyor.com/project/startaq/mame/builds/28137399
    https://ci.appveyor.com/project/startaq/mame/builds/28138349
    https://ci.appveyor.com/project/startaq/mame/builds/28131126
    https://ci.appveyor.com/project/startaq/mame/builds/28131126

  7. Support Staff 7 Posted by Feodor Fitsner on 18 Oct, 2019 05:24 PM

    Feodor Fitsner's Avatar

    We've updated Visual Studio 2019 image with some additional optimizations. It's faster now, at least we match the performance of VS 2017 image for now.

    You can give it another try and let me know how it worked.

  8. 8 Posted by belegdol on 18 Oct, 2019 09:01 PM

    belegdol's Avatar

    Thank you for looking into this! It doesn't appear that much faster unfortunately, only 3 minutes:
    https://ci.appveyor.com/project/belegdol/mame/builds/28215654
    2017 build was 10 minutes slower than last time but still almost 10 minutes faster than 2019.
    Are the fixes you mentioned rolled out already? The latest update to 2019 image mentioned on https://www.appveyor.com/updates/ was on 13 October.

  9. Support Staff 9 Posted by Feodor Fitsner on 18 Oct, 2019 09:07 PM

    Feodor Fitsner's Avatar

    There is no update record for that changes (yet). The change was mostly around disabling a bunch of services.

    We will continue investigating what is slowing down your project on Visual Studio 2019 image. It's interesting that we are getting an approximately the same time when running this image on Hyper-V or Azure. That could mean it's not the problem in hypervisor.

Reply to this discussion

Internal reply

Formatting help / Preview (switch to plain text) No formatting (switch to Markdown)

Attaching KB article:

»

Attached Files

You can attach files up to 10MB

If you don't have an account yet, we need to confirm you're human and not a machine trying to post spam.

Keyboard shortcuts

Generic

? Show this help
ESC Blurs the current field

Comment Form

r Focus the comment reply box
^ + ↩ Submit the comment

You can use Command ⌘ instead of Control ^ on Mac