MSYS2 build times out

Tim Teulings's Avatar

Tim Teulings

23 Dec, 2015 12:02 PM

I'm running into the 1 hour timeout with my new build https://ci.appveyor.com/project/Framstag/libosmscout. The build is MSYS2 based. Time is spend for package installation, configure and mainly building. As you can see from https://travis-ci.org/ a similar build under Linux or OS X take around 8-12 minutes (and it builds more).

Though there might be differences in (virtual) hardware I'm not sure if hardware is an issue. gcc under Windows is also very slow on my local machine and thus might be the final cause.

I already tried to make use of ccache, but this actually does not seem to change anything (it does speed up builds under Linux).

There are some options on my side I could do:
* Install only required packages (mingw-w64-x86_64-toolchain installs more than required, but that will not give me too much time)
* Try to use the Qt installations already there instead of using the MingGW package (though I might get into trouble there)

I possibly also could use the caching feature to speed up package installation? Would this help? Are there other steps, where caching would help? Does anybody have actual experience?

Are there other things I could do (besides "money")?

I see, that there are not much MSYS2 packages preinstalled. I would assume that the toolchain package would be an obvious package to preinstall). However even that will only save me a few minutes...

Everything else worked like a charm. Setting up the project was simple. Good work!

  1. Support Staff 1 Posted by Feodor Fitsner on 23 Dec, 2015 10:04 PM

    Feodor Fitsner's Avatar

    Hi Tim,

    I've enabled more powerful CPU for your account - wondering if it make any faster.
    Definitely, you should consider using Qt installed on build workers. If you miss some libraries there - let me know. The same for MSYS2 - we could add more libraries for it too (of course if they don't weight dozens of GB :)

  2. 2 Posted by Tim Teulings on 26 Dec, 2015 08:23 AM

    Tim Teulings's Avatar

    Seems like he GCC build is indeed CPU bound, the build is now roughly twice as fast, taking around 32 minutes.

    I'll take a look, if I can switch to the preinstalled Qt in the next few days. I could also try to optimize the list of " to be installed MingGW/MSYS2 packages", but this will save only a few minutes.

    Regarding preinstalled packages:
    Here (https://sourceforge.net/p/msys2/wiki/MSYS2%20installation/) is sounds like base-devel is a group of packages that are likely needed.
    Here (https://sourceforge.net/p/msys2/wiki/MSYS2%20introduction/) it seems like the toolchain packages are a good start. I do not know, which of the both is the "bigger" one.

    I tried the toolchain packages because I wanted to avoid installing all build relevant packages manually, it seems like the package dependencies are rather complex :-) The toolchain packages also installs support for ada and fortran, this I/you could optimize because I do not need them and not many other will need them either ;-)

    I think, one of above should be preinstalled.

    I also saw that update-core installed new packages. This was mentioned somewhere else. MinGW seems to make regular updates. I/People of course will always want the most current stuff and will call update-core. You could of course do this in your preinstalled image, too (some cron-like stuff, that regulary creates new images). I cannot judge how complex this would be for you and if regular changes, that might bring in bugs, too, is in your interests :-/

    Installing more packages did not take that much time. Most of the installation time was required by the Qt5 package (in my case). So if you want to preinstall more besides the basic compiler toolchain, in my case this would be the package that save a lot time (5-10 minutes in the original slower build? Other packages required much less time). But this packages is likely to be big, so it is again up to you.

    Since Qt is (one of) *the* platform neutral GUI framework, it is likely that I'm not the only one using it.

    Since I'm only an open source developer not paying for your service I hesitate to "order" something, it is up to you to make decisions and I have to see how I can make my build fit :-)

  3. 3 Posted by Tim Teulings on 30 Dec, 2015 08:11 AM

    Tim Teulings's Avatar

    I have further reduced the compile time by disabling the build of static libraries in the autoconf files by default. This should half the compile time. Switching to preinstalled Qt and more selective installation of packages is still pending on my side.

    I now however have the problem, that the MinGW shows errors on install. First there was a problem with a missing PGP signature, now it seems the package DB is broken. I'm a little bit confused. Should the build not use a fresh base image all the time? So can it bee, that the database is broken?

    See: https://ci.appveyor.com/project/Framstag/libosmscout/build/job/gw44bls0qud9gdch

    As far as I remember, there are pacman calls to repair the database...

  4. Support Staff 4 Posted by Feodor Fitsner on 31 Dec, 2015 03:48 AM

    Feodor Fitsner's Avatar

    Well, it's a fresh VM all the time. Try using RDP to see what's going wrong: http://www.appveyor.com/docs/how-to/rdp-to-build-worker

  5. 5 Posted by Tim Teulings on 31 Dec, 2015 10:11 AM

    Tim Teulings's Avatar

    I restarted the build today and now everything is working again (without any change). Seems like the problems might haven be on the MSYS2/MinGW side and was fixed in the mean time.

  6. Ilya Finkelshteyn closed this discussion on 25 Aug, 2018 02:03 AM.

Comments are currently closed for this discussion. You can start a new one.

Keyboard shortcuts

Generic

? Show this help
ESC Blurs the current field

Comment Form

r Focus the comment reply box
^ + ↩ Submit the comment

You can use Command ⌘ instead of Control ^ on Mac