Home → Suggestions →

Support for MSYS2

• Edit

Someone

24 Mar, 2015 08:28 PM

MSYS2 is awesome, it makes building Windows software much easier. Pidgin++ for example mostly eliminated the need for this thanks to MSYS2: https://developer.pidgin.im/wiki/BuildingWinPidgin.

1. 1 Posted by David Macek on 24 Mar, 2015 08:42 PM

   

   +1 for this.

   MSYS2 provides up-to-date mingw-w64 toolchains with a lot of library packages available plus supports autotools, make and other GNU tools. Updating the system, toolchains and libraries is also very easy, it's a three step process at worst.

   • Edit

2. Support Staff 2 Posted by Feodor Fitsner on 25 Mar, 2015 12:27 PM

   

   ...and all that goodness could be installed with one of two installers on this page? Btw, which one would you need: 32- or 64-bit?

   • Edit

3. 3 Posted by David Macek on 25 Mar, 2015 12:49 PM

   

   The 64-bit version is preferred, and allows compiling both 32-bit and 64-bit Windows applications. The installer gets you the basics, including the package manager. The build tools and compilers can be installed with pacman -S base-devel mingw-w64-i686-toolchain mingw-w64-x86_64-toolchain (run inside MSYS2 Shell).

   Users will then probably want to use the package manager to download the libraries they depend on (*), which you can also pre-install, but that depends on how much package installation/updating you want to off-load to your users.

   *) e.g. mingw-w64-{i686,x86_64}-openssl, mingw-w64-{i686,x86_64}-gtk3

   • Edit

4. 4 Posted by Ebrahim on 05 May, 2015 08:58 PM

   

   Yes, please make this happen. I wanted to add support of appveyor CI to an important cross library that just on msys2 it can be easily compiled and tested on Windows (because it has some dependencies to other libraries which can not easily done via mingw or cygwin). Installing via script is hard (yes I've tried the approach) and probably won't be accepted on upstream I am trying add the appveyor on, so having this would help a lot.

   > ...and all that goodness could be installed with one of two installers on this page? Btw, which one would you need: 32- or 64-bit?

   Both are equal and 64 is preferred.

   • Edit

5. 5 Posted by Ebrahim on 02 Jun, 2015 11:09 PM

   

   Just for the record, https://github.com/behdad/harfbuzz/pull/112 and https://github.com/khaledhosny/ots/pull/67

   • Edit

6. 6 Posted by kunda on 02 Jun, 2015 11:11 PM

   

   +1 on this

   • Edit

7. 7 Posted by Drew Wells on 14 Jul, 2015 07:41 PM

   

   WoW this is awesome. Thanks for sharing those useful pull requests on how to implement with msys. Glad I can finally use make on appveyor now :)

   • Edit

8. 8 Posted by Renato on 04 Aug, 2015 03:59 PM

   

   Is there any news on this topic? I based myself on these very useful pull requests to use MSYS2 in Appveyor, but my builds fail when I try to download a package with pacman and the VM connection slows down for more than 10 sec. Although I have tried different solutions (using curl in XferCommand option in pacman.conf), having MSYS2 installed in Appveyor would eliminate these tricks. And, sure, not downloading the most common packages would save lots of time in the build.

   • Edit

9. Support Staff 9 Posted by Feodor Fitsner on 04 Aug, 2015 04:03 PM

   

   Can you drop download link to the library you require?

   - Feodor

   • Edit

10. 10 Posted by Renato on 05 Aug, 2015 12:48 AM

    

    I believe the ideia is the same as discussed by David and Ebrahim. To install MSYS2, I'm using:

    appveyor DownloadFile "http://downloads.sourceforge.net/project/msys2/Base/i686/msys2-base-i686-20150512.tar.xz" -FileName "msys2.tar.xz"
    7z x msys2.tar.xz
    7z x msys2.tar > NUL
    

    I used the 32 bit version (i686 in the SourceForge path) because I didn't specify the bits in my appveyor.yml. The 64 bits version, however, would be prefered for this platform.

    Then installing the packages through pacman:
    

    C:\msys32\usr\bin\bash -lc "pacman -S make bc base-devel mingw-w64-i686-toolchain"
    

    Would give MSYS2 version of gcc / g++ / gfortran installed. If they were right available with the VM, it would spare me the download of pacman's packages.

    • Edit

11. Support Staff 11 Posted by Feodor Fitsner on 05 Aug, 2015 04:36 AM

    

    OK, thanks. So, as I understand this is the thing that should be installed on AppVeyor: http://sourceforge.net/projects/msys2/files/Base/x86_64/msys2-base-... ?

    Presumably, C:\msys32 is the directory where it should be unpacked to? In our case it's probably C:\msys. Can you please share a link to your build to use it as sample?

    • Edit

12. 12 Posted by Andrew Chadwick on 05 Aug, 2015 12:18 PM

    

    Very interested in seeing this on AppVeyor - it'd be brilliant to see some Windows CI for https://github.com/mypaint/mypaint :D

    Feodor: that's just the os-specific tree for making 64-bit targets. MSYS2 needs three environments. It would be better to install everything the installer at https://msys2.github.io/ puts there. The structure that msys2-x86_64-20150512.exe installs is what we expect:

    • C:\msys64\ - the root of paths that POSIX-style stuff expects. "/" from bash's perspective.
      • mingw32\{bin,usr,lib,share,...} - toolchain for MINGW32 target (native Win32 binaries, no cygwin)
      • mingw64\{bin,usr,lib,share,...} - toolchain for MINGW64 target (native Win64 binaries, no cygwin)
      • {bin,usr,lib,share,...} - cygwinnish base-devel toolchain and helpers; any compilers in here build for the MSYS target.

    [**EDIT**: fix markdown layout]

    Note that msys2 are in the process of moving their repos and installer releases away from sourceforge. The new package repository for the installed pacman is http://repo.msys2.org/, configured in /etc/pacman.d/mirrorlist.*=C:\mingw64\etc\pacman.d\mirrorlist.*.

    Note the path equivalence. Shell scripts and makefiles written for msys2 will be fairly independent of where you choose to install it as a result. Conversely, Windows drive letter paths are accessible from POSIXland as /c/Windows/System32 etc.

    When we're dealing with msys2 in our build scripting, we make heavy use of the MSYSTEM environment variable which switches between the three environments above. The msys2 guys switch between environments based on this setting early on in their bash startup too. When invoking a POSIX/cygwin Bourne-compatible shell like bash as above, the installed /etc/profile (= C:\msys64\etc\profile) sets up PATH, ACLOCAL_PATH and other goodies based on the target name in MSYSTEM. You'll end up with shared tools like make, aclocal or scons being /usr/bin/make for all targets, but target-specific tools like gcc being either /mingw32/bin/gcc or /mingw64/bin/gcc depending on the context.

    It is very worth looking at what the installed C:\msys64\mingw32_shell.bat and its msys2_shell and mingw64_shell buddies do.

    I'm very new to AppVeyor, and I haven't been able to get the installer to operate in silent mode just yet (unfamiliarity with QT Installer Framework, and the fact that I really don't get on with QT-IFW!)

    If you want to see a potential project that might be a good fit for AppVeyor, take a look at our MyPaint's windows/ folder for scripts we've written against local msys2 installs.

    If AppVeyor want to support this kind of stuff, mirroring pacman's packages to its cache regularly would save on network traffic I guess. But I don't know yet how that would work with your infra.

    • Edit

13. 13 Posted by Ebrahim on 05 Aug, 2015 12:35 PM

    

    I completely agree with Andrew Chadwick. Thank you for the detailed documentation.

    • Edit

14. 14 Posted by fracting on 05 Aug, 2015 03:35 PM

    

    I also tried msys2 in Appveyor just today, but I got some error:

    mkdir: cannot change permissions of '/dev/shm': Permission denied
    mkdir: cannot change permissions of '/dev/mqueue': Permission denied
    ln: failed to create symbolic link '/dev/stdin': No such file or directory

    Link: https://ci.appveyor.com/project/fracting/xz/build/1.0.17

    Anyone has idea what's wrong? Thanks a lot!

    BTW, someone might interesting in msys2 in Travis CI <grin>
    It is not stable yet, but fun enough to play with:
    https://github.com/fracting/wine-fracting/wiki/MSYS2-on-Wine#build-status

    Cheers!

    • Edit

15. 15 Posted by Someone on 05 Aug, 2015 08:54 PM

    

    Feodor, think of MSYS2 like a bundle of:

    • A modern MSYS, up-to-date with Cygwin
    • A distribution of MinGW-w64 toolchains
    • Many software packages available

    These three elements put together make building Windows software easier. Think of you not needing to follow the mentioned Pidgin wiki instructions just because you can install everything easily with pacman. Once MSYS2 is supported, adding Pidgin++ to AppVeyor should be nearly trivial, for instance.

    You people will probably want to tweak the MSYS2 installation for the AppVeyor needs. For example, I mount /home at C:\Users for convenience. But here are the basic steps:

    1. Execute the 64-bit installer, which will install a 64-bit MSYS.
       
    2. Update the system with update-core, restart and pacman -Su (details).
       
    3. Install as many popular packages as you want, for example native 32/64-bit Windows GCC and basic development tools:

    pacman --sync mingw-w64-x86_64-gcc
    pacman --sync mingw-w64-i686-gcc
    pacman --sync base-devel
    

    Then you can keep updating the system once in a while, and add pre-installed libraries as requested by users. You can talk with MSYS2 people in the mailing list or the #msys2 IRC channel.

    • Edit

16. 16 Posted by Ray Donnelly on 05 Aug, 2015 09:25 PM

    

    Andrew Chadwick: Regarding Qt IFW and commandline driven installation .. it's just not ready for users yet. I fixed the crash bugs, and I can install it from a script, but that script needs to be embedded into the installer as it's a bit too big to remember or ask the user to type:

    https://github.com/Alexpux/MSYS2-packages/blob/master/msys2-installer/auto-install.js

    .. unfortunately, the --script option doesn't allow you to provide a complete script at top-level scope, but instead nests your script inside a JavaScript function making it impossible to include (import or whatever the correct JavaScript equivalent is) the auto-install.js script.

    I should add that, until I get round to looking at this again, you can always install MSYS2 simply by untarring one of the release tarballs:

    http://sourceforge.net/projects/msys2/files/Base/x86_64/msys2-base-x86_64-20150512.tar.xz/download
    http://sourceforge.net/projects/msys2/files/Base/i686/msys2-base-i686-20150512.tar.xz/download

    Also, the idea that AppVeyor could mirror our repositories is a good one. It's very easy to do. You just need to rsync them. We're looking at moving them to bintray at present as we've had connectivity (and worse) issues with SourceForge lately. They're currently hosted at repo.msys2.org on a graciously donated server, but we're not sure if that's going to work out.

    • Edit

17. 17 Posted by Ray Donnelly on 21 Aug, 2015 11:14 AM

    

    Is it possible to get an update on the status of this request please?

    • Edit

18. Support Staff 18 Posted by Feodor Fitsner on 21 Aug, 2015 03:04 PM

    

    Sorry for the delay with that, but honestly I'm stuck by wondering what would be correct installation procedure for MSYS2 that works for all of you guys. Seems we have a number of recipes here.

    Can we make it clear that please?

    • Edit

19. 19 Posted by Ray Donnelly on 07 Sep, 2015 12:47 AM

    

    Hi Feodor,

    Since I don't know too much about how AppVeyor works it's a bit tricky for me to comment, but as best I can figure, the issues are:

    As MSYS2 is a rolling release software distribution, I'm not sure what would be best for AppVeyor and your customers / clients. We make new base-system installers and tarballs (which, once installed are functionally equivalent) about 3-4 times a year.

    We constantly release new signed binaries as upstream projects release new versions or as we fix bugs reported by our users.

    So at points, the base-system can be 4-5 months behind the latest and greatest, and most packages will be out of date by that time.

    You could leave it up to users to update and install the packages they need. The downside of having the users do it is that you'll find yourselves / your clients [1] downloading and [2] installing the same packages over and over again.

    [1] Pacman stores a cache of packages in C:/msys64/var/cache/pacman/pkg so to mitigate the bandwidth issues would it be possible to have that on a shared drive?

    [2] You could update the MSYS2 base-system (with some frequently used packages?) on your system image periodically (e.g. from cmd.exe):
    ```C:\msys64\usr\bin\sh.exe --login -c 'C:/msys64/usr/bin/update-core' && C:\msys64\usr\bin\sh.exe --login -c 'C:/msys64/usr/bin/pacman -Syu'```

    Hopefully I've not muddied things too badly! It'd be really great to see MSYS2 on AppVeyor happen.

    • Edit

20. 20 Posted by Andrew Chadwick on 07 Sep, 2015 08:11 AM

    

    To address the problem with [1], a caching web proxy would seem a good measure. I assume that AppVeyor already has something similar between its build VMs and the outside world. I'm reasonably certain that it's just http/https and package files with sensible date/time semantics.

    • Edit

21. Support Staff 21 Posted by Feodor Fitsner on 07 Sep, 2015 07:31 PM

    

    OK, thanks for your notes, guys.

    I'm going to install MSYS2 per these instructions: https://github.com/appveyor/ci/issues/352#issuecomment-138149606

    Please let me know if I miss something.

    • Edit

22. 22 Posted by maiksensi on 23 Nov, 2015 10:46 AM

    

    I am not sure if this is related, but we are facing a strange mingw error with git.

    Cloning into 'cookiecutter-pypackage'...
    fatal: unable to access 'https://github.com/audreyr/cookiecutter-pypackage.git/': error setting certificate verify locations:
      CAfile: C:/Program Files/Git/mingw64/libexec/ssl/certs/ca-bundle.crt
    

    I havent done extended testing on this, but I guess it is something related to the installation of mingw and git's default settings. (This cloning is done in an virtualenvironment created with Tox).

    • Edit

23. 23 Posted by David Macek on 23 Nov, 2015 11:29 AM

    

    Seems unrelated.

    Anyway, check that you're using C:/Program Files/Git/cmd/git.exe and not any other git. Then check the value of git configuration options http.sslCAinfo and http.sslCAPath. http.sslCAinfo should be set to /ssl/certs/ca-bundle.crt.

    • Edit

24. 24 Posted by maiksensi on 23 Nov, 2015 12:26 PM

    

    Thanks! Well, I think it is because there are several git Versions installed (see installed software). I guess in a new shell the git executable path points to the old version, where the path seems to be different. Whatever, we fixed it by changing the system wide http.sslcainfo path to the correct one.

    • Edit

25. 25 Posted by Someone on 04 Jan, 2016 07:50 PM

    

    Feodor, thanks for adding MSYS2 to AppVeyor. Is it possible to create a wiki page or something where people could suggest new pre-installed packages? Sometimes package installation takes too long and is subject to network timeouts. The same applies for system updates, so could you add some scheduled task to update the installation once in a while? The gist of it would be like:

    update-core
    # terminate all MSYS2 processes
    pacman -Syu
    

    • Edit

26. 26 Posted by Someone on 12 Jan, 2016 01:45 AM

    

    Actually, I think that is not going to work because at the moment update-core is interactive-mode only. So can you please at least run a manual update with the steps below? Current installation is quite outdated already.

    # Switch from SF to msys2.org (default, much faster)
    pacman --sync pacman-mirrors
    
    # Restart MSYS2 after this
    update-core
    
    # Should take a while
    pacman -Syu
    

    • Edit

27. Support Staff 27 Posted by Feodor Fitsner on 12 Jan, 2016 02:30 AM

    

    I've added a new issue: https://github.com/appveyor/ci/issues/597

    • Edit

28. 28 Posted by ryan.prichard on 04 Apr, 2016 10:53 PM

    

    Is there some way I could get a 32-bit MSYS2 environment?

    I need the i686-pc-msys-g++ compiler to produce artifacts like winpty-0.2.2-msys2-2.4.1-ia32.tar.gz at https://github.com/rprichard/winpty/releases/tag/0.2.2. I know that 64-bit MSYS2 can cross-compile to either architecture for native Windows, but I'm targeting the MSYS2/Cygwin environment itself, and AFAICT, there's no way to cross-compile from 64-bit MSYS2 to 32-bit MSYS2.

    • Edit

29. Support Staff 29 Posted by Feodor Fitsner on 04 Apr, 2016 11:21 PM

    

    Will look into deploying it on build workers.

    Does installation procedure for MSYS 2 look correct if applied to i686 installer: https://github.com/appveyor/ci/issues/597? Any packages missing?

    • Edit

30. 30 Posted by ryan.prichard on 05 Apr, 2016 01:07 AM

    

    Thanks for looking into it. The package list looks OK to me for 32-bit MSYS2, with the exception of gyp-svn, which seems to have been renamed to gyp-git (for 32-bit and 64-bit). Otherwise, everything installed, and the C++ compiler seems to work.

    • Edit