Java - Windows build takes over 1 hour and fails, Linux build takes 20 mins

Adam Retter's Avatar

Adam Retter

15 May, 2019 11:03 AM

Hi there,

We have a pretty standard large Java project which is built with Maven. On your Ubuntu Linux instances the project builds in ~20 mins quite successfully. Also on Travis this builds in approx ~20 mins on Linux and Mac - https://travis-ci.com/eXist-db/exist/builds/111856227

However, on your Windows instances, the build takes over 1 hour and then is killed for exceeding the time-limit.

As we are using Maven, each module in our build reports the elapsed time that it took to run. However, even when I request the full log from the AppVeyor Dashboard, I am still be presenting with a truncated log :-( So I cannot work out which stages are so much slower on Windows. Our build is here - https://ci.appveyor.com/project/AdamRetter/exist/builds/24554085

Could someone provide us with the full log, or an explanation for why Windows on AppvVeyor is so much slower than Linux on AppVeyor?

At a guess, it seems Windows is not correctly caching the .m2 directory, whereas the Linux builds are.

  1. Support Staff 1 Posted by Owen McDonnell on 15 May, 2019 03:22 PM

    Owen McDonnell's Avatar

    We truncate logs when they exceed 8MB in size.

    Can you try a build with maven set to quiet logging mode, so we can at least see what happens before that command.

    Depending on the data center of the build and the number of builds since cache was implemented, it could be that cache is restored on one build but not on another.

  2. 2 Posted by Adam Retter on 15 May, 2019 03:38 PM

    Adam Retter's Avatar

    Okay just pushed that `-q` for Maven. Let's see what happens...

  3. 3 Posted by adam.retter on 15 May, 2019 07:56 PM

    adam.retter's Avatar

    Owen, the build has run now, but I'm still no clearer on the issue. Any ideas?

  4. Support Staff 4 Posted by Owen McDonnell on 15 May, 2019 09:40 PM

    Owen McDonnell's Avatar

    I see that there is a simple spelling error in the cache section of your appveyor.yml.

    C:\Users\appveryor\.m2
    

    should be

    C:\Users\appveyor.m2
    

    Fix that, run a few builds and see if caching brings the windows build more in line with ubuntu build.

  5. 5 Posted by Adam Retter on 16 May, 2019 03:07 PM

    Adam Retter's Avatar

    Hi Owen.

    All I can say is... Doh! Thanks for spotting that.

    Sadly... it doesn't seem to have fixed the problem though - https://ci.appveyor.com/project/AdamRetter/exist/builds/24589531
    I changed the AppVeyor config like so:

    ```
    cache:
      - '%USERPROFILE%\.m2'
      - '$HOME/.m2'
    ```

    I also have APPVEYOR_SAVE_CACHE_ON_ERROR true set in AppVeyor settings.

    Unfortunately it still does not seem to cache at all on Windows :-(

  6. Support Staff 6 Posted by Owen McDonnell on 19 May, 2019 03:54 AM

    Owen McDonnell's Avatar

    So, i did some testing on a fork (some fairly random, some less so). At first I thought it may just be a pretty intensive build and so i changed to a more powerful build cloud and extended the time, but i still hit the same hanging on the maven test call.

    I tried some other things like changing the thread count to -T 1C and one of those jobs made it further but fell down on tests that rely on some xml file (macbeth.xml i believe). I deleted that test and another test failed on that file as well so i gave up on that (though i guess you may have some path error to that file).

    Do you have this building successfully on a local windows machine, and if so what are the specs of that machine?

    Also, I want to make sure you're aware of the possibility of debugging via RDP session on the build VM.

    Finally, caching was working on my builds, but only for builds that ended in an error. Caching doesn't happen on builds that timeout.

  7. 7 Posted by Adam Retter on 20 May, 2019 02:06 PM

    Adam Retter's Avatar

    Hi Owen,

    I have the build completing on Windows here running in a VirtualBox VM. Specs are:

    Microsoft Windows 10 Pro N
    10.0.17134
    4GB RAM
    2vCPU

    Oracle Java
    1.8.0_144-b01

    I run a very similar command to our AppVeyor.yml config, i.e. - `mvn -T 2C test`
    Three small tests are failing, and I am just trying to fix those, but either way the build does complete.

    I will take a look at the RDP option, to see if I can figure out what is hanging the build.

    Thanks, I will report back... :-)

  8. 8 Posted by Adam Retter on 21 May, 2019 06:37 AM

    Adam Retter's Avatar

    Owen, when a build times-out does it invalidate the cache?

  9. 9 Posted by Adam Retter on 21 May, 2019 08:38 AM

    Adam Retter's Avatar

    Owen, I was able to login with RDP and obtain Java stack traces, which show are build is hanging due to an issue in the Quartz Scheduler Library that we use - https://github.com/quartz-scheduler/quartz/issues/448

    I guess the reason it hangs on AppVeyor but not on my VM is due to timing issues when running tests in parallel.

    Thanks for your help.

  10. Support Staff 10 Posted by Owen McDonnell on 21 May, 2019 03:24 PM

    Owen McDonnell's Avatar

    Right, I did get some different results when i changed the command to mvn -T 1C test.
    Though i'm still surprised this parallel testing completes on the low powered VirtualBox you run it in.

    Build timeouts will not invalidate the cache.

Reply to this discussion

Internal reply

Formatting help / Preview (switch to plain text) No formatting (switch to Markdown)

Attaching KB article:

»

Attached Files

You can attach files up to 10MB

If you don't have an account yet, we need to confirm you're human and not a machine trying to post spam.

Keyboard shortcuts

Generic

? Show this help
ESC Blurs the current field

Comment Form

r Focus the comment reply box
^ + ↩ Submit the comment

You can use Command ⌘ instead of Control ^ on Mac