Build cache not always retrieved

madmax's Avatar

madmax

09 Aug, 2017 09:29 AM

One of our projects seems to be having issues getting the build cache. Just made a commit on GitHub and the build failed because it did not restore the build cache? Manually restarted a new build and it restored it just fine?
Project: https://ci.appveyor.com/project/MaNGOS/server-wtbhv
Failed Build: https://ci.appveyor.com/project/MaNGOS/server-wtbhv/build/0.21.238

Build after this restored the cache just fine. Is there an issue somewhere?

  1. 1 Posted by richard.dubay on 09 Aug, 2017 01:45 PM

    richard.dubay's Avatar

    Hey, @madmax! I've been having pretty much the same problem yesterday and today, only that no matter how many times I build/rebuild, cache is never restored. For reference, all our builds were fine last week. I'll keep watching this to see if you figure it out, but wanted to give you my +1 for this issue!

  2. 2 Posted by richard.dubay on 09 Aug, 2017 03:52 PM

    richard.dubay's Avatar

    And just like that, the build caches started working again. ¯_(ツ)_/¯

  3. Support Staff 3 Posted by Ilya Finkelshte... on 09 Aug, 2017 08:07 PM

    Ilya Finkelshteyn's Avatar

    We are expanding to new datacenter, which has its own cache storage. If your build happens to be executed in that datacenter, first time it is nothing to restore, but after successful build cache is saved in hew datacenter for you as well. If you do not want to depend on the build success to save cache, you can set APPVEYOR_SAVE_CACHE_ON_ERROR environment variable to true.

  4. madmax closed this discussion on 10 Aug, 2017 09:34 PM.

  5. madmax re-opened this discussion on 13 Aug, 2017 01:08 PM

  6. 4 Posted by madmax on 13 Aug, 2017 01:08 PM

    madmax's Avatar

    This issue is really bad for this project https://ci.appveyor.com/project/MaNGOS/server-80qcn/

    I've restarted about 30times just now and not getting a cache and RDP is not always working. (fails to connect).

  7. 5 Posted by madmax on 13 Aug, 2017 01:18 PM

    madmax's Avatar

    This is completely broken for us, i have sat here for 25minutes attempting to rebuild pull request on this project https://ci.appveyor.com/project/MaNGOS/server-80qcn
    will need a member of staff to take a look for us.
    I cannot even get into remote desktop, anything on the 67.x.x.x IP range just times out.

  8. 6 Posted by madmax on 13 Aug, 2017 01:35 PM

    madmax's Avatar

    It looks like after around 40 attempts the build cache was able to restore. Remote desktop is still very flakey.
    The time i did get in the o/s asked me if i wanted to discover computers on the local network?!

    I wonder if the servers in the new datacenter are opening the right ports for remote desktop?

  9. Support Staff 7 Posted by Ilya Finkelshte... on 15 Aug, 2017 01:40 AM

    Ilya Finkelshteyn's Avatar

    This is definitely not right, we fixed cache issue. We identified issue with RDP, should be fiuxed soon. For than nasty "Networks" dialog we have this issue. However it should not affect you, if you are not using UI testing scenarios. Sorry for the trouble.

  10. 8 Posted by madmax on 15 Aug, 2017 11:58 AM

    madmax's Avatar

    Hi,

    This morning it took me 6 attempts to get the build cache to restore! Did you say this is already fixed or a fix is coming?

    You can see each of the builds, i manually use the rebuild pull request option and cancel when i see no cache restored, repeat until it does.

    https://ci.appveyor.com/project/MaNGOS/server-80qcn/build/0.21.205
    https://ci.appveyor.com/project/MaNGOS/server-80qcn/build/0.21.206
    https://ci.appveyor.com/project/MaNGOS/server-80qcn/build/0.21.207
    https://ci.appveyor.com/project/MaNGOS/server-80qcn/build/0.21.208
    https://ci.appveyor.com/project/MaNGOS/server-80qcn/build/0.21.209
    This one worked?! https://ci.appveyor.com/project/MaNGOS/server-80qcn/build/0.21.210

  11. Support Staff 9 Posted by Ilya Finkelshte... on 16 Aug, 2017 01:56 AM

    Ilya Finkelshteyn's Avatar

    @madmax problem is that cache saving is disabled on Pull Request (to prevent outsiders from spoiling the cache). If non-pr build happens to run in new environment, then cache is saved. I filed this GitHub issue and will try to fix it soon.

  12. 10 Posted by madmax on 16 Aug, 2017 07:07 AM

    madmax's Avatar

    So it sounds like the cache should restore on pull requests and instead add an option to ignore updating/saving the cache on pull requests? That way a PR could do what it likes to the cache files but they would be safe.

  13. Support Staff 11 Posted by Ilya Finkelshte... on 17 Aug, 2017 12:25 AM

    Ilya Finkelshteyn's Avatar

    Cache is being restored in pull request. But as long as build happens in new datacenter it is nothing to restore. After we add saving cache on pull requests (soon), after single PR build save cache in new datacenter, subsequent ones have what to restore.

  14. 12 Posted by madmax on 10 Jan, 2018 10:57 PM

    madmax's Avatar

    Hey, We are still getting issues with this, you can see from our build history that only some of the time the cache gets restored, this is even with just normal commits. Example: https://ci.appveyor.com/project/MaNGOS/server-wtbhv

  15. Support Staff 13 Posted by Ilya Finkelshte... on 12 Jan, 2018 03:21 AM

    Ilya Finkelshteyn's Avatar

    @madmax seems that your project cache is corrupted in one of our datacenters. You need to delete the cache using this API (samples). Unfortunately it will delete cache in the both datacenters so it will take a several builds to repopulate it everywhere. Sorry for the trouble.

    By the way we implemented option to save cache in PR builds, you can set it in UI, General settings (Save build cache in Pull Requests).

  16. 14 Posted by madmax on 12 Jan, 2018 09:02 PM

    madmax's Avatar

    Hey, I tried this on our main project - https://ci.appveyor.com/project/MaNGOS/server-9fytl
    Just before I ran build 0.21.989 I used a powershell script to wipe the cache. Then during this build i used remote desktop to set the mysql files we want and let it continue which you can see it saved the cache files at the end.
    You can see on each build history if the cache restores or not. We had several failures followed by 5 or so sucessful restores of the cache, then the cache was not restored, in fact it does not even mention cache restoring in build 0.21.998.

    Can you look into this some more for us and see what the issue is?

    Thanks :)

  17. Support Staff 15 Posted by Ilya Finkelshte... on 13 Jan, 2018 02:48 AM

    Ilya Finkelshteyn's Avatar

    Build 0.21.989 happened in one datacenter, so cache saved in that datacenter cache server, but build 0.21.998 happened in another datacenter where in never saved.

    Generally it is not supposed to work this way. Cache (like generally any other cache in other systems) provides high speed access to some data, but it never the only way to get that data. If cache is broken it should always be some way to re-populate it.

    In your case populating cache manually over RDP is very fragile and unsupported scenario. We cannot guarantee that cache will be always available. We can replace cache server, or open new datacenter location with empty cache server. You need to write a script which checks cached folders and if they are empty or contains stale data, download that data from some location again. There are a lot of ways to download files on AppVeyor.

    Usually cache folders used by package managers (nuget, npm, etc.) and those package managers implement logic described above. But in this case you need to implement that logic yourself.

  18. 16 Posted by madmax on 13 Jan, 2018 04:08 PM

    madmax's Avatar

    Hey, So I think I figured what I need to do in powershell... Hoping my developers can figure out how to script this (I'm not a dev).

    OK so heres what we need to do using powershell...

    Use Test-Path "C:\Program Files (x86)\MySql\MySQL Server 5.7" -pathType container
    If this folder does not exist, we need to:
    Invoke-WebRequest "http://assets.getmangos.eu/mysql-5.7.20-win32.zip" -OutFile c:\mysql.zip

    Then run:
    7z -bso0 -bsp0 e -y c:\mysql.zip lib -o"C:\Program Files (x86)\MySql\MySQL Server 5.7\lib"
    7z -bso0 -bsp0 e -y c:\mysql.zip include -o"C:\Program Files (x86)\MySql\MySQL Server 5.7\include"

    But those only need to happen if the first comes back false (i.e the test path is false).

    This way we will not be reliant on the cache if it does not restore :)

  19. 17 Posted by madmax on 13 Jan, 2018 06:57 PM

    madmax's Avatar

    We got this scripted now, if the cache does not restore we will download from our server and extract. :)

  20. madmax closed this discussion on 13 Jan, 2018 06:57 PM.

Comments are currently closed for this discussion. You can start a new one.

Keyboard shortcuts

Generic

? Show this help
ESC Blurs the current field

Comment Form

r Focus the comment reply box
^ + ↩ Submit the comment

You can use Command ⌘ instead of Control ^ on Mac