Home → Problems →

git-lfs and build cache

• Edit

sdawkins

16 Mar, 2017 01:32 PM

Hi,

I am finding that appveyor is burning through my git-lfs bandwidth quota pretty quickly, as each build is on a clean VM every build is pulling down the same files again and again.

My initial thought was to use the appveyor build cache to preserve .git/lfs/objects folder between builds.
Unfortunately the build cache is restored after the repository is cloned so this doesn't help me.

Is there a way to somehow cache git-lfs data between builds?

Maybe if there is a way to interact with the build cache programmatically I could set something up on the init event.

Or if there was a way to have the build cache restore run before the repository is cloned.

Many Thanks

1. 1 Posted by Ilya Finkelshte... on 17 Mar, 2017 02:35 AM

   

   Hi Steven,

   What you can do is to clone you repo manually at install stage, and to prevent AppVeyor from cloning before cache restored, so something no-op like echo skip clone as clone_script.

   Let us know if this helps.

   Thanks!
   Ilya.

   • Edit

2. 2 Posted by sdawkins on 20 Mar, 2017 02:40 PM

   

   awesome, thanks very much.

   doing a manual clone in the install stage worked perfectly.

   the only problem I have is I don't know when to invalidate the build cache but I am happy to clear this manually for this project.

   below are the pertinent pieces of my appveyor.yml for future reference to anyone attempting the same sort of thing.

   bits and pieces taken from: https://www.appveyor.com/docs/how-to/git-push/

   environment:
     access_token:
       secure: <REDACTED>
   cache:
   - '.git\lfs\objects'
   clone_script: echo skip
   install:  
     - git config --global credential.helper store
     - ps: Add-Content "$env:USERPROFILE\.git-credentials" "https://$($env:access_token):x-oauth-basic@github.com`n"
     - git config --global user.email "<REDACTED>"
     - git config --global user.name "developer"
     - git init %APPVEYOR_BUILD_FOLDER%
     - cd %APPVEYOR_BUILD_FOLDER%
     - git remote add origin https://github.com/%APPVEYOR_REPO_NAME%.git
     - git fetch
     - git checkout %APPVEYOR_REPO_BRANCH%
   

   Many Thanks
   Steve

   • Edit

3. Support Staff 3 Posted by Feodor Fitsner on 20 Mar, 2017 03:26 PM

   

   Cool, thanks for the update!

   You can use "cache dependency" to invalidate the cache: https://www.appveyor.com/docs/build-cache/#addchange-dependency

   • Edit

4. 4 Posted by trevor.sandy on 05 Oct, 2017 09:14 PM

   

   Nothing wrong with the solution above but there are some superfluous commands.

   Unless you expect the push content back to your git repo, there's no need to create git-credentials and its associated secure content. The benefit of a lighter approach being if your repo is cloned by a 3rd party, they won't have to update the yml with new credentials etc...

   If your use case is to preserve .git\lfs\objects - or any other content in the 'clone_folder', this can be done in the following manner - which most closely resembles the skipped AppVeyor default cloning behavior:

   install:  
     # git fetch/checkout - 'clone_dir' can have cached content
     # git will see the 'clone_dir' as empty as long as there is no .git/HEAD file
     - git init %APPVEYOR_BUILD_FOLDER%
     - cd %APPVEYOR_BUILD_FOLDER%
     - git remote add origin https://github.com/%APPVEYOR_REPO_NAME%.git
     # the APPVEYOR_REPO_BRANCH and tags will be updated even if it does not fast-forward, because it is prefixed with a plus sign
     - git fetch -qfup --depth=5 origin +%APPVEYOR_REPO_BRANCH% +refs/tags/*:refs/tags/*
     # default AppVeyor checkout
     - git checkout -qf %APPVEYOR_REPO_COMMIT%
   

   Cheers,

   • Edit

5. 5 Posted by sdawkins on 05 Oct, 2017 09:14 PM

   

   I am out of the office on annual leave until Monday the 9th of October

   For support on any IT-related matter please contact [email blocked]

   • Edit

6. Ilya Finkelshteyn closed this discussion on 25 Aug, 2018 02:20 AM.