git-lfs and build cache

sdawkins's Avatar

sdawkins

16 Mar, 2017 01:32 PM

Hi,

I am finding that appveyor is burning through my git-lfs bandwidth quota pretty quickly, as each build is on a clean VM every build is pulling down the same files again and again.

My initial thought was to use the appveyor build cache to preserve .git/lfs/objects folder between builds.
Unfortunately the build cache is restored after the repository is cloned so this doesn't help me.

Is there a way to somehow cache git-lfs data between builds?

Maybe if there is a way to interact with the build cache programmatically I could set something up on the init event.

Or if there was a way to have the build cache restore run before the repository is cloned.

Many Thanks

  1. Support Staff 1 Posted by Ilya Finkelshte... on 17 Mar, 2017 02:35 AM

    Ilya Finkelshteyn's Avatar

    Hi Steven,

    What you can do is to clone you repo manually at install stage, and to prevent AppVeyor from cloning before cache restored, so something no-op like echo skip clone as clone_script.

    Let us know if this helps.

    Thanks!
    Ilya.

  2. 2 Posted by sdawkins on 20 Mar, 2017 02:40 PM

    sdawkins's Avatar

    awesome, thanks very much.

    doing a manual clone in the install stage worked perfectly.

    the only problem I have is I don't know when to invalidate the build cache but I am happy to clear this manually for this project.

    below are the pertinent pieces of my appveyor.yml for future reference to anyone attempting the same sort of thing.

    bits and pieces taken from: https://www.appveyor.com/docs/how-to/git-push/

    environment:
      access_token:
        secure: <REDACTED>
    cache:
    - '.git\lfs\objects'
    clone_script: echo skip
    install:  
      - git config --global credential.helper store
      - ps: Add-Content "$env:USERPROFILE\.git-credentials" "https://$($env:access_token):x-oauth-basic@github.com`n"
      - git config --global user.email "<REDACTED>"
      - git config --global user.name "developer"
      - git init %APPVEYOR_BUILD_FOLDER%
      - cd %APPVEYOR_BUILD_FOLDER%
      - git remote add origin https://github.com/%APPVEYOR_REPO_NAME%.git
      - git fetch
      - git checkout %APPVEYOR_REPO_BRANCH%
    

    Many Thanks
    Steve

  3. Support Staff 3 Posted by Feodor Fitsner on 20 Mar, 2017 03:26 PM

    Feodor Fitsner's Avatar

    Cool, thanks for the update!

    You can use "cache dependency" to invalidate the cache: https://www.appveyor.com/docs/build-cache/#addchange-dependency

  4. 4 Posted by trevor.sandy on 05 Oct, 2017 09:14 PM

    trevor.sandy's Avatar

    Nothing wrong with the solution above but there are some superfluous commands.

    Unless you expect the push content back to your git repo, there's no need to create git-credentials and its associated secure content. The benefit of a lighter approach being if your repo is cloned by a 3rd party, they won't have to update the yml with new credentials etc...

    If your use case is to preserve .git\lfs\objects - or any other content in the 'clone_folder', this can be done in the following manner - which most closely resembles the skipped AppVeyor default cloning behavior:

    install:  
      # git fetch/checkout - 'clone_dir' can have cached content
      # git will see the 'clone_dir' as empty as long as there is no .git/HEAD file
      - git init %APPVEYOR_BUILD_FOLDER%
      - cd %APPVEYOR_BUILD_FOLDER%
      - git remote add origin https://github.com/%APPVEYOR_REPO_NAME%.git
      # the APPVEYOR_REPO_BRANCH and tags will be updated even if it does not fast-forward, because it is prefixed with a plus sign
      - git fetch -qfup --depth=5 origin +%APPVEYOR_REPO_BRANCH% +refs/tags/*:refs/tags/*
      # default AppVeyor checkout
      - git checkout -qf %APPVEYOR_REPO_COMMIT%
    

    Cheers,

  5. 5 Posted by sdawkins on 05 Oct, 2017 09:14 PM

    sdawkins's Avatar

    I am out of the office on annual leave until Monday the 9th of October

    For support on any IT-related matter please contact [email blocked]

Reply to this discussion

Internal reply

Formatting help / Preview (switch to plain text) No formatting (switch to Markdown)

Attaching KB article:

»

Attached Files

You can attach files up to 10MB

If you don't have an account yet, we need to confirm you're human and not a machine trying to post spam.

Keyboard shortcuts

Generic

? Show this help
ESC Blurs the current field

Comment Form

r Focus the comment reply box
^ + ↩ Submit the comment

You can use Command ⌘ instead of Control ^ on Mac