git-lfs and build cache

sdawkins's Avatar

sdawkins

16 Mar, 2017 01:32 PM

Hi,

I am finding that appveyor is burning through my git-lfs bandwidth quota pretty quickly, as each build is on a clean VM every build is pulling down the same files again and again.

My initial thought was to use the appveyor build cache to preserve .git/lfs/objects folder between builds.
Unfortunately the build cache is restored after the repository is cloned so this doesn't help me.

Is there a way to somehow cache git-lfs data between builds?

Maybe if there is a way to interact with the build cache programmatically I could set something up on the init event.

Or if there was a way to have the build cache restore run before the repository is cloned.

Many Thanks

  1. Support Staff 1 Posted by Ilya Finkelshte... on 17 Mar, 2017 02:35 AM

    Ilya Finkelshteyn's Avatar

    Hi Steven,

    What you can do is to clone you repo manually at install stage, and to prevent AppVeyor from cloning before cache restored, so something no-op like echo skip clone as clone_script.

    Let us know if this helps.

    Thanks!
    Ilya.

  2. 2 Posted by sdawkins on 20 Mar, 2017 02:40 PM

    sdawkins's Avatar

    awesome, thanks very much.

    doing a manual clone in the install stage worked perfectly.

    the only problem I have is I don't know when to invalidate the build cache but I am happy to clear this manually for this project.

    below are the pertinent pieces of my appveyor.yml for future reference to anyone attempting the same sort of thing.

    bits and pieces taken from: https://www.appveyor.com/docs/how-to/git-push/

    environment:
      access_token:
        secure: <REDACTED>
    cache:
    - '.git\lfs\objects'
    clone_script: echo skip
    install:  
      - git config --global credential.helper store
      - ps: Add-Content "$env:USERPROFILE\.git-credentials" "https://$($env:access_token):x-oauth-basic@github.com`n"
      - git config --global user.email "<REDACTED>"
      - git config --global user.name "developer"
      - git init %APPVEYOR_BUILD_FOLDER%
      - cd %APPVEYOR_BUILD_FOLDER%
      - git remote add origin https://github.com/%APPVEYOR_REPO_NAME%.git
      - git fetch
      - git checkout %APPVEYOR_REPO_BRANCH%
    

    Many Thanks
    Steve

  3. Support Staff 3 Posted by Feodor Fitsner on 20 Mar, 2017 03:26 PM

    Feodor Fitsner's Avatar

    Cool, thanks for the update!

    You can use "cache dependency" to invalidate the cache: https://www.appveyor.com/docs/build-cache/#addchange-dependency

Reply to this discussion

Internal reply

Formatting help / Preview (switch to plain text) No formatting (switch to Markdown)

Attaching KB article:

»

Attached Files

You can attach files up to 10MB

If you don't have an account yet, we need to confirm you're human and not a machine trying to post spam.

Keyboard shortcuts

Generic

? Show this help
ESC Blurs the current field

Comment Form

r Focus the comment reply box
^ + ↩ Submit the comment

You can use Command ⌘ instead of Control ^ on Mac