All builds failing due to artifact storage limit exceeded

DonLakeFlyer's Avatar

DonLakeFlyer

25 Oct, 2018 04:47 PM

Always get "Maximum allowed artifact storage size of 50000 Mb will be exceeded.". Tried deleting old builds which didn't help. Any way to clear this somehow? We are working to reduce our storage on each build but until then we are stuck. Example failure: https://ci.appveyor.com/project/Dronecode/qgroundcontrol/builds/19789546

  1. Support Staff 1 Posted by Owen McDonnell on 25 Oct, 2018 05:17 PM

    Owen McDonnell's Avatar

    Your artifact storage size has been increased. Let us know if you can build again.

  2. 2 Posted by DonLakeFlyer on 25 Oct, 2018 06:22 PM

    DonLakeFlyer's Avatar

    Now I get: Maximum allowed artifact storage size of 70000 Mb will be exceeded.

    This is a limit for all builds right? Not just for the artifacts on a signal build.

  3. Support Staff 3 Posted by Owen McDonnell on 25 Oct, 2018 07:39 PM

    Owen McDonnell's Avatar

    Yes, the limit is for all builds. For now the retention policy is that artifacts older than 6 months are removed making space for more, however we plan to add a more flexible policy in the future.
    Storage size has been increased again.

  4. 4 Posted by DonLakeFlyer on 25 Oct, 2018 08:28 PM

    DonLakeFlyer's Avatar

    90000 Mb exceeded now. Is it possible for you to blow away all of our artifacts?

  5. Support Staff 5 Posted by Owen McDonnell on 25 Oct, 2018 09:21 PM

    Owen McDonnell's Avatar

    Do you need all the artifacts your building? Can you possibly restrict artifact creation to the branch that really needs them?

    It looks like you have a very active project that would quickly fill any artifact storage.

  6. 6 Posted by DonLakeFlyer on 25 Oct, 2018 09:22 PM

    DonLakeFlyer's Avatar

    Yup, we have work in progress to cut way down on artifact storage. Just trying to get things moving forward again while we work on that.

  7. Support Staff 7 Posted by Owen McDonnell on 25 Oct, 2018 09:55 PM

    Owen McDonnell's Avatar

    It seems we're both caught in between improvements....

    We don't have a way to intervene in our artifacts retention policy yet so your limit has been increased again.

    Might I suggest adding this to your yaml,

    for:
      - 
        branches:
          only: 
            - <branches-that-need-artifact-creation>
        artifacts: 
          - path: <path-to-artifact>
            name: <artifact-name>
    

    It may differ depending on how you create artifacts. But we're happy to assist you in implementation.

  8. 8 Posted by DonLakeFlyer on 26 Oct, 2018 05:43 PM

    DonLakeFlyer's Avatar

    Still failing due to limit. The problem is that we push our installer and pdb to S3 on each build. We continually release daily builds to our users on each change. Which I believe means we need to store them as an artifact first. We don't really need the artifact to be kept in appveyor. We just need it to get to S3. Hence asking whether it possible to just toss all old artifacts for our project since we don't need them.

  9. 9 Posted by DonLakeFlyer on 29 Oct, 2018 06:10 PM

    DonLakeFlyer's Avatar

    Still stuck here. Becoming a really bad problem since we are now on multiple weeks without a new Windows build being pushed up for our users.

  10. 10 Posted by DonLakeFlyer on 31 Oct, 2018 05:40 PM

    DonLakeFlyer's Avatar

    Still dead in the water. No appveyor builds for multiple weeks now. No response for help for 5 days. At a loss as to what to do other than start considering other competitor options for other open source and paid accounts.

  11. Support Staff 11 Posted by Ilya Finkelshte... on 31 Oct, 2018 06:04 PM

    Ilya Finkelshteyn's Avatar

    We extended your limit more to unblock you. Sorry for the trouble.

  12. 12 Posted by DonLakeFlyer on 31 Oct, 2018 06:42 PM

    DonLakeFlyer's Avatar

    Builds are working now. Thank you so much. No we can get working on changing things so we don't use so much artifact storage.

  13. 13 Posted by DonLakeFlyer on 02 Nov, 2018 06:21 PM

    DonLakeFlyer's Avatar

    Worked for a couple days. Failing again: https://ci.appveyor.com/project/Dronecode/qgroundcontrol/builds/20009518.

    Maximum allowed artifact storage size of 50000 Mb will be exceeded.

    Sigh...

  14. Support Staff 14 Posted by Ilya Finkelshte... on 02 Nov, 2018 07:38 PM

    Ilya Finkelshteyn's Avatar

    It was reset back to default value. You probably subscribed to the different plan recently? This is the only reason I can think of for resetting the artifact size quota. I bumped it again ti unblock you.

  15. 15 Posted by DonLakeFlyer on 02 Nov, 2018 07:54 PM

    DonLakeFlyer's Avatar

    Ok, thanks. Hmm, may have been someone else. I think we were signing for a paid open source account or something so we could get multiple builds. Heard that mentioned.

    So for our builds we want all of our master builds to upload to S3 both the installer and the .pdb files. On S3 this gets overwritten with each new build such that there is only one master installer and .pdb file. But for appveyor this ends up blowing up into saved artifacts for every single build. The concept of making all devs builds available to users would seem to me to be a common thing to do. Is there no way to set this up such that it runs afoul of the artifact limit. We don't really need the artifacts stored in appveyor, we just need to get the files to S3.

  16. Support Staff 16 Posted by Ilya Finkelshte... on 02 Nov, 2018 08:07 PM

    Ilya Finkelshteyn's Avatar

    Yes, we understand. We plan to implement "transient" artifacts (which live until build ends) for that. I cannot say specific ETA, but it should take too long as we unhappy with this situation too. Meanwhile we are bumping storage for users like you if we do not see an abuse.

    Other option available to you right now is to deploy with a script instead of S3 deployment provider. AWS CLI is installed on build worker image and you can simple use files, for CLI without artifacts abstraction layer. Secrets can live in secure variables. In this case you do not need to package your files as artifacts at all. Admittedly this is less handy than 1st class S3 deployment, but you can consider it. We can help with samples if you need.

  17. 17 Posted by DonLakeFlyer on 02 Nov, 2018 08:09 PM

    DonLakeFlyer's Avatar

    Yeah, that would work perfectly. I guess if transient artifacts don't come soon enough and we keep hitting the limit even with the work we are doing to reduce artifacts we can switch to the hard way of uploading manually. Thanks for the info.

  18. Support Staff 18 Posted by Ilya Finkelshte... on 02 Nov, 2018 08:20 PM

    Ilya Finkelshteyn's Avatar

    By "manually" you mean with custom script which runs as deploy_script?

  19. 19 Posted by DonLakeFlyer on 03 Nov, 2018 07:43 PM

    DonLakeFlyer's Avatar

    Correct

  20. Support Staff 20 Posted by Ilya Finkelshte... on 05 Nov, 2018 04:26 PM

    Ilya Finkelshteyn's Avatar

    Created GitHub issue to track this.

  21. Ilya Finkelshteyn closed this discussion on 05 Nov, 2018 04:26 PM.

  22. DonLakeFlyer re-opened this discussion on 29 Nov, 2018 02:37 AM

  23. 21 Posted by DonLakeFlyer on 29 Nov, 2018 02:37 AM

    DonLakeFlyer's Avatar

    Wondering if you could help taking a look at this: https://github.com/mavlink/qgroundcontrol/pull/7039. Trying to make artifact storage only for master and Stable_V3.4 branch on repro. Sorry, not much YAML experience.

    ```
    for:
      -
        branches:
          only: master
            artifacts:
              -
                path: QGroundControl-installer.exe
                name: qgcinstaller
          only: Stable_V3.4
            artifacts:
              -
                path: symbols\**\*.*_
                name: symbols
    ```

  24. 22 Posted by DonLakeFlyer on 29 Nov, 2018 02:39 AM

    DonLakeFlyer's Avatar

    That's not quite right. I want the QGroundControl-install.exe to be stored for both master and Stable_V3.4. Whereas symbols should only be stored for Stable_V3.4.

  25. Support Staff 23 Posted by Ilya Finkelshte... on 29 Nov, 2018 03:39 AM

    Ilya Finkelshteyn's Avatar

    Try this:

    for:
    -
      branches:
        only:
          - master
      artifacts:
      - path: QGroundControl-installer.exe
        name: qgcinstaller
    -
      branches:
        only:
          - Stable_V3.4
      artifacts:
        - path: QGroundControl-installer.exe
          name: qgcinstaller
        - path: symbols\**\*.*_
          name: symbols
    

    And remove this. You can move deportment setting to under the same for.branches construct, but for deployment it is simpler to use conditional deployment.

    Also check Validate YAML tool under account Settings menu.

  26. 24 Posted by DonLakeFlyer on 29 Nov, 2018 04:56 AM

    DonLakeFlyer's Avatar

    Awesome, thanks so much Ilya.

  27. Ilya Finkelshteyn closed this discussion on 29 Nov, 2018 12:53 PM.

Comments are currently closed for this discussion. You can start a new one.

Keyboard shortcuts

Generic

? Show this help
ESC Blurs the current field

Comment Form

r Focus the comment reply box
^ + ↩ Submit the comment

You can use Command ⌘ instead of Control ^ on Mac