Cache is broken

ozkriff's Avatar

ozkriff

10 Oct, 2017 09:08 AM

Hi. I can't use cargo's cache anymore in my project because of the "The given key was not present in the dictionary." error :(

Example: https://ci.appveyor.com/project/ozkriff/zemeroth/build/1.0.200/job/ddk1yhpxo6cdq3t8

  1. Support Staff 1 Posted by Ilya Finkelshte... on 10 Oct, 2017 03:10 PM

    Ilya Finkelshteyn's Avatar

    I see that you are disabled cache to unblock your build. Most probably if you enable it again, it will work OK. However we strongly recommend to use cache dependency. In this case only single file will be scanned for a change instead of the whole cache, which is much less expensive. I am not sure what file can be used as a dependency, as I am not familiar with cargo, but I believe it should be some single file which can be used to identify if cargo cache should be updated or not.

  2. 2 Posted by ozkriff on 11 Oct, 2017 11:46 AM

    ozkriff's Avatar

    > Most probably if you enable it again, it will work OK. However we strongly recommend to use cache dependency.

    I've created a separate branch for this:

    https://github.com/ozkriff/zemeroth/commits/appveyor_cache_test

    But it doesn't seem to help much:

    https://ci.appveyor.com/project/ozkriff/zemeroth/build/1.0.267
    https://ci.appveyor.com/project/ozkriff/zemeroth/build/1.0.268
    https://ci.appveyor.com/project/ozkriff/zemeroth/build/1.0.269

  3. 3 Posted by Anatol Belski on 21 Oct, 2017 10:08 AM

    Anatol Belski's Avatar

    Hi,

    seems we hit this issue, too

    https://ci.appveyor.com/project/php/php-src/build/master.build.4222

    Disabling and reenabling the cache didn't work unfortunately, as the issue comes back then. I also found this older ticket

    http://help.appveyor.com/discussions/problems/1270-broken-unusable-cache-after-commenting-out-a-cache-item

    which tells the cache can be reset for a project. I'd like to ask for a cache reset for https://ci.appveyor.com/project/php/php-src . Perhaps there is some mistake in the config, please check http://git.php.net/?p=php-src.git;a=blob;f=.appveyor.yml;hb=HEAD

    Thanks.

  4. Support Staff 4 Posted by Ilya Finkelshte... on 24 Oct, 2017 08:18 PM

    Ilya Finkelshteyn's Avatar

    Hi Andrey and Anatol,

    Just want to let you know that we are still investigating it and will update you soon.

    Ilya.

  5. Support Staff 5 Posted by Ilya Finkelshte... on 25 Oct, 2017 09:37 PM

    Ilya Finkelshteyn's Avatar

    We identified the problem and working on fix.

  6. Support Staff 6 Posted by Feodor Fitsner on 29 Oct, 2017 12:07 AM

    Feodor Fitsner's Avatar

    Hi guys,

    We've deployed an update fixing the issue above. Please let us know how that worked.

  7. 7 Posted by Anatol Belski on 29 Oct, 2017 10:37 AM

    Anatol Belski's Avatar

    Hi,

    Thanks for staying on the fix. I've re-enabled caching and so far there's a bit different issue that repeated itself in two different jobs

    https://ci.appveyor.com/project/php/php-src/build/PHP-7.1.build.4417/job/g73b9x053d34rmth
    https://ci.appveyor.com/project/php/php-src/build/PHP-7.2.build.4419/job/mab8mt0nl65dl23n

    Another build from the same job was ok, however. Perhaps it has something to do with the concurrency, but i'm not sure. I'll keep the cache enabled for some time and observe the behavior.

    Thanks.

  8. 8 Posted by ozkriff on 29 Oct, 2017 07:35 PM

    ozkriff's Avatar

    Builds with the cache are not broken for me anymore, but the cache isn't saved most of the time with the following message:

    ```
    Updating build cache...
    Cache 'target' - Uploading (213,668,818 bytes)...100%Error uploading cache entry to the cache storage: Remote server returned 500: There is not enough space on the disk.
    Cache 'C:\Users\appveyor\.cargo\registry' - Uploading (71,033,396 bytes)...100%Error uploading cache entry to the cache storage: Remote server returned 500: There is not enough space on the disk.
    ```

    See:
    https://ci.appveyor.com/project/ozkriff/zemeroth/build/1.0.316
    https://ci.appveyor.com/project/ozkriff/zemeroth/build/1.0.316/job/c9hl7cqirtg0n5x6

  9. 9 Posted by Anatol Belski on 30 Oct, 2017 07:21 AM

    Anatol Belski's Avatar

    Hi,

    on my side, the unzip error constantly comes back, like here

    https://ci.appveyor.com/project/php/php-src/build/PHP-7.1.build.4423/job/ugjv3rfqu9bw7u8o

    Thanks.

  10. Support Staff 10 Posted by Feodor Fitsner on 30 Oct, 2017 07:02 PM

    Feodor Fitsner's Avatar

    @ozkriff that's because OSS accounts take 1 GB (per account) for cache disk space.

  11. Support Staff 11 Posted by Feodor Fitsner on 30 Oct, 2017 07:03 PM

    Feodor Fitsner's Avatar

    @Anatol - looks like you re-created the cache and it's OK now, right?

  12. 12 Posted by Anatol Belski on 30 Oct, 2017 07:19 PM

    Anatol Belski's Avatar

    Hi Feodor,

    i was working today to mitigate the unzip fails, but they still persist. Please see

    https://ci.appveyor.com/project/php/php-src/build/PHP-7.1.build.4466

    The job fail is because of the unrelated reason, but one can see that one run had unzip error, while the other one could restore cache. Both in the same matrix run. I suspect that it were same issue with the cache size, as the doc tells

    "Total cache size per account consist of all caches from all projects. If projects contain more than one job in build matrix or in parallel testing, each job has its own separate cache, which cannot be shared with other jobs or projects."

    And a cache for one job could be like 300mb, so likely it gets invalidated when other jobs save their caches. Whereby from the concrete project sight - in the same matrix, the cache should be valid for every job. It's anyway a bit confusing, that AppVeyor tells it was an unzip error. I can be also wrong at this point, please advise :)

    I've anyway fixed the scripts so the build run would proceed further if the cache couldn't be unzipped. If jobs cache concurrently and the cache size is exceeded, it'll re-fetch and continue the operation.

    Thanks.

  13. Support Staff 13 Posted by Feodor Fitsner on 31 Oct, 2017 04:55 AM

    Feodor Fitsner's Avatar

    Hi Anatol,

    I've deleted the cache for php-src project, so it will be re-created on the next build. Let see if that fixes the issue; otherwise will be further investigating.

  14. 14 Posted by ozkriff on 31 Oct, 2017 06:13 AM

    ozkriff's Avatar

    Hmm, then I don't understand how the cache limit works: I've reduced the number of build configurations

    But I'm still getting this every time now:

    Updating build cache...
    Cache 'target' - Uploading (47,079,371 bytes)...100%
    Error uploading cache entry to the cache storage: Remote server returned 500: There is not enough space on the disk.
    Cache 'C:\Users\appveyor\.cargo\registry' - Uploading (71,178,903 bytes)...100%
    Error uploading cache entry to the cache storage: Remote server returned 500: There is not enough space on the disk.
    

    3 * 118mb should be less than 1gb. What am I doing wrong?

  15. 15 Posted by Anatol Belski on 31 Oct, 2017 09:44 PM

    Anatol Belski's Avatar

    Hi Feodor,

    i was watching the PHP builds all the day today. I couldn't spot the unzip issue anymore. Some builds tell there were no build cache at all like here

    https://ci.appveyor.com/project/php/php-src/build/master.build.4491

    but anyway the build scripts are prepared to this situation and it seems logic according to the cache sie policy anyway. ATM i'd say the cache issue is fixed so far.

    Thanks.

    Anatol

  16. Support Staff 16 Posted by Feodor Fitsner on 31 Oct, 2017 09:56 PM

    Feodor Fitsner's Avatar

    Cool, thanks for the update!

  17. 17 Posted by ozkriff on 19 Nov, 2017 08:01 PM

    ozkriff's Avatar
  18. Support Staff 18 Posted by Feodor Fitsner on 21 Nov, 2017 01:57 AM

    Feodor Fitsner's Avatar

    @ozkriff sorry for the delay with reply! Cache size is limited per account, not individual project, so probably cache is taken by other projects under your account?

  19. 19 Posted by ozkriff on 21 Nov, 2017 06:18 AM

    ozkriff's Avatar

    Ouch, got it. Deleted old projects cache with this - https://github.com/appveyor/ci/issues/985#issuecomment-279199811 - everything works now.

    Yay! Thanks! :)

Reply to this discussion

Internal reply

Formatting help / Preview (switch to plain text) No formatting (switch to Markdown)

Attaching KB article:

»

Attached Files

You can attach files up to 10MB

If you don't have an account yet, we need to confirm you're human and not a machine trying to post spam.

Keyboard shortcuts

Generic

? Show this help
ESC Blurs the current field

Comment Form

r Focus the comment reply box
^ + ↩ Submit the comment

You can use Command ⌘ instead of Control ^ on Mac