Repeated build failures due to NuGet package restore timeouts

James Skimming's Avatar

James Skimming

30 Sep, 2014 04:00 PM

We're receiving fairly regular (about 1 in 5) build failures due to timeouts when restoring a solution's NuGet packages (in the before_build step), the error often looks like this (the packages differ every time):

The operation has timed out
Unable to find version '3.2.2.0' of package 'Ninject'.
Unable to find version '3.2.0.0' of package 'Ninject.Extensions.Factory'.
Command exited with code 1

Given AppVeyor creates a new VM every build means the package restore is never cached and as a result the build is dependant on the NuGet servres, I'm not aware of how AppVeyor provisions the VM, I'm amusing there's a VM HDD reset, it is possible the VM IPs are reused and NuGet is throttling the requests?

Short of checking in the packages to source control, is there anything that can be done to mitigate the problem?

  1. Support Staff 1 Posted by Feodor Fitsner on 30 Sep, 2014 04:31 PM

    Feodor Fitsner's Avatar

    Have you looked at build cache solution: http://www.appveyor.com/docs/build-cache ?

  2. Support Staff 2 Posted by Feodor Fitsner on 30 Sep, 2014 04:33 PM

    Feodor Fitsner's Avatar

    I've just got the same error: https://ci.appveyor.com/project/appvyr/simple-web

    Seems like a NuGet.org issue?..

  3. Support Staff 3 Posted by Feodor Fitsner on 30 Sep, 2014 04:40 PM

    Feodor Fitsner's Avatar

    Right, I've removed local NuGet cache and did nuget restore. This is what I currently have on my dev machine - see attached.

  4. 4 Posted by James Skimming on 01 Oct, 2014 10:03 AM

    James Skimming's Avatar

    The build cache is awesome, I've chosen to just store the NuGet Cache and not the package folder directly, as that way the build is "cleaner". Here is my appveyor.yml:

    # Retain the NuGet package cache between builds.
    cache:
      - C:\Users\appveyor\AppData\Local\NuGet\Cache
    

    We have another project that's a mixture of .NET and AngulaJS and we're using node NPM and bower as well as NuGet, I've went for the following config

    cache:
      - node_modules
      - C:\Users\appveyor\AppData\Local\NuGet\Cache
      - C:\Users\appveyor\AppData\Roaming\npm-cache
      - C:\Users\appveyor\AppData\Roaming\bower\cache\packages
    

    I chosen to keep the node modules as installing the node packages takes the majority the build time, (I think because it's building a lot of the packages like phantomjs) But it's not restoring the folder node_modules even though that's the example given in the (documentation)(http://www.appveyor.com/docs/build-cache).

    Am I doing something wrong?

  5. 5 Posted by James Skimming on 01 Oct, 2014 10:28 AM

    James Skimming's Avatar

    Ignore that last comment, I need to put the relative path to the node_modules as it's not off the root of the repository.

    But now I'm receiving the following error after the build

    The specified path, file name, or both are too long. The fully qualified file name must be less than 260 characters, and the directory name must be less than 248 characters.
    

    I suspect it's caused by the deep nesting of node packages. Other than moving the site to the root of the repository, do you know if this can be resolved?

  6. Support Staff 6 Posted by Feodor Fitsner on 01 Oct, 2014 03:10 PM

    Feodor Fitsner's Avatar

    I'll be working on "path too long" issue with build cache. Have some ideas to try.

    What are those NPM packages giving long paths I can try locally to reproduce the issue?

  7. 7 Posted by James Skimming on 01 Oct, 2014 08:17 PM

    James Skimming's Avatar

    It's just the node packages of the AngularSeed project. I've created a build from my fork, but it's not failing presumably because the project is in he root of the repo.

    If you take the same packages and build it from a deeper folder structure I believe it will fail.

  8. Support Staff 8 Posted by Feodor Fitsner on 02 Oct, 2014 06:11 AM

    Feodor Fitsner's Avatar

    Hi James,

    We've deployed AppVeyor update with a fix for "path too long" issue. You can enable cache for folders of any depth now.

  9. 9 Posted by James Skimming on 02 Oct, 2014 09:43 AM

    James Skimming's Avatar

    Hi Feodor

    Thanks for the quick turn around.

    I've tried and I'm no longer receiving the path too long error, but I'm receiving a failure with bower.

    In my angular-seed fork, you can see the package restore worked fine when the site was in the rot of the repo, see successful build 0.1.6 using the build cache.

    I then moved the site to a deeper folder. The initial build 0.1.8 went through fine as the node packaged were rebuilt due to not having been cached.

    I then kicked off another build 0.1.9 where the node packages were restored from the build cache, but the build failed because I don't think the packages were correctly restored, resulting in the error with bower.

    Are you able to look at the project to investigate, it's all open source, so you can fork my fork and run it for yourself on the DeepFolders branch..

  10. Support Staff 10 Posted by Feodor Fitsner on 02 Oct, 2014 02:28 PM

    Feodor Fitsner's Avatar

    Will take a look.-Feodor

  11. Support Staff 11 Posted by Feodor Fitsner on 03 Oct, 2014 09:28 PM

    Feodor Fitsner's Avatar

    Could you give it another try please? We've deployed an update with some fixes.

  12. 12 Posted by James Skimming on 03 Oct, 2014 09:50 PM

    James Skimming's Avatar

    Hi Feodor

    I'm afraid it didn't work. The first build 0.1.10 worked as there was no cache to restore (did you delete it?) at the end the node_modules appear to be saved fine.

    I then kicked off another build 0.1.11 but this time the build failed evan earlier at the point where it was restoring the node_modules.

    Hope this helps with your investigations.

  13. Support Staff 13 Posted by Feodor Fitsner on 03 Oct, 2014 09:53 PM

    Feodor Fitsner's Avatar

    Great, will continue digging.

  14. 14 Posted by James Skimming on 03 Oct, 2014 09:55 PM

    James Skimming's Avatar

    Hi Feodor

    I'd getting late here in the UK and I'll soon be shutting down for the night, if you want to continue using my build, give me your email address and I'll add you to the Team, I'm following @appveyor so you can DM me @JamesSkimming you don't want to post it here.

  15. Support Staff 15 Posted by Feodor Fitsner on 03 Oct, 2014 10:01 PM

    Feodor Fitsner's Avatar

    No worries, I'll setup it locally.

  16. 16 Posted by James Skimming on 15 Oct, 2014 08:28 AM

    James Skimming's Avatar

    Did you manage to get any further on this? Caching the various package caches has mostly mitigated any build issues.

    I suspect this one could be a bit of a pain to solve and is inherent in the 255 character file path limit Windows sometimes has. Therefore in the interest of pragmatism, we can live with the occasional failed builds especially if you can provide the rebuild button.

  17. Support Staff 17 Posted by Feodor Fitsner on 15 Oct, 2014 05:27 PM

    Feodor Fitsner's Avatar

    You mean this issue has sporadic nature?

  18. 18 Posted by James Skimming on 15 Oct, 2014 05:48 PM

    James Skimming's Avatar

    Sorry didn't mean to confuse. The node package restore where the project root is deep but using the build cache to cache the node_modules folder is 100% reproducible. We're now running without that setting.

    I initially raised this because of NuGet, Bower and npm restore issues. Adding the local package caches to the build cache has mostly mitigated the issue, but npm install still occasionally fails.

    Basically, if fixing this too much effort, we'd be happy to live with the occasional build failures if we had a rebuild button.

  19. Support Staff 19 Posted by Feodor Fitsner on 15 Oct, 2014 10:44 PM

    Feodor Fitsner's Avatar

    Sure, I'm going to investigate this caching issue.

  20. Support Staff 20 Posted by Feodor Fitsner on 16 Oct, 2014 04:12 AM

    Feodor Fitsner's Avatar

    Hi James,

    Build cache for long files names has been finally fixed!

    Looks at the results: https://ci.appveyor.com/project/appvyr/angular-seed/build/0.1.2

    Almost 2 times faster: https://ci.appveyor.com/project/appvyr/angular-seed/history :)

  21. James Skimming closed this discussion on 27 Dec, 2015 10:58 PM.

Comments are currently closed for this discussion. You can start a new one.

Keyboard shortcuts

Generic

? Show this help
ESC Blurs the current field

Comment Form

r Focus the comment reply box
^ + ↩ Submit the comment

You can use Command ⌘ instead of Control ^ on Mac