Home → Suggestions →

npm caching ala codeship

• Edit

Rasmus Lauridsen

02 Sep, 2014 09:38 AM

We have a project that where use npm to fetch our javascript dependencies.
npm install at the moment takes 2-3 minutes and sometimes fails in horrible ways.

It would be great if there was a feature that cached the local npm cache between builds.

Codeship has a solution where they cache npm packages for a build and restores them again on the next one.
All without the user being involved.

Could also be done a bit more manual where we users have to make an active choice.
Could be two powershell commandlets or something:
One that restores the npm cache to be used in init
A second that stores the npm cache to be used postbuild

Read about Codeships solution:
http://blog.codeship.io/2013/09/05/the-tech-room-dependency-cache-and-sphinx-support.html

Thanks
Rasmus

1. Support Staff 1 Posted by Feodor Fitsner on 03 Sep, 2014 03:01 AM

   

   Hi Rasmus,

   Sounds like a great idea! I've read through CodeShip article - thanks for the link. For Windows is it the same like preserving C:\Users\user\AppData\Roaming\npm-cache folder?

   • Edit

2. 2 Posted by Rasmus Lauridse... on 03 Sep, 2014 09:19 AM

   

   I'm not super into npm, but I would imagine that that would do it.

   Ok just did a very unscientific test on my dev machine.

   node_modules empty	cache empty	result
   X	X	5 minutes
   X		3.20 minutes
   		1.20 minutes

   As you can see above npm-cache does make a difference, but the real winner would be the node_modules folder in the project.
   npm install still does alot of copying on the disc from cache to npm_modules and this takes a while.

   Some maybe relevant experiences I have had with npm

   Long file names

   We tried checking our npm packages into Git (node_modules in project folder), which lead to a very long day of fighting the tools we use. The windows 260 character limit gave us some hassles. Lo-dash for instance has a very deep dependency tree leading to a very deep folder structure.
   Git for windows was a big part of this.

   appveyor artifact zip of directory

   I also tried to make the node_modules folder an artifact to make it zip. But the zip tool used in appveyor also balks at long filenames and fails.
   Maybe 7zip can work around this.

   • Edit

3. Support Staff 3 Posted by Feodor Fitsner on 03 Sep, 2014 06:36 PM

   

   OK, I see, thanks for checking this out. I guess node_modules could be local and global where local depends on the project, but in the most cases it's build root.

   I will see how we can do a cache for this and other stuff similar to CodeShip.

   You should give 7z a try.

   • Edit

4. Support Staff 4 Posted by Feodor Fitsner on 18 Sep, 2014 03:42 AM

   

   Hi Rasmus,

   There is a new caching mechanism allowing to preserve directories and files between builds. I've just added an article about it: http://www.appveyor.com/docs/build-cache

   You can give it a try and let me know what you think.

   • Edit

5. 5 Posted by Rasmus Lauridse... on 18 Sep, 2014 07:37 AM

   

   Hi Feodor

   Nice that you got started on it. Seems like I run into the filepath too long problem though. The very last line in my build when I turn it on, on the node_modules folder inside our web project.

   "The specified path, file name, or both are too long. The fully qualified file name must be less than 260 characters, and the directory name must be less than 248 characters."

   This also fails the build which I am not certain is the right thing to do. Especially since it happens after it started a deployment to the development environment.
   The build itself was a success after all.

   Looking forward to trying the next iteration on this feature, since it can make our current build process way simpler.
   We currently do the "caching" in our source control by having gulp zip up the node_modules. The js/node tools don't seem to have a problem with long paths.
   I then use 7z to unzip this file before building.

   Pros:

   • Doesn't seem to run into the npm errors that we ran into on the AppVeyor build servers ( BIG PRO :-)

   Cons:

   • Manual step to remember to commit the zip file.
   • Slow 7zip takes about 2 mins to unpack the modules.About the same time as the "npm install" command took before.
   • Zipping adds time to the builds on our local machines.
   • Edit

6. Support Staff 6 Posted by Feodor Fitsner on 18 Sep, 2014 02:32 PM

   

   Will take a look.

   -Feodor

   • Edit

7. 7 Posted by Rasmus Lauridse... on 18 Sep, 2014 03:39 PM

   

   Thank you :-)

   • Edit

8. Support Staff 8 Posted by Feodor Fitsner on 18 Sep, 2014 04:23 PM

   

   Is it possible for you to provide a build script reproducing "long path" issue?

   • Edit

9. 9 Posted by Rasmus Lauridse... on 19 Sep, 2014 06:50 AM

   

   Can I add you as a collaborator then you can see it in the build?

   We turned it off again but we can easily turn it back on.

   • Edit

10. Support Staff 10 Posted by Feodor Fitsner on 19 Sep, 2014 01:44 PM

    

    Sure, add me as "feodor AT appveyor.com"

    -Feodor

    • Edit

11. Ilya Finkelshteyn closed this discussion on 25 Aug, 2018 01:48 AM.