Builds failing on Chromedriver download (Angular 6 CLI e2e tests step)

jeroen.heijmans's Avatar

jeroen.heijmans

19 Sep, 2018 09:16 AM

We have two Angular 6 CLI projects where the e2e tests step fails on AppVeyor:

This is the anonymized tail of the log:

[13:43:00] I/file_manager - creating folder C:\projects\projectnamehere\node_modules\protractor\node_modules\webdriver-manager\selenium
[13:43:00] I/config_source - curl -oC:\projects\projectnamehere\node_modules\protractor\node_modules\webdriver-manager\selenium\chrome-response.xml https://chromedriver.storage.googleapis.com/
ℹ 「wdm」: Compiled successfully.
events.js:183
      throw er; // Unhandled 'error' event
      ^
Error: read ECONNRESET
    at TLSWrap.onread (net.js:622:25)
npm ERR! code ELIFECYCLE
npm ERR! errno 1
npm ERR! projectnamehere@0.0.0 e2e: `node git-version.js && ng e2e`
npm ERR! Exit status 1
npm ERR! 
npm ERR! Failed at the projectnamehere@0.0.0 e2e script.
npm ERR! This is probably not a problem with npm. There is likely additional logging output above.
npm ERR! A complete log of this run can be found in:
npm ERR!     C:\Users\appveyor\AppData\Roaming\npm-cache\_logs\2018-09-18T13_43_01_245Z-debug.log
Command exited with code 1

At the point where AppVeyor now fails, it previously (and locally for the current commits) showed:

[13:43:21] I/file_manager - creating folder C:\projects\projectnamehere\node_modules\protractor\node_modules\webdriver-manager\selenium
[13:43:21] I/config_source - curl -oC:\projects\projectnamehere\node_modules\protractor\node_modules\webdriver-manager\selenium\chrome-response.xml https://chromedriver.storage.googleapis.com/
ℹ 「wdm」: Compiled successfully.
[13:43:22] I/downloader - curl -oC:\projects\projectnamehere\node_modules\protractor\node_modules\webdriver-manager\selenium/chromedriver_2.42.zip https://chromedriver.storage.googleapis.com/2.42/chromedriver_win32.zip
[13:43:22] I/update - chromedriver: unzipping chromedriver_2.42.zip
[13:43:23] I/launcher - Running 1 instances of WebDriver
[13:43:23] I/direct - Using ChromeDriver directly...
Jasmine started

etc.

This happens intermittently: sometimes we try a fix/workaround (upgrading Node, putting a curl command in the test_scripts) and it seems to work, but the same commit might fail a build if re-run.

Given that the same commit sometimes fails, and sometimes doesn't, I presume this is an AppVeyor issue?

Here's our appveyor.yml relevant parts:

version: 1.0.{build}
image: Visual Studio 2017
skip_branch_with_pr: true

environment:
  nodejs_version: "8"

cache:
  - node_modules -> package.json

install:
  - ps: Install-Product node $env:nodejs_version
  - npm install npm@6 -g
  - npm ci --silent
  - node --version
  - npm --version
  - choco install googlechrome --limit-output --ignore-checksum

build: off

test_script:
  - npm run lint
  - npm test -- --progress=false
  - npm run e2e

We tried switching to the "Previous Visual Studio 2017" image, to no avail.

I tried contacting webcare but got no response.

Any advice?

  1. 1 Posted by jeroen.heijmans on 19 Sep, 2018 11:54 AM

    jeroen.heijmans's Avatar

    I've reproduced said issue in an MCVE.

    This repository was nothing else but:

    1. New repo
    2. ng new app --inline-style
    3. Add the yml file (see repo)
    4. Create AppVeyor build

    The build works locally. So I think the issue is with the AppVeyor VM image?

  2. Support Staff 2 Posted by Owen McDonnell on 20 Sep, 2018 02:38 AM

    Owen McDonnell's Avatar

    As a temporary work-around as we investigate can you try adding appveyor-retry before the problematic command.

      - appveyor-retry npm run e2e
    
  3. 3 Posted by jeroen.heijmans on 20 Sep, 2018 07:52 AM

    jeroen.heijmans's Avatar

    The temporary workaround seems to have worked (at least on my MCVE repository, mentioned above), so that'll help for now.

    Looking forward to a more permanent solution.

  4. 4 Posted by jeroen.heijmans on 11 Oct, 2018 10:04 AM

    jeroen.heijmans's Avatar

    It's been three weeks, the workaround is starting to become less "temporary" :-)

    Any news on this issue @OwenMcDonnel and colleagues?

  5. Owen McDonnell closed this discussion on 12 Oct, 2018 05:39 AM.

  6. Owen McDonnell re-opened this discussion on 12 Oct, 2018 07:22 AM

  7. Support Staff 5 Posted by Owen McDonnell on 12 Oct, 2018 07:34 AM

    Owen McDonnell's Avatar

    You mention putting a curl command in the test_script section which made it seem to work. What command was that?

  8. 6 Posted by jeroen.heijmans on 12 Oct, 2018 11:35 AM

    jeroen.heijmans's Avatar

    The curl command in the logs I posted (the one just before the failure). However, I don't recall if that was a reliable fix or not.

  9. Support Staff 7 Posted by Owen McDonnell on 12 Oct, 2018 04:27 PM

    Owen McDonnell's Avatar

    I was able to get reliable builds by making the following changes.
    To package.json scripts,

        "pree2e": "webdriver-manager update --standalone false --gecko false",
        "e2e": "ng e2e --no-webdriver-update"
    

    and to appveyor.yml at the end of install: stage,

    - mkdir C:\projects\experiments-appveyor\node_modules\protractor\node_modules\webdriver-manager\selenium
    - curl -oC:\projects\experiments-appveyor\node_modules\protractor\node_modules\webdriver-manager\selenium\chrome-response.xml https://chromedriver.storage.googleapis.com/
    
  10. 8 Posted by jeroen.heijmans on 15 Oct, 2018 08:41 AM

    jeroen.heijmans's Avatar

    Thank you for your response Owen.

    Those changes seem to me like another workaround though, not really a solution. Compared to the earlie "retry" solution, it not only tightly couples my yml file to AppVeyor (which is okay, of course), but it also introduces complexity in my package.json file specific to the build server.

    Given the type of workaround it is, I feel even more strongly that the root cause is inside the build images? I'd prefer to see a fix there. (Or, perhaps I'm wrong, and one of my NPM packages has a bug?)

    Of the two workarounds, I guess the 'retry' version is the lesser evil, so I'll stick with that until a proper solution is available.

  11. Support Staff 9 Posted by Owen McDonnell on 15 Oct, 2018 10:50 PM

    Owen McDonnell's Avatar

    I agree that it doesn't seem like a true solution. However, i'm not yet convinced this is a build image problem.

    In researching the problem I came across numerous problems (and various workarounds) with respect to webdriver-manager.

    I didn't mean to propose this as a final solution but rather an update on our findings (though alternative workarounds can be helpful as well).

    We will continue to investigate, as we hope you will.

  12. 10 Posted by jeroen.heijmans on 16 Oct, 2018 07:08 AM

    jeroen.heijmans's Avatar

    Allright, I understand. Thanks for the updates!

    I will be honest and admit that I am using the first workaround and no longer investigating. Since I cannot reproduce this locally I don't have any convenient means (as far as I know) to further troubleshoot this.

    Though I guess I could try and run the same thing on other CI tools (perhaps my own GitLab CI environment) to see if the issue truly is AppVeyor-specific.

  13. Support Staff 11 Posted by Owen McDonnell on 23 Oct, 2018 11:39 PM

    Owen McDonnell's Avatar

    I think this is likely some network latency issue since it looks like the webdriver-manager module is downloading an xml then immediately parsing it and making another request for the latest drivers.
    As such, it is not really revelatory to try other CI machines and compare results. But if you're interested, I've enabled building from GCE workers so if you want to try some builds on those machines just add appveyor_build_worker_cloud: gce to your environment variables. Be aware that GCE workers take a little longer to provision so your build may take a few minutes to begin.

    I think short of making a pull request to said npm module, I would consider the "workaround" as a resolution.

  14. 12 Posted by jeroen.heijmans on 24 Oct, 2018 06:44 AM

    jeroen.heijmans's Avatar

    Thanks for your response. I'm not entirely sure yet why the package would be the cause (since for me, this issue only occurs on AppVeyor, and I'd also think that 'network latency' issues are specific to the environment, not the used code?).

    But I also trust that you know more about these things than I do :D - so I've asked the webdriver-manager community for help as you suggested.

    Thanks for your time and effort. Will wait and see if the community there can come up with another solution.

  15. Support Staff 13 Posted by Owen McDonnell on 24 Oct, 2018 03:53 PM

    Owen McDonnell's Avatar

    Package could be the cause within a certain environment. I.e. the package may not be resilient to network latency. Its a matter of semantics whether we assign that problem to the package or the environment.
    For example, you can have a build chain that produces errors when threads are scarce, but runs fine on a more powerful machine. Whether you declare the machine or the code in error depends on what you are able to remedy.

    Most important to us is that your build is not blocked : )

  16. Owen McDonnell closed this discussion on 24 Oct, 2018 03:53 PM.

  17. jeroen.heijmans re-opened this discussion on 24 Oct, 2018 03:58 PM

  18. 14 Posted by jeroen.heijmans on 24 Oct, 2018 03:58 PM

    jeroen.heijmans's Avatar

    Yes, of course, you're right.

    In any case I appreciate all the help you've provided!

    And with said workarounds I can carry on of course. Let's wait and see what the package maintainers/community have to offer.

  19. Owen McDonnell closed this discussion on 24 Oct, 2018 04:44 PM.

Comments are currently closed for this discussion. You can start a new one.

Keyboard shortcuts

Generic

? Show this help
ESC Blurs the current field

Comment Form

r Focus the comment reply box
^ + ↩ Submit the comment

You can use Command ⌘ instead of Control ^ on Mac