SSL Certificates appear to be broken on some build instances

Richard's Avatar

Richard

30 Jan, 2019 03:26 PM

When running a curl command to submit test results to your servers I'm getting a certificate verification failed. The command that I'm running is:

find "$APPVEYOR_BUILD_FOLDER" -type f -name 'TEST*.xml' -print0 | xargs -0 -I '{}' curl -F 'file=@{}' "https://ci.appveyor.com/api/testresults/junit/$APPVEYOR_JOB_ID"

The error is:

curl: (60) server certificate verification failed. CAfile: /etc/ssl/certs/ca-certificates.crt CRLfile: none
More details here: http://curl.haxx.se/docs/sslcerts.html

Here's an example of a build where its failed:

https://ci.appveyor.com/project/RichardWarburton/artio/builds/21998252/job/ik8n7q0qlbufmqpt

I believe this is the result of an error with the SSL certificate store configuration on your build images.

  1. Support Staff 1 Posted by Wasa Pleshakov on 30 Jan, 2019 09:27 PM

    Wasa Pleshakov's Avatar

    As I can see this is a transient error: you have ~70 files but only ~6 curl calls fails. So it sent some tests.
    I was able to reproduce the issue. We will investigate it further to pin down the root cause.
    Meanwhile, you may do next:
    1. add -t option to xargs for better visibility that files it sends
    2. add --retry 3 option to curl command to instruct it retry send 3 times
    3. add -v option to curl to see conversation between curl and server.

    find "$APPVEYOR_BUILD_FOLDER" -type f -name 'TEST*.xml' -print0 | xargs -t -0 -I '{}' curl --retry 3 -F 'file=@{}' "https://ci.appveyor.com/api/testresults/junit/$APPVEYOR_JOB_ID"
    

    Am I correct you experienced this issue today starting from build # 1.0.77 ?

  2. 2 Posted by Richard on 30 Jan, 2019 09:31 PM

    Richard's Avatar

    Thanks for following up.

    Yes, it did start with 1.0.77. 1.0.85 only passed because I put a -k parameter to curl in which ignores the verification error.

  3. 3 Posted by Vitali Khlebko on 30 Jan, 2019 10:18 PM

    Vitali Khlebko's Avatar

    Have the same issue. Our automated pipeline is failing (Python code:)

    SSLError: HTTPSConnectionPool(host='ci.appveyor.com', port=443): Max retries exceeded with url: /api/projects/GuillaumeBoisvert/smo365/build/1.17.0.1037 (Caused by SSLError(SSLError("bad handshake: Error([('SSL routines', 'tls_process_server_certificate', 'certificate verify failed')],)",),))

  4. 4 Posted by Vitali Khlebko on 31 Jan, 2019 12:41 PM

    Vitali Khlebko's Avatar

    Any updates on that?

    It prevents us from using AppVeyor entirely right now.

    Most likely, one of the load-balancing proxies has an outdated or incorrect SSL. It would explain this intermittent issue.

  5. Support Staff 5 Posted by Wasa Pleshakov on 31 Jan, 2019 03:02 PM

    Wasa Pleshakov's Avatar

    Vitali,
    We testing our frontend to pin this issue.
    Meanwhile can you please use -k option for curl as Richard did ?

  6. 6 Posted by Vitali Khlebko on 31 Jan, 2019 04:24 PM

    Vitali Khlebko's Avatar

    Wasa,

    or pipeline is Python based. Though I just patched it to skip verification:

            response = requests.get(url, headers=self._get_auth_headers(), verify=self._ssl_verify)

  7. Support Staff 7 Posted by Ilya Finkelshte... on 01 Feb, 2019 02:31 AM

    Ilya Finkelshteyn's Avatar

    It should be fixed now. It was one of web frontend servers on ci.appveyor.com side who did not correctly provide certificate chain on SSL handshake. Please let us know if you still see this issue.

  8. 8 Posted by Vitali Khlebko on 01 Feb, 2019 04:12 PM

    Vitali Khlebko's Avatar

    It works fine now, thank you!

    Temporary workaround is removed

  9. Support Staff 9 Posted by Ilya Finkelshte... on 01 Feb, 2019 05:37 PM

    Ilya Finkelshteyn's Avatar

    Good, thanks a lot for the update!

  10. Ilya Finkelshteyn closed this discussion on 01 Feb, 2019 05:37 PM.

Comments are currently closed for this discussion. You can start a new one.

Keyboard shortcuts

Generic

? Show this help
ESC Blurs the current field

Comment Form

r Focus the comment reply box
^ + ↩ Submit the comment

You can use Command ⌘ instead of Control ^ on Mac

 

22 Aug, 2019 08:13 PM
21 Aug, 2019 11:06 PM
21 Aug, 2019 02:08 PM
21 Aug, 2019 01:37 PM
21 Aug, 2019 10:21 AM
21 Aug, 2019 05:56 AM