Home → Problems →

HTTP 500 error when uploading NUnit3 test results (suspected failure on parameterized test cases)

• Edit

craig

13 Oct, 2024 02:17 PM

Hi, I'm running tests for my open source project and uploading some of the test results to AppVeyor has recently begun failing with an HTTP 500 error. I suspect that this might be an issue with the test result parser (failing on a valid result file). I have attached a result file (for the record, this was generated on my own PC, not on AppVeyor).

I'm running the tests via Coverlet, to generate coverage reports as well as test results. You can see the test script here, as well as the rest of the repo if you wish. I'm not suspicious of coverlet or the test script. I have three test projects and two of them are uploading just fine with that same test script. The problem seems specific to one of the three projects.

Here's an example of a recent build in which the test results didn't upload. I had a look through the history of builds because these results were uploading once; I have narrowed down the first commit which stopped uploading results to this one.

In that commit, I introduced use of the parameterized [TestCase( ... )] technique, in order to test many combinations of values with just a small number of tests. By coincidence, the values I am testing include a number of open & close-brace characters {}, which might thwart a parser which expects them to represent something else. If my suspicions are correct, and it is this technique which is causing the problem, you'll find the troublesome tests in the ReportFormatCreatorTests test suite.

Is there anything that can be done about this? I appreciate that I could probably work around it by using just a single test, with many test values and Assert.Multiple(() => { /* test each of them */ }); but it seems that perhaps there is a real issue here.

• TestResults.xml 130 KB

1. Support Staff 1 Posted by Owen McDonnell on 15 Oct, 2024 10:58 PM

   

   As a preliminary sanity check can you push the test results as an artifact following the docs here and verify that all the expected test results files are uploaded as expected?

   • Edit

2. 2 Posted by craig on 28 Oct, 2024 01:37 PM

   

   Sorry for the delay; this is now done. Here is a build which includes the problematic test result artifact uploaded in the manner requested.

   The artifact file does upload (it's very similar to the file I attached to this ticket) but the tests from that results XML file do not appear on the Tests tab of the build info. Only 11 tests are visible; those are from different, much smaller, test projects in the solution. The tests associated with the results file attached to this issue (nearly 200 tests) are missing from the test results in AppVeyor's build.

   • Edit

3. Support Staff 3 Posted by Owen McDonnell on 12 Nov, 2024 10:21 PM

   

   I don't think it has to do with curly brace characters, as i can upload your results fine in this example.
   I changed the api path to nunit instead of nunit3. I wonder if there is some inconsistency in coverlets output between the different test suites.

   • Edit

4. 4 Posted by craig on 13 Nov, 2024 01:28 PM

   

   Thanks for this. Certainly using the nunit endpoint rather than nunit3 has worked and as far as I'm concerned is a perfectly acceptable workaround.

   I'm not convinced it's Coverlet to blame though; the results file which fails to upload is NUnit's own report file, not Coverlet's report. I am not uploading or attempting to upload Coverlet's report to AppVeyor. I'm not aware that running NUnit3 + Coverlet or NUnit3 alone would have any impact on the output XML report file. I certainly can't see any from a skim though.

   If it's not the curly-brace characters that are causing the error with the NUnit3 endpoint, I wonder if it might be the duration attributes. I noticed that on the troublesome tests, they have incredibly short durations; that makes sense because it's only testing an in-memory string formatting function. NUnit's report XML formatter has rendered some of these in scientfic/exponent-based number formats. For example: duration="6.2E-05". Perhaps that could be throwing your parser for a loop?

   Might this even be an NUnit bug? Their own docco indicates that duration should be "expressed as a real number". Technically that doesn't rule out scientific notation, but I wonder if that's expected behaviour.

   • Edit

5. Support Staff 5 Posted by Owen McDonnell on 15 Nov, 2024 11:04 PM

   

   It looks like you're bang on. We are not parsing exponential notation properly for nunit3 results.
   I will add this to the next build agent release. Thanks for the catch.

   • Edit