Tests tab could better-report test results in the case that test names get repeated.

Patrick Lioi's Avatar

Patrick Lioi

03 Mar, 2016 01:56 AM

Sometimes, a test run will legitimately include more than one test result with the same reported testName value. For instance, one build script might run the same test suite twice, once targeting 32bit and once targeting 64bit. Or, some parameterized tests may report under the same name because a test framework won't always guarantee unique names for each set of input parameters.

Currently, the overall test count is correct. For example: https://ci.appveyor.com/project/plioi/fixie-runners-sandbox/build/1%20(master)/tests

Here, the overall total count of 36 test results is correct. Note, however, that there are far fewer test result lines in the table, because most of these tests were run once as 32 bit and once as 64 bit.

TeamCity handles this by showing how many times each test name was executed.

My main concern is that if two tests are reported to the API with the same name, and only one of them is a passing test, it's not clear whether AppVeyor would display one success and one failure.

  1. Support Staff 1 Posted by Feodor Fitsner on 09 Mar, 2016 02:01 AM

    Feodor Fitsner's Avatar

    Hi Patrick,

    Thanks for reporting that. Yes, I see the issue with missing tests if assemblies have the same names in the same build session.

    Well, we could bet on the fact that tests against different assemblies (same name but different architecture) are not being run in parallel and then add some suffix to "File name" field (e.g. Assembly1.dll (2)) if the same "test name - assembly" pair occurs again. I don't know if that would be looking nice though - you won't be able to distinguish between x86 and x64 results. Maybe be could add "platform" as another dimension as this problem is mostly around platform issue?

  2. 2 Posted by Patrick Lioi on 09 Mar, 2016 02:07 AM

    Patrick Lioi's Avatar

    My example shows how x86 vs x64 is one way to run into trouble, but I think it's even more likely in the real world for this to happen on parameterized tests. Test names just aren't unique as a rule, though they often are by coincidence. If the same test name gets multiple results, could the UI simply list them all?

  3. Ilya Finkelshteyn closed this discussion on 25 Aug, 2018 02:04 AM.

Comments are currently closed for this discussion. You can start a new one.

Keyboard shortcuts

Generic

? Show this help
ESC Blurs the current field

Comment Form

r Focus the comment reply box
^ + ↩ Submit the comment

You can use Command ⌘ instead of Control ^ on Mac

Recent Discussions

12 Aug, 2022 08:39 PM
11 Aug, 2022 03:44 PM
07 Aug, 2022 05:11 PM
06 Aug, 2022 11:02 AM
03 Aug, 2022 10:03 AM

 

02 Aug, 2022 02:35 PM
29 Jul, 2022 06:45 PM
29 Jul, 2022 03:35 PM
22 Jul, 2022 03:16 PM
21 Jul, 2022 10:05 PM
15 Jul, 2022 03:08 PM