Tests tab could better-report test results in the case that test names get repeated.
Sometimes, a test run will legitimately include more than one test result with the same reported testName value. For instance, one build script might run the same test suite twice, once targeting 32bit and once targeting 64bit. Or, some parameterized tests may report under the same name because a test framework won't always guarantee unique names for each set of input parameters.
Currently, the overall test count is correct. For example: https://ci.appveyor.com/project/plioi/fixie-runners-sandbox/build/1%20(master)/tests
Here, the overall total count of 36 test results is correct. Note, however, that there are far fewer test result lines in the table, because most of these tests were run once as 32 bit and once as 64 bit.
TeamCity handles this by showing how many times each test name was executed.
My main concern is that if two tests are reported to the API with the same name, and only one of them is a passing test, it's not clear whether AppVeyor would display one success and one failure.
Comments are currently closed for this discussion. You can start a new one.
Keyboard shortcuts
Generic
| ? | Show this help |
|---|---|
| ESC | Blurs the current field |
Comment Form
| r | Focus the comment reply box |
|---|---|
| ^ + ↩ | Submit the comment |
You can use Command ⌘ instead of Control ^ on Mac
Support Staff 1 Posted by Feodor Fitsner on 09 Mar, 2016 02:01 AM
Hi Patrick,
Thanks for reporting that. Yes, I see the issue with missing tests if assemblies have the same names in the same build session.
Well, we could bet on the fact that tests against different assemblies (same name but different architecture) are not being run in parallel and then add some suffix to "File name" field (e.g.
Assembly1.dll (2)) if the same "test name - assembly" pair occurs again. I don't know if that would be looking nice though - you won't be able to distinguish between x86 and x64 results. Maybe be could add "platform" as another dimension as this problem is mostly around platform issue?2 Posted by Patrick Lioi on 09 Mar, 2016 02:07 AM
My example shows how x86 vs x64 is one way to run into trouble, but I think it's even more likely in the real world for this to happen on parameterized tests. Test names just aren't unique as a rule, though they often are by coincidence. If the same test name gets multiple results, could the UI simply list them all?
Ilya Finkelshteyn closed this discussion on 25 Aug, 2018 02:04 AM.