Recognizing test result output

dhickey

12 Jun, 2014 05:57 AM

Hi,

I like to use build scripts to fully encapsulate a build and maximize portability - script runs locally as it does in AppVeyor, TeamCity etc. My scripts run my tests. TeamCity has the concept of service messages ( http://confluence.jetbrains.com/display/TCD7/Build+Script+Interacti... ) to capture test results and that runners (such as xunit) support it ootb. Does AppVeyor support service messages or anything similar?

Cheers.

  1. #1 Posted by Feodor Fitsner Support staff on 12 Jun, 2014 04:27 PM

    Yes, there is REST API for real-time communication with build worker: http://www.appveyor.com/docs/build-agent-api

    MSBuild uses custom Logger (/logger:"C:\Program Files\AppVeyor\BuildAgent\Appveyor.MSBuildLogger.dll") to communicate.
    NUnit and VSTest.Console have custom logger too, and xUnit 1.9.2 installed on build worker has custom "callback" to push test results.

    Let me know what's your desired scenario or if you have any questions.

  2. #2 Posted by dhickey on 21 Jun, 2014 07:32 PM

    I'm using psake to run my builds, including running my xunit tests. This give my build script maximum portability - works same locally as it does in CI. I don't wish to use AppVeyor's test runner. How do I pipe the xunit output to the build agent?

    The projects are open source, you can see the build here https://ci.appveyor.com/project/damianh/limitsmiddleware

    (At the moment I am using AppVeyor's test runner but I want to turn that off)

  3. #3 Posted by Feodor Fitsner Support staff on 23 Jun, 2014 03:40 AM

    Right now it could be done through Build Agent API only. However, we are going to add batch upload of test results from XML for xUnit, MSTest and NUnit. Will keep you updated.

  4. #4 Posted by dhickey on 03 Jan, 2015 07:54 PM

    Hi Feodor, any update on this?

  5. #5 Posted by Feodor Fitsner Support staff on 04 Jan, 2015 01:52 AM

    Yeah, it's now possible to upload XML test results: http://www.appveyor.com/docs/running-tests#test-results

    Also, if you switch to xUnit 2.0 it has built-in AppVeyor support already.

  6. #6 Posted by emeric.fermas on 04 Jan, 2015 09:44 AM

    >Also, if you switch to xUnit 2.0 it has built-in AppVeyor support already.

    Great!

    @Feodor Could you please share the name of the Xunit 2.0 console runners, please?

  7. #7 Posted by dhickey on 04 Jan, 2015 02:34 PM

    Uploading the xml to an AppVeyor andpoint couples my build script to AppVeyor.

    Would it not be better to just be be able to tell AppVeyor though configuration to "look in this folder for test results xml"? TeamCity does this, it's simple and works well.

  8. #8 Posted by Feodor Fitsner Support staff on 04 Jan, 2015 10:45 PM

    There are %xunit20%\xunit.console and %xunit20%\xunit.console.x86.

    @dhickey - great suggestion - will add to the issues. Thank you!

  9. #9 Posted by gep13 on 20 Jan, 2015 01:42 PM

    Feodor, just wanted to touch base on this one to see if you had made any progress.

    Thanks

  10. #10 Posted by ferventcoder on 20 Jan, 2015 01:46 PM

  11. #11 Posted by dhickey on 07 Apr, 2015 02:47 PM

    I've upgraded a project to xunit 2.0 (which seems to have built in AppVeyor support https://github.com/xunit/xunit/blob/master/src/xunit.console/Visito... ). While this solves my problem I still think AppVeyor should be able to parse nunit/xunit xml output.

    I have a failing test https://ci.appveyor.com/project/damianh/limitsmiddleware/build/55#L68 https://ci.appveyor.com/project/damianh/limitsmiddleware/build/55/t... (I can't link to a test directly?)

    Any idea why the build is still green?

  12. #12 Posted by dhickey on 07 Apr, 2015 03:13 PM

    OK it appears that exit code must be non-zero.

    Teamcity can fail a build based on other output. Test failures reported for instance. Also smarter stuff such as metrics

    May I suggest that AppVeyor extends what can be considered a failing build?

  13. #13 Posted by Feodor Fitsner Support staff on 07 Apr, 2015 04:04 PM

    Sure, your suggestions are welcome!

    - Feodor

  14. #14 Posted by dennis.doomen on 06 Apr, 2016 05:34 PM

    I fully agree with Damian. I use PSake as well and don't want to tie myself to AppVeyor. TeamCity's service messages are a great way to avoid that.

  15. Ilya Finkelshteyn closed this discussion on 25 Aug, 2018 02:05 AM.

Comments are currently closed for this discussion. You can start a new one.

Keyboard shortcuts

Generic

? Show this help
ESC Blurs the current field

Comment Form

r Focus the comment reply box
^ + ↩ Submit the comment

You can use Command ⌘ instead of Control ^ on Mac

Recent Discussions