Home → Knowledge Base →

Integrating Chutzpah into AppVeyor

• Edit

johnny_reilly

03 Sep, 2014 12:27 PM

This started off as a comment on this discussion. However, I thought it might be worth breaking it out into it's own discussion as it's subtley different.

What I'm trying to do is to get AppVeyor to run my JavaScript tests using Chutzpah. Chutzpah is "an open source JavaScript test runner which enables you to run unit tests using QUnit, Jasmine, Mocha, CoffeeScript and TypeScript". I use it to run my Jasmine tests inside Visual Studio and in TFS / Visual Studio Online. (There's a couple of slightly rambly blog posts I wrote about this here and here).

I figured that given AppVeyor's .NET emphasis Chutzpah be a good way to get Jasmine / Mocha / QUnit etc JavaScript tests running as part of the build. I have a public GitHub repo set up which contains a project with Jasmine tests and Chutzpah config in:

https://github.com/johnnyreilly/Proverb/

The Jasmine tests can be found here (dependant upon the chutzpah.json in the root):

AngularTypeScript/Proverb.Web.Tests.JavaScript

These can be tested out inside Visual Studio 2012 / 2013 using the Chutzpah extension / add-in. There's a command line tool included in Chutzpah available here which also allows me to run the tests. With that I can type the following in Powershell:

PS C:\chutzpah> .\chutzpah.console.exe C:\Users\John\Documents\GitHub\Proverb\AngularTypeScript\Proverb.Web.Tests.JavaScript\

and be greeted with this:

Chutzpah console test runner (64-bit .NET 4.0.30319.34014) Version 3.2.3.0 Copyright (C) 2013 Matthew Manela (http://matthewmanela.com).

chutzpah.dll: Version 3.2.3.0

File: C:\Users\John\Documents\GitHub\Proverb\AngularTypeScript\Proverb.Web.Tests.JavaScript\app\sages\sageDetail.js 6 total, 0 failed, took 0.00 seconds

File: C:\Users\John\Documents\GitHub\Proverb\AngularTypeScript\Proverb.Web.Tests.JavaScript\app\sages\sages.js 5 total, 0 failed, took 0.00 seconds

Tests complete: 11

This output can be customised using some of the options displayed here

So the missing piece is this: how do I plug this into AppVeyors build process? I've had a read of the documentation but I'm afraid I didn't come away with any useful insights on how to achieve my aims. My guess is that I would need to package up Chutzpah inside my repo and then have AppVeyor trigger a Chutzpah run with something like this: .\chutzpah.console.exe SOME_UNKNOWN_PATH\Proverb\AngularTypeScript\Proverb.Web.Tests.JavaScript\

If anyone has any ideas I'd love to hear them!

1. 1 Posted by johnny_reilly on 04 Sep, 2014 05:09 AM

   

   I've taken this a bit further in terms of investigation. Chutzpah's console runner contains a flag to output results in junit format. So doing this:

   PS C:\Chutzpah> .\chutzpah.console.exe C:\Users\John\Documents\GitHub\Proverb\AngularTypeScript\Proverb.Web.Tests.JavaScript /junit c:\junit-with-error.xml
   

   Results in a c:\junit-with-error.xml file that contains this: (error deliberately included)

   <?xml version="1.0" encoding="UTF-8" ?>
   <testsuites>
     <testsuite name="C:\Users\John\Documents\GitHub\Proverb\AngularTypeScript\Proverb.Web.Tests.JavaScript\app\sages\sages.js" tests="5" failures="1" time="0">
       <testcase name="Proverb.Web -&gt; app-&gt; controllers -&gt; sages -&gt; on creation -&gt;:controller should have a title of &apos;Sages&apos;" />
       <testcase name="Proverb.Web -&gt; app-&gt; controllers -&gt; sages -&gt; on creation -&gt;:controller should have no sages" />
       <testcase name="Proverb.Web -&gt; app-&gt; controllers -&gt; sages -&gt; on creation -&gt;:datacontext.sage.getById should be called" />
       <testcase name="Proverb.Web -&gt; app-&gt; controllers -&gt; sages -&gt; activateController -&gt;:should set sages to be the resolved promise values" />
       <testcase name="Proverb.Web -&gt; app-&gt; controllers -&gt; sages -&gt; activateController -&gt;:should log &apos;Activated Sages View&apos;">
         <failure message="Expected spy log to have been called with [ &apos;Activated Sages View2&apos; ] but actual calls were [ &apos;Activated Sages View&apos; ].&#10;&#9;    at stack (file:///C:/Chutzpah/TestFiles/jasmine/v2/jasmine.js:1293)&#10;&#9;    at buildExpectationResult (file:///C:/Chutzpah/TestFiles/jasmine/v2/jasmine.js:1270)&#10;&#9;    at file:///C:/Chutzpah/TestFiles/jasmine/v2/jasmine.js:484&#10;&#9;    at file:///C:/Chutzpah/TestFiles/jasmine/v2/jasmine.js:260&#10;&#9;    at addExpectationResult (file:///C:/Chutzpah/TestFiles/jasmine/v2/jasmine.js:442)"></failure>
       </testcase>
     </testsuite>
     <testsuite name="C:\Users\John\Documents\GitHub\Proverb\AngularTypeScript\Proverb.Web.Tests.JavaScript\app\sages\sageDetail.js" tests="6" failures="0" time="0">
       <testcase name="Proverb.Web -&gt; app-&gt; controllers -&gt; sageDetail -&gt; on creation -&gt;:controller should have a title of &apos;Sage Details&apos;" />
       <testcase name="Proverb.Web -&gt; app-&gt; controllers -&gt; sageDetail -&gt; on creation -&gt;:controller should have no sage" />
       <testcase name="Proverb.Web -&gt; app-&gt; controllers -&gt; sageDetail -&gt; on creation -&gt;:datacontext.sage.getById should be called" />
       <testcase name="Proverb.Web -&gt; app-&gt; controllers -&gt; sageDetail -&gt; activateController -&gt;:should set sages to be the resolved promise values" />
       <testcase name="Proverb.Web -&gt; app-&gt; controllers -&gt; sageDetail -&gt; activateController -&gt;:should log &apos;Activated Sage Details View&apos; and set title with name" />
       <testcase name="Proverb.Web -&gt; app-&gt; controllers -&gt; sageDetail -&gt; gotoEdit -&gt;:should set $location.path to edit URL" />
     </testsuite>
   </testsuites>
   

   I understand that JUnit and XUnit are very similar so I'm wondering if I could upload JUnit output (as above) to AppVeyor using the XUnit flag?

   • Edit

2. Support Staff 2 Posted by Feodor Fitsner on 04 Sep, 2014 05:34 AM

   

   I'm afraid it's different from both xUnit and NUnit, but we could easily add its support in the meantime!

   • Edit

3. 3 Posted by johnny_reilly on 04 Sep, 2014 05:47 AM

   

   That would be fantastic Feodor!

   • Edit

4. 4 Posted by Jacob T. Nielse... on 04 Sep, 2014 05:23 PM

   

   @Feodor
   I would also very much appreciate if you found the time to implement the Chutzpah test runner.

   • Edit

5. Support Staff 5 Posted by Feodor Fitsner on 04 Sep, 2014 05:39 PM

   

   You mean importing JUnit test results?

   • Edit

6. 6 Posted by Jacob T. Nielse... on 04 Sep, 2014 05:49 PM

   

   I might have misunderstood were your discussion had gone. I personally have Jasmine unit tests for testing my JavaScript that I run with the Chutzpah test runner for Visual Studio.

   I would like Appveyor to be able to run these as well - is that possible?

   Kind RegardJacob

   • Edit

7. Support Staff 7 Posted by Feodor Fitsner on 04 Sep, 2014 06:36 PM

   

   Will take a look.

   • Edit

8. 8 Posted by johnny_reilly on 04 Sep, 2014 07:05 PM

   

   For what it's worth if AppVeyor supports the JUnit format for uploading XML test results then I think it should become fairly straightforward. Not tested but here's how I think it could work:

   1. Either add Chutzpah to the repo or add the Chutzpah nuget package - the second course of action is probably better.

   2. Add an "after build step" that looks something like this:

   # Run tests using Chutzpah and export results as JUnit format to chutzpah-results.xml
   $ChutzpahCmd = "$($env:APPVEYOR_BUILD_FOLDER)\packages\Chutzpah.3.2.3\tools\chutzpah.console.exe $($env:APPVEYOR_BUILD_FOLDER)\AngularTypeScript\Proverb.Web.Tests.JavaScript /junit .\chutzpah-results.xml"
   Write-Host $ChutzpahCmd
   Invoke-Expression $ChutzpahCmd
   
   # upload results to AppVeyor
   $wc = New-Object 'System.Net.WebClient'
   $wc.UploadFile("https://ci.appveyor.com/api/testresults/junit/$($env:APPVEYOR_JOB_ID)", (Resolve-Path .\chutzpah-results.xml))
   

   Obviously I'd love AppVeyor to make Chutzpah even easier to plug in but this looks like a potential approach that should work.

   @Feodor - does it sound sensible to you?

   • Edit

9. 9 Posted by johnny_reilly on 06 Sep, 2014 07:35 AM

   

   Coming on in leaps and bounds! I added the Chutzpah NuGet package to Proverb (my project) and added the following PowerShell "after build step":

   # Run tests using Chutzpah and export results as JUnit format to chutzpah-results.xml
   $ChutzpahCmd = "$($env:APPVEYOR_BUILD_FOLDER)\AngularTypeScript\packages\Chutzpah.3.2.3\tools\chutzpah.console.exe $($env:APPVEYOR_BUILD_FOLDER)\AngularTypeScript\Proverb.Web.Tests.JavaScript /junit .\chutzpah-results.xml"
   Write-Host $ChutzpahCmd
   Invoke-Expression $ChutzpahCmd
   
   Write-Host "Chutzpah results:"
   Get-Content .\chutzpah-results.xml
   
   # upload results to AppVeyor
   $wc = New-Object 'System.Net.WebClient'
   #$wc.UploadFile("https://ci.appveyor.com/api/testresults/junit/$($env:APPVEYOR_JOB_ID)", (Resolve-Path .\chutzpah-results.xml))
   

   Jasmine tests are now running using Chutzpah and results are visible in the build window. Now just need that JUnit support so the results can be hooked into the build!

   You can see an example of a successful build as describe above here: https://ci.appveyor.com/project/JohnReilly/proverb/build/1.0.11

   • Edit

10. 10 Posted by johnny_reilly on 06 Sep, 2014 12:56 PM

    

    By Jove I think I've got it!

    I've now got a powershell "Before tests script" which runs the tests using Chutzpah, writes to output to a JUnit format and then plays the results back into AppVeyor result by result:

    # Run tests using Chutzpah and export results as JUnit format to chutzpah-results.xml
    $ChutzpahCmd = "$($env:APPVEYOR_BUILD_FOLDER)\AngularTypeScript\packages\Chutzpah.3.2.3\tools\chutzpah.console.exe $($env:APPVEYOR_BUILD_FOLDER)\AngularTypeScript\Proverb.Web.Tests.JavaScript /junit .\chutzpah-results.xml"
    Write-Host $ChutzpahCmd
    Invoke-Expression $ChutzpahCmd
    
    # Upload results to AppVeyor one by one
    
    $testsuites = [xml](get-content .\chutzpah-results.xml)
    
    foreach ($testsuite in $testsuites.testsuites.testsuite) {
        write-host " $($testsuite.name)"
        foreach ($testcase in $testsuite.testcase){
            $failed = $testcase.failure
            if ($failed) {
                write-host "Failed   $($testcase.name) $($testcase.failure.message)"
                Add-AppveyorTest $testcase.name -Outcome Failed -FileName $testsuite.name -ErrorMessage $testcase.failure.message
            }
            else {
                write-host "Passed   $($testcase.name)"
                Add-AppveyorTest $testcase.name -Outcome Passed -FileName $testsuite.name
            }
    
        }
    }
    

    You can see a build with a deliberately broken test here demonstrating that both passed and failed tests are collected. The only tricky thing appears to be getting a failed test to fail the build - I can't see anything in the API docs about this.

    @Feodor - can you be of any assistance?

    By the way, I'll try and blog about this when I get a moment so the knowledge doesn't just stay with me.

    • Edit

11. 11 Posted by johnny_reilly on 06 Sep, 2014 01:14 PM

    

    Cracked it. Now failing the build when any tests fail using the PS below. See this build for an example of failure: https://ci.appveyor.com/project/JohnReilly/proverb/build/1.0.17

    # Run tests using Chutzpah and export results as JUnit format to chutzpah-results.xml
    $ChutzpahCmd = "$($env:APPVEYOR_BUILD_FOLDER)\AngularTypeScript\packages\Chutzpah.3.2.3\tools\chutzpah.console.exe $($env:APPVEYOR_BUILD_FOLDER)\AngularTypeScript\Proverb.Web.Tests.JavaScript /junit .\chutzpah-results.xml"
    Write-Host $ChutzpahCmd
    Invoke-Expression $ChutzpahCmd
    
    # Upload results to AppVeyor one by one
    
    $testsuites = [xml](get-content .\chutzpah-results.xml)
    
    $anyFailures = $FALSE
    foreach ($testsuite in $testsuites.testsuites.testsuite) {
        write-host " $($testsuite.name)"
        foreach ($testcase in $testsuite.testcase){
            $failed = $testcase.failure
            if ($failed) {
                write-host "Failed   $($testcase.name) $($testcase.failure.message)"
                Add-AppveyorTest $testcase.name -Outcome Failed -FileName $testsuite.name -ErrorMessage $testcase.failure.message
                Add-AppveyorMessage "$($testcase.name) failed" -Category Error
                $anyFailures = $TRUE
            }
            else {
                write-host "Passed   $($testcase.name)"
                Add-AppveyorTest $testcase.name -Outcome Passed -FileName $testsuite.name
            }
    
        }
    }
    
    if ($anyFailures -eq $TRUE){
        write-host "Failing build as there are broken tests"
        $host.SetShouldExit(1)
    }
    

    • Edit

12. 12 Posted by johnny_reilly on 07 Sep, 2014 04:48 AM

    

    I've also blogged about this: http://icanmakethiswork.blogspot.com/2014/09/running-javascript-uni...

    • Edit

13. Support Staff 13 Posted by Feodor Fitsner on 07 Sep, 2014 04:26 PM

    

    That's great, thank you for your support!

    • Edit

14. 14 Posted by johnny_reilly on 08 Sep, 2014 12:51 PM

    

    My pleasure Feodor!

    • Edit

15. 15 Posted by johnny_reilly on 09 Sep, 2014 10:29 AM

    

    Oh and if you wanted to use the blog content to add to your http://www.appveyor.com/docs/running-tests docs you'd be entirely welcome.

    • Edit

16. Support Staff 16 Posted by Feodor Fitsner on 09 Sep, 2014 01:43 PM

    

    Thank you!

    -Feodor

    • Edit

17. johnny_reilly closed this discussion on 31 Dec, 2014 11:24 AM.