tag:help.appveyor.com,2012-11-13:/discussions/questions/390-running-jasmine-on-appveyorAppVeyor: Discussion 2018-10-19T08:16:37Ztag:help.appveyor.com,2012-11-13:Comment/336777062014-07-07T14:03:56Z2014-07-07T14:03:56ZRunning jasmine on appveyor<div><p>There is REST API for sending test results to AppVeyor: <a href=
"http://www.appveyor.com/docs/build-agent-api">http://www.appveyor.com/docs/build-agent-api</a>
and currently xUnit, nUnit, MSpec and MSTest are integrated.</p>
<p>It's either phantomjs should be extended or XML results are
parsed (if it's able generating it).</p></div>Feodor Fitsnertag:help.appveyor.com,2012-11-13:Comment/336777062014-07-17T11:25:17Z2014-07-17T11:25:19ZRunning jasmine on appveyor<div><p>Hello Feodor,</p>
<p>Regarding this, we had some issues using the REST API - we were
running our tests on post build, and hitting the API to report all
our tests results. However, only a 1/3 of the tests were showing up
on AppVeyor because it seems as soon as the post build script
finishes, AppVeyor moves on to Testing stage and stops processing
the API requests. I had to add in a generous sleeping period to
make sure all the requests were processed, to stop it moving on to
the Test stage.</p>
<p>Is there a better way to do this?</p></div>Joao Inaciotag:help.appveyor.com,2012-11-13:Comment/336777062014-07-17T11:41:09Z2014-07-17T11:41:09ZRunning jasmine on appveyor<div><p>Hi Joao,</p>
<p>Do you make synchronous calls to REST API or async?</p></div>Feodor Fitsnertag:help.appveyor.com,2012-11-13:Comment/336777062014-07-17T12:41:58Z2014-07-17T12:41:59ZRunning jasmine on appveyor<div><p>They are async.<br>
Also, is there a way we can do a call for multiple tests results
(to batch them together) instead of one per each?</p></div>Joao Inaciotag:help.appveyor.com,2012-11-13:Comment/336777062014-07-17T12:43:37Z2014-07-17T12:43:37ZRunning jasmine on appveyor<div><p>Not right now, but we could implement that (don't think it's
hard). Would supporting NUnit and xUnit formats be sufficient?</p></div>Feodor Fitsnertag:help.appveyor.com,2012-11-13:Comment/336777062014-07-17T13:02:07Z2014-07-17T13:02:09ZRunning jasmine on appveyor<div><p>ATM we're using simple REST API posts, with json as the
body:<br></p>
<pre>
<code>jQuery.ajax({
type: 'POST',
url: this.reportBaseUrl + 'api/tests',
data: {
"testName": testName,
"testFramework": "jasmine",
"fileName": "jasmine",
"outcome": passedOutcome(passed),
"ErrorMessage": "",
"ErrorStackTrace": stackTrace,
"StdOut": "",
"StdErr": ""
}
});</code>
</pre>
Just wondering if it could me more efficient if we bundled the
calls in a few of them, instead of 1 per each test.
<p>Do you reckon I should execute the ajax calls in sync mode?
Won't that slow down the whole process for a considerable time?</p></div>Joao Inaciotag:help.appveyor.com,2012-11-13:Comment/336777062014-07-17T13:18:07Z2014-07-17T13:18:07ZRunning jasmine on appveyor<div><p>It's slowing down any way: either running them in sync or adding
a sleep like you did.</p>
<p>Batches could work though my concern is batch size. There is a
maximum message size limitation that could be sent from build
worker to AppVeyor. I mean you should be aware that sending a batch
could fail if contains a lot of data.</p>
<p>Another way is uploading combined test results report to some
URL in XML or JSON format. Very large data set could be uploaded
though no real-time log update.</p>
<p>What do you think?</p></div>Feodor Fitsnertag:help.appveyor.com,2012-11-13:Comment/336777062014-07-17T13:34:08Z2014-07-17T13:34:10ZRunning jasmine on appveyor<div><p>The more flexibility the better as far as I'm concerned - even
if you don't get the real time update, as long as you have them
accounted for in the end and know if they all passed / what failed,
I'm all for it.</p>
<p>I will give the sync calls a go for now and see if it improves
the total run time of the jasmine tests, cheers.</p></div>Joao Inaciotag:help.appveyor.com,2012-11-13:Comment/336777062014-07-17T13:37:27Z2014-07-17T13:37:27ZRunning jasmine on appveyor<div><p>OK, we will implement batches first, then uploading XMLs.</p></div>Feodor Fitsnertag:help.appveyor.com,2012-11-13:Comment/336777062014-07-27T14:35:18Z2014-07-27T14:35:18ZRunning jasmine on appveyor<div><p>Great news! We've just deployed an update with batch updates and
XML support!</p>
<p>How to import XML test results during the build:<br>
<a href=
"http://www.appveyor.com/docs/running-tests#test-results">http://www.appveyor.com/docs/running-tests#test-results</a></p>
<p>Build Worker API for batch test updates:<br>
<a href=
"http://www.appveyor.com/docs/build-worker-api#add-tests">Add
tests</a> and <a href=
"http://www.appveyor.com/docs/build-worker-api#update-tests">Update
tests</a> topics updated with new endpoints.</p>
<p>Let me know how that works for you!</p></div>Feodor Fitsnertag:help.appveyor.com,2012-11-13:Comment/336777062014-08-06T16:32:25Z2014-08-06T16:32:26ZRunning jasmine on appveyor<div><p>Hi Feodor,</p>
<p>Following up on this but slightly unrelated to the original
question from my colleagues<br>
- we are able to detect if our phantom test runner captured any
failed tests. I would then like to fail the build but it continues
to the next part of the build step (.Net Unit Tests) and if that
goes ok we get a successful build.<br>
Currently I am trying to fail with something like this:</p>
<p>console.error(someErrorMessage);<br>
phantom.exit(1);</p>
<p>this may be more of a question about phantomjs so apologies. But
any guidance would be appreciated. Is there something I can report
using the REST API that would call this a failed build?</p>
<p>Thanks again, Daniel</p></div>Dan Jonestag:help.appveyor.com,2012-11-13:Comment/336777062014-08-06T16:32:53Z2014-08-06T16:32:53ZRunning jasmine on appveyor<div><p>I'll be out of the office until the 11th of August. Any urgent
queries please e-mail <a href=
"mailto:conor.malone@adazzle.com">conor.malone@adazzle.com</a>.</p>
<p>Thank you,<br>
Joao Inacio</p></div>Joao Inaciotag:help.appveyor.com,2012-11-13:Comment/336777062014-08-06T23:47:39Z2014-08-06T23:47:39ZRunning jasmine on appveyor<div><p>Hi Daniel,</p>
<p>Exit code is the best soltuion in the most cases. That's strange
that it's not passing it. Are you calling Phantom runner as
<code>ps:</code> or <code>cmd:</code>? It definitely should work
with <code>cmd:</code>.</p>
<p>Alternatively, in PS mode you can use
<code>$host.SetShouldExit(1)</code></p></div>Feodor Fitsnertag:help.appveyor.com,2012-11-13:Comment/336777062014-08-07T13:43:44Z2014-08-07T13:43:45ZRunning jasmine on appveyor<div><p>Hi Feodor,</p>
<p>The solution to this was quite simple, and as you said the exit
code should have been respected...I changed this</p>
<p>if (failures > 0) console.error(failBuildMessage);
phantom.exit(1); } phantom.exit(0);</p>
<p>to this</p>
<p>if (failures > 0) {<br>
console.error(failBuildMessage); phantom.exit(1); } else {
phantom.exit(0); }</p>
<p>thinking that the first exit would stop code execution but the
second exit with 0 error code still ran.</p></div>Dan Jonestag:help.appveyor.com,2012-11-13:Comment/336777062014-08-07T13:48:40Z2014-08-07T13:48:40ZRunning jasmine on appveyor<div><p>Cool, thanks for the update!</p></div>Feodor Fitsnertag:help.appveyor.com,2012-11-13:Comment/336777062014-08-07T14:17:18Z2014-08-07T14:17:18ZRunning jasmine on appveyor<div><p>Would be cool if you could share your experience for running
jasmine tests on AppVeyor with community ;) We could an article in
our docs then.</p></div>Feodor Fitsnertag:help.appveyor.com,2012-11-13:Comment/336777062014-08-07T14:22:15Z2014-08-07T14:22:18ZRunning jasmine on appveyor<div><p>Of course - I'll put something together and post back later.</p></div>Dan Jonestag:help.appveyor.com,2012-11-13:Comment/336777062014-09-03T10:41:26Z2014-09-03T12:30:55ZRunning jasmine on appveyor<div><p>I'd be very interested in how this turns out. I've been using
<a href=
"https://chutzpah.codeplex.com/">https://chutzpah.codeplex.com/</a>
to run my Jasmine tests inside Visual Studio and in TFS / Visual
Studio Online. (There's a slightly rambly blog post I wrote on this
here: <a href=
"http://icanmakethiswork.blogspot.co.uk/2014/03/the-surprisingly-happy-tale-of-visual.html">
http://icanmakethiswork.blogspot.co.uk/2014/03/the-surprisingly-hap...</a>
)</p>
<p>Given AppVeyor's .NET emphasis would Chutzpah be a good way to
get Jasmine / Mocha / QUnit etc JavaScript tests running as part of
the build? Perhaps someone has mentioned this already? I'd love to
know!</p>
<p>FWIW there are details about the Chutzpah command line runner
<a href=
"https://chutzpah.codeplex.com/wikipage?title=runTests">here</a>
and <a href=
"https://chutzpah.codeplex.com/wikipage?title=Command%20Line%20Options&referringTitle=Documentation">
here</a> and it supports the following JavaScript testing
frameworks:<br>
- QUnit - Jasmine - Mocha</p>
<p>In case it helps, here's an example of a solution with a
separate Jasmine JavaScript test project included (using
Chutzpah):</p>
<p><a href=
"https://github.com/johnnyreilly/Proverb/">https://github.com/johnnyreilly/Proverb/</a></p>
<p>The Jasmine tests can be found here (dependant upon the
<code>chutzpah.json</code> in the root):</p>
<p>AngularTypeScript/Proverb.Web.Tests.JavaScript</p>
<p>You can test this out inside Visual Studio 2012 / 2013 using the
Chutzpah extension / add-in</p>
<p>Since this has become quite a long comment on a slightly
different area from the original topic I've broken this into a
<a href=
"http://help.appveyor.com/discussions/questions/495-integrating-chutzpah-into-appveyor">
separate discussion</a>.</p></div>johnny_reillytag:help.appveyor.com,2012-11-13:Comment/336777062014-09-03T10:51:50Z2014-09-03T10:51:59ZRunning jasmine on appveyor<div><p>Apologies for the delay - I do have a rather sketchy series of
notes on what we did. I will tidy up and post back...</p></div>Dan Jonestag:help.appveyor.com,2012-11-13:Comment/336777062014-09-03T11:33:17Z2014-09-03T11:33:25ZRunning jasmine on appveyor<div><p>Thanks Dan. I'm an AppVeyor noob so any guidance on how to make
the 2 play nice is appreciated!</p></div>johnny_reillytag:help.appveyor.com,2012-11-13:Comment/336777062014-09-03T14:31:13Z2014-09-03T14:31:16ZRunning jasmine on appveyor<div><p>It's still a bit sketchy, but basically:</p>
<p>The general need is this, I'm sure it's familiar...</p>
<p>We want to run builds on each pull request and have a Github
hook to send a pay load when a pull request is synchronised<br>
This kicks off a build of the pull request branch.<br>
We want to get an early indication of the status of that code.
There are several indicators some from the build server, some
manual checks recorded in GitHub and other systems<br>
Code Review status + Code builds ok + Unit Tests not broken +
Jasmine tests not broken ( + Browser Tests + Manual BAT + Manual QA
process etc)<br>
We also have a process to build a selection of pull requests and
deploy to staging server to get an indication of the status of all
the code together (where BAT and QA take place)</p>
<p>The issue was that we needed Jasmine tests firstly reporting
into the Tests Build output.<br>
Then we require that a test failure actually failed the build as
.Net Unit tests did rather than just carrying on and reporting a
successful build in the pull request</p>
<p>Basic Solution (amended code - assumes knowledge of running
jasmine reporters via javascript)</p>
<p>General flow - start phantom via build step cmd pointing to
testRunner.html and testRunner.js<br>
testRunner.html - includes script references<br>
jasmine appVeyor Reporter script which hooks into jasmine events
when we get a test result and logs consle messages<br>
testRunner.js - kicks things off and deals with total results.
Fails build if necessary</p>
<p>page.open(url, function (status) {<br>
//Page is loaded! if (status !== 'success') { console.log('Unable
to load the address!'); phantom.exit(1); } else { // Run our code
here console.log('Initializing tests from phantom...');
page.evaluate(function (appVeyorUrl) { //Custom reporter script
which hooks into necessary events and reports on status var
tcReporter = new jasmine.AppVeyorReporter(appVeyorUrl); var
oldCallback = tcReporter.reportRunnerResults;
tcReporter.reportRunnerResults = function (runner) {
oldCallback.apply(this, arguments); this.log('##jasmine.complete');
//NOTE - this is used in testRunner.js }
jasmine.getEnv().addReporter(tcReporter);
jasmine.getEnv().addReporter(new jasmine.TrivialReporter());
//kicks it all off - all describes must have been run by now..
//TODO have some form of event so we know all tests have been
registered //for now, just use a good old timeout
setTimeout(function () { jasmine.getEnv().execute(); }, 3000); },
reportBaseUrl); //Using a delay to make sure the JavaScript is
executed in the browser window.setTimeout(function () {
page.render("output.png"); //if the test runner never got started
for some reason quite after 20 seconds if (!anyLogs) { //fail the
build console.log("status - phantom test runner did not report any
results after 20 secs"); phantom.exit(1); }</p>
<pre>
<code> }, 20000);</code>
</pre>
<p>} });</p>
<p>We log jasmine messages with a custom script which essentially
just logs to the console when it gets a result back from the
jasmine test.<br>
We log ##jasmine.failed, ##jasmine.passed, ##jasmine.complete and
the onConsoleMessage that phantom calls into processes the current
status.<br>
We also report this to the tests build output in appveyor via:</p>
<p>//Inside our jasmine.appVeyorReporter script when we get a test
result jQuery.ajax({ type: 'POST', url: this.reportBaseUrl +
'build/compilationmessages', data: {</p>
<pre>
<code> "message": passed ? "Passed " + testName : "Failed " + testName,
"category": passed ? "information" : "error",
"details": passed ? "" : stackTrace,
"fileName": testName,
"line": "",
"column": "",
"projectName": "jasmine",
"projectFileName": ""
},
// Feodor suggestion to avoid waiting for requests: http://help.appveyor.com/discussions/questions/390-running-jasmine-on-appveyor
async: false
});</code>
</pre>
<p>We hook into the phantom page.OnConsoleMessage function to
process each message – counting failures and successes or
indicating whether we have finished.<br>
If we have finished we check if there are any failures and exit
phantom with the appropriate exit code (1 for a failed build
otherwise 0).<br>
AppVeyor cmd build step respects this and fails our build if
necessary</p>
<p>page.onConsoleMessage = function (msg) {<br>
if (msg) { anyLogs = true; if (msg.indexOf('##jasmine.passed') !==
-1) { testCount++; }</p>
<pre>
<code> if (msg.indexOf("##jasmine.failed") !== -1) {
console.log(msg);
failures++;
}
else if (!onlyReportFailures) {
console.log(msg);
}
if (msg.indexOf("##jasmine.complete") !== -1) {
console.log(msg);
failures > 0 ? console.log('!! Finished with ' + failures + ' failures out of ' + testCount + ' tests !!') : console.log('Win! Completed ' + testCount + ' tests with no failures.');
if (failures > 0) {
var failBuildMessage = "There are failing jasmine tests. I'm gonna have to fail this build. Fix the tests and come back later.";
console.error(failBuildMessage);
phantom.exit(1);
}
else {
phantom.exit(0);
}
}
}</code>
</pre>
<p>};</p>
<p>And our appVeyor build step cmd script:</p>
<p>cd [buildFolder][pathToTestRunnerPage.html]<br>
[buildFolder][pathToPhatom]\phantomjs.exe.1.8.0\phantomjs.exe
testrunner.appveyor.js testrunner.html false %APPVEYOR_API_URL%</p>
<p>That's it. It's not a complete coded solution - but hopefully
you get the idea</p></div>Dan Jonestag:help.appveyor.com,2012-11-13:Comment/336777062014-09-03T19:29:55Z2014-09-03T19:29:56ZRunning jasmine on appveyor<div><p>Hi Dan,</p>
<p>Thanks for the very fleshed out answer - that's really helpful!
I'm not at a computer to try this out at present but I mean to.
I've a couple of questions if you'd be so good:</p>
<ol>
<li>
<p>You're invoking Phantom to run your Jasmine tests - I'm guessing
the phantomjs.exe is part of your repo? I ask as Chutzpah similarly
includes phantom and my ideal scenario would being able to use
Chutzpah both in visual studio and in appveyor so I don't have to
have duplicate config etc. If you're invoking your tests by running
phantom in your repo it gives me hope I might be able to do similar
with Chutzpah. And if that fails then trying your approach is
clearly a good option.</p>
</li>
<li>
<p>Your script samples certainly give the gist of how things fit
together. Could be a little clearer on the contents of
testrunner.js and the jasmine appVeyor Reporter script please? I'm
not sure which script samples go into which files? Or if the
jasmine appVeyor Reporter script is something else entirely? If
you'd be able to share these files in their entirety that would be
wonderful. Your call of course.</p>
</li>
</ol>
<p>Thanks for all the help so far.</p>
<p>Best<br>
John</p></div>johnny_reillytag:help.appveyor.com,2012-11-13:Comment/336777062014-09-04T11:42:28Z2014-09-04T11:42:36ZRunning jasmine on appveyor<div><p>Sure, To clarify -</p>
<p>we consume phantomjs.exe as a nuget package. We exclude from the
github repository but it is restored on build and then referenced
as a build step in that path so
path_to_installed_packages\phatnomjs\phantomjs.exe</p>
<p>The scripts:<br>
testRunner.js simply defines the phantomjs objects/functions:</p>
<p>so</p>
<p>var args = require("system").args;<br>
//validate the correct args are passed in</p>
<p>page.settings.localToRemoteUrlAccessEnabled = true; //prevents
cross origin issues when loading local resources ie tmpls<br>
//This is required because PhantomJS sandboxes the website and it
does not show up the console messages form that page by default</p>
<p>page = new WebPage()<br>
page.onConsoleMessage = function (msg) { ]<br>
//process reporter result //keep a count of failures, successes,
total tests etc //if we have finished all the specs log one more
console message with the //results and phantom.exit(0) or
phantom.exit(1) as necessary } page..open(url, function (status)
{<br>
//setup reporter and kick off test execution var tcReporter = new
jasmine.AppVeyorReporter(appVeyorUrl);<br>
tcReporter.reportRunnerResults = function (msg){
this.log('##jasmine.complete'); //tell page.onConsoleMessage that
we have //finished } jasmine.getEnv().addReporter(tcReporter);<br>
jasmine.getEnv().execute();<br>
}</p>
<p>The custom reporter is setup and certain functions will be
called into when the tests are executing most importantly for
us:</p>
<p>reportSuiteResults: function (suite) {<br>
var results = suite.results();</p>
<p>for (var i = 0, ilen = results.items_.length; i < ilen; i++)
{ { var spec = results.items[i] var passed = true; for (var j = 0,
jlen = spec.items_.length; j < jlen; j++) { var result =
spec.items_[j]; passed = passed && result.passed_; } if
(!passed ) {</p>
<pre>
<code> //then log to appveyor api test messages
jQuery.ajax({
type: 'POST',
url: this.reportBaseUrl + 'build/compilationmessages',
data: {
...
},
async: false
});
//this will cause the page.onConsoleMessage above to be called
jasmine.getGlobal().console.log("##jasmine.failed " + spec.description)
}</code>
</pre>
<p>} }</p>
<p>Hope this makes sense!</p></div>Dan Jonestag:help.appveyor.com,2012-11-13:Comment/336777062014-09-04T11:44:28Z2014-09-04T11:44:29ZRunning jasmine on appveyor<div><p>and to clarify further testRunner.html must reference the
testReporter script and of course all other scripts required during
the actual testing</p></div>Dan Jonestag:help.appveyor.com,2012-11-13:Comment/336777062014-09-04T12:44:44Z2014-09-04T12:44:44ZRunning jasmine on appveyor<div><p>Thanks Dan - it pretty much does. <a href=
"http://help.appveyor.com/discussions/questions/495-integrating-chutzpah-into-appveyor">
I'm going to try the Chutzpah route first and having done a little
digging I think it might work quite well</a>. If it doesn't come
off then I'm going to give your approach a try.</p>
<p>Actually your approach has an advantage over Chutzpah in that
you can get test by test results whereas Chutzpah will only allow
batch results.</p>
<p>There is a Chutzpah NuGet package available so I might follow
your lead and include the Chutzpah NuGet package as you've included
the Phantom one.</p></div>johnny_reillytag:help.appveyor.com,2012-11-13:Comment/336777062014-09-04T13:06:25Z2014-09-04T13:06:25ZRunning jasmine on appveyor<div><p>Here's a question; are you setting this up as a Before tests
script, After tests script, Before build script or After build
script?</p></div>johnny_reillytag:help.appveyor.com,2012-11-13:Comment/336777062014-09-04T13:52:21Z2014-09-04T13:52:23ZRunning jasmine on appveyor<div><p>After Build.<br>
So our steps are currently</p>
<p>Before build script : nuget restore<br>
Build<br>
After Build Script : jasmine tests + Unit Tests</p>
<p>The tests section of our project settings is actually off - we
do it all in the after build scripts.</p></div>Dan Jonestag:help.appveyor.com,2012-11-13:Comment/336777062014-09-04T14:04:16Z2014-09-04T14:04:16ZRunning jasmine on appveyor<div><p>Nice - I'll give that a try when I get to a machine. Time to
dust down my rusty powershell skills...</p></div>johnny_reillytag:help.appveyor.com,2012-11-13:Comment/336777062014-09-06T11:31:40Z2014-09-07T04:47:27ZRunning jasmine on appveyor<div><p>Hi Dan,</p>
<p>Just wanted to let you know that following your help I've been
making some progress with Chutzpah and Jasmine. You can see how far
I've got <a href=
"http://help.appveyor.com/discussions/questions/495-integrating-chutzpah-into-appveyor#comment_34474204">
here</a> - Thanks for all your help so far!</p>
<p>I've also blogged about it: <a href=
"http://icanmakethiswork.blogspot.com/2014/09/running-javascript-unit-tests-in-appveyor.html">
http://icanmakethiswork.blogspot.com/2014/09/running-javascript-uni...</a></p>
<p>Best,<br>
John</p></div>johnny_reilly