Selenium Tests

scascarini's Avatar


21 Jul, 2014 03:49 PM

I'm pushing on to run my Selenium tests.

I have a web site to test and a selenium test project with the tests.

I build and deploy the test site locally. I then run the selenium tests under mstest.

The good part is that the tests run and report correctly.

The problem is that the build fails. As the tests run I get a lot of 'noise' in the console and this includes some errors from Selenium (I googled and they appear to be known bugs). These don't cause the tests to fail however they're being flagged in red in the console and I assume they are what is causing the build to fail (it passes without the tests being run). Is there some way to prevent these errors failing the build (and ideally suppress the noise).

These test all run on my local build server (under Jenkins and Mstest) without any problem. (Other than the usual Selenium flakiness :-( )


eg From the console

1405954940886 addons.manager DEBUG Application has been upgraded
11081405954940913 addons.xpi DEBUG startup
11091405954940915 addons.xpi DEBUG Skipping unavailable install location app-system-local
11101405954940915 addons.xpi DEBUG Skipping unavailable install location app-system-share
11111405954940922 addons.xpi DEBUG checkForChanges
11121405954940937 addons.xpi DEBUG Directory state JSON differs: cache [] state [{name":"app-global","addons":{"{972ce4c6-7e08-4474-a285-3208198ce6fd}":{"descriptor":"C:\Program Files (x86)\Mozilla Firefox\browser\extensions\{9
111372ce4c6-7e08-4474-a285-3208198ce6fd}","mtime":1402594368166,"rdfTime":1402020940000}}},{"name":"app-profile","addons":{"[email blocked]":{"descriptor":"C:\Users\appveyor\AppData\Local\Temp\anonymous.377c7c4fb3404453
11148085a4870d1a58e2.webdriver-profile\extensions\[email blocked]","mtime":1405954940139,"rdfTime

11151405954940952 addons.xpi-utils DEBUG Opening XPI database C:\Users\appveyor\AppData\Local\Temp\anonymous.377c7c4fb34044538085a4870d1a58e2.webdriver-profile\extensions.json
11161405954940953 addons.xpi DEBUG New add-on [email blocked] installed in app-profile
1117*** Blocklist::_loadBlocklistFromFile: blocklist is disabled
11181405954941425 addons.xpi-utils DEBUG Make addon app-profile:[email blocked] visible
11191405954941427 DeferredSave.extensions.json DEBUG Save changes
11201405954941432 DeferredSave.extensions.json DEBUG Save changes
11211405954941432 addons.xpi DEBUG New add-on {972ce4c6-7e08-4474-a285-3208198ce6fd} installed in app-global
11221405954941435 addons.xpi-utils DEBUG Make addon app-global:{972ce4c6-7e08-4474-a285-3208198ce6fd} visible
11231405954941436 DeferredSave.extensions.json DEBUG Save changes
11241405954941436 DeferredSave.extensions.json DEBUG Save changes
11251405954941452 addons.xpi DEBUG Updating database with changes to installed add-ons
11261405954941452 addons.xpi-utils DEBUG Updating add-on states
11271405954941453 addons.xpi-utils DEBUG Writing add-ons list
11281405954942492 DeferredSave.extensions.json DEBUG Starting timer
11291405954942528 DeferredSave.extensions.json DEBUG Starting write
11301405954942727 DeferredSave.extensions.json DEBUG Write succeeded
11311405954942728 addons.xpi-utils DEBUG XPI Database saved, setting schema version preference to 16
11321405954942759 addons.manager DEBUG shutdown
11331405954942760 addons.xpi DEBUG shutdown
11341405954942765 addons.xpi-utils DEBUG shutdown
11351405954942768 addons.xpi DEBUG Notifying XPI shutdown observers
11361405954942776 addons.manager DEBUG Async provider shutdown done
1138* Call to xpconnect wrapped JSObject produced this error: *
1139[Exception... "Component returned failure code: 0x8000ffff (NS_ERROR_UNEXPECTED) [nsIAnnotationService.getItemsWithAnnotation]" nsresult: "0x8000ffff (NS_ERROR_UNEXPECTED)" location: "JS frame :: resource://app/components/nsBro
1140wserGlue.js :: BG_EPDQI_runBatched :: line 1715" data: no]
1142System JS : ERROR resource://gre/modules/BookmarkHTMLUtils.jsm:897 - NS_ERROR_UNEXPECTED: Component returned failure code: 0x8000ffff (NS_ERROR_UNEXPECTED) [nsINavBookmarksService.removeFolderChildren]
1143System JS : ERROR resource://gre/modules/BookmarkHTMLUtils.jsm:897 - NS_ERROR_UNEXPECTED: Component returned failure code: 0x8000ffff (NS_ERROR_UNEXPECTED) [nsINavBookmarksService.removeFolderChildren]
11451405954943274 addons.xpi DEBUG startup
11461405954943276 addons.xpi DEBUG Skipping unavailable install location app-system-local
11471405954943276 addons.xpi DEBUG Skipping unavailable install location app-system-share
11481405954943282 addons.xpi DEBUG checkForChanges
11491405954943296 addons.xpi DEBUG No changes found
1150*** Blocklist::_loadBlocklistFromFile: blocklist is disabled
1151!!! error running onStopped callback: TypeError: callback is not a function

  1. Support Staff 1 Posted by Feodor Fitsner on 21 Jul, 2014 03:58 PM

    Feodor Fitsner's Avatar

    I've tried to disable that logging for FireFox at no avail. I suspect those messages come directly from FireFox to StdOut. Though Chrome driver seems working the best.

    Regarding "they're being flagged in red in the console" - could you please drop a screenshot of console to see how it looks like?

  2. 2 Posted by scascarini on 21 Jul, 2014 04:02 PM

    scascarini's Avatar
  3. 3 Posted by scascarini on 21 Jul, 2014 04:04 PM

    scascarini's Avatar
  4. Support Staff 4 Posted by Feodor Fitsner on 21 Jul, 2014 04:05 PM

    Feodor Fitsner's Avatar

    I see. Do you call vstest.console command as part of your build script, not using tests "autodiscovery"?

  5. 5 Posted by scascarini on 21 Jul, 2014 04:07 PM

    scascarini's Avatar

    Yes because I want to run the tests after deployment.

  6. Support Staff 6 Posted by Feodor Fitsner on 21 Jul, 2014 04:12 PM

    Feodor Fitsner's Avatar

    Perfect! The problem is that PowerShell treats everything written to StdErr as exceptions. Switch "after deploy" script to "CMD" mode - that should fix a problem.

  7. 7 Posted by scascarini on 21 Jul, 2014 04:49 PM

    scascarini's Avatar

    OK tweaked that and kicked off a new build. All looking good so far - tests are running and console is clear.

    Thanks again.

  8. 8 Posted by scascarini on 22 Jul, 2014 10:35 AM

    scascarini's Avatar

    So the tests run as 'CMD' but run very slowly. About 5 ran in 30 minutes and then the build timed out.

    Based on your comments above I did some experimenting and found that if I set the /InIsolation flag I could switch back to running under Powershell and I don't see any errors in the console.

    So I'm happy but you may want to investigate why the tests ran so slowly under 'cmd'. I was running

    vstest.console /logger:Appveyor test/bin/Debug/Spiro.Angular.Selenium.Test.dll

    as an after deployment script and it ran 5 selenium tests in 30 minutes

    Currently I'm running

    vstest.console /logger:Appveyor /InIsolation test/bin/Debug/Spiro.Angular.Selenium.Test.dll

    as Powershell and it runs about 50 selenium tests in 15 minutes or so.

    Thanks for all your help.

  9. Support Staff 9 Posted by Feodor Fitsner on 22 Jul, 2014 02:10 PM

    Feodor Fitsner's Avatar

    Wow, that's really interesting observation! I will try to understand this difference between PS and external command mode.

    Thank you for the update!

  10. 10 Posted by Rifat Erdem Sah... on 27 Nov, 2014 12:05 PM

    Rifat Erdem Sahin's Avatar

    Hello guys,

    Any updates on the issue? Why there is a time difference ?

    We are also suffering from slow acceptance tests.

  11. Support Staff 11 Posted by Feodor Fitsner on 27 Nov, 2014 07:02 PM

    Feodor Fitsner's Avatar

    I believe it's been fixed for the new environment.

    Look at that: - ran basic Selenium test against FF, Chrome and IE (!) drivers in just 47 seconds. Artifacts contain screenshots.

  12. Ilya Finkelshteyn closed this discussion on 25 Aug, 2018 01:50 AM.

Comments are currently closed for this discussion. You can start a new one.

Keyboard shortcuts


? Show this help
ESC Blurs the current field

Comment Form

r Focus the comment reply box
^ + ↩ Submit the comment

You can use Command ⌘ instead of Control ^ on Mac