Home → Problems →

Unit tests all work on my dev machine but some fail in AppVeyor

• Edit

Pure Krome

04 Nov, 2014 11:26 AM

Heya!

like the title says, unit tests all go green on my machine, but some fail on AppVeyor.

here's a sample..

Shouldly.ChuckedAWobbly : 
    result.Query
        should be
    "?pewpew=woot&foo=bar&x=y&state=0aa44508-991a-47db-b6b4-d8edd4c0bf40&a=1&b%5E%2A%26%2F234=%26%257as%20ad7%206%2A%20SA%20"
        but was
    "?pewpew=woot&foo=bar&x=y&state=0aa44508-991a-47db-b6b4-d8edd4c0bf40&a=1&b%5E*%26%2F234=%26%257as%20ad7%206*%20SA%20"

Ok .. notice this difference :

%5E%2A%26

vs

%5E*%26

So what's happening here is that the * is getting encoded on my machine (and in this .NET Fiddle). On AppVeyor, it's not.

I've asked this question on StackOverflow but haven't gotten any luck what so ever.

I'm not sure what's going on :(

NOTE: here's the repo if you wanted to download and build on your own machine.

1. Support Staff 1 Posted by Feodor Fitsner on 04 Nov, 2014 03:36 PM

   

   Could it be the same issue with repository CRLF settings as described here: http://help.appveyor.com/discussions/problems/948-tests-fail-on-app... ?

   • Edit

2. 2 Posted by Pure Krome on 05 Nov, 2014 12:05 AM

   

   /me just read that thread.

   I don't think it does because it's not a string comparison failure where i can't see the failure difference.

   In this case, my local (and the .NET Fiddle) are encoding an asterix. On AppV, it is NOT encoding the asterix. Sounds like a .NET issue. (.NET changed this behaviour in 4.5+).

   • Edit

3. Support Staff 3 Posted by Feodor Fitsner on 05 Nov, 2014 12:20 AM

   

   Nice one. There is 4.5.2 installed.

   • Edit

4. 4 Posted by Pure Krome on 05 Nov, 2014 11:12 PM

   

   OK, so i've done a quick experiment. I'm trying to figure out what the expected result should be so we know if there is a problem or not.

   So, here's the results.

   `` Data: abcde *.(.)."

   .NET 2.0: abcde%20*.(.). .NET 3.0: abcde%20*.(.). .NET 3.5: abcde%20*.(.). .NET 4.0: abcde%20*.(.).

   -- MSDN NOTES: Bug now fixed here --

   .NET 4.5: abcde%20%2A.%28.%29. .NET 4.5.1: abcde%20%2A.%28.%29. .NET 4.5.2: abcde%20%2A.%28.%29. .NET 4.5.3: abcde%20%2A.%28.%29.

   
   So .. looking at this code .. it says that it should be encoding an asterix or parenthisis.
   
   
   
   
   AppVeyor is not.
   
   
   
   
   Which is really weird, because AV is running the test as 4.5.2 .. and under that framework, it should be encoding.
   
   
   
   
   Can you confirm/deny my current experiment?
   
   
   
   
   HINT: Use the code from this gist in a File -> New -> Console app.
   

   • Edit

5. 5 Posted by Pure Krome on 05 Nov, 2014 11:13 PM

   

   Urgh :( I can't edit my previous post :(

   • Edit

6. Support Staff 6 Posted by Feodor Fitsner on 05 Nov, 2014 11:20 PM

   

   Interesting findings. Will take a look!

   -Feodor

   • Edit

7. Support Staff 7 Posted by Feodor Fitsner on 06 Nov, 2014 07:11 PM

   

   OK, I can confirm that there is .NET 4.5.2 is installed. According to this article build workers have Release : 379893 in the registry.

   An easy experiment on AppVeyor environment using PowerShell (it's v3.0 that runs .NET 4.0) shows that it's encoding correctly (as per your notes): https://ci.appveyor.com/project/FeodorFitsner/appveyor-bot-test/bui... (lines 6,7)

   My question would be what was the command to run unit tests?

   • Edit

8. 8 Posted by Pure Krome on 06 Nov, 2014 09:01 PM

   

   the command to run unit tests? Er.. no idea :( I'm using xUnit.

   The project is OSS so here's the full AppVeyor log output and this is the branch for the code .. so you can easily clone (or fork) and then push it up to AppVeyor with your own settings to confirm stuff.

   BTW- i'm not doing anything custom - just using the defaults.

   • Edit

9. Support Staff 9 Posted by Feodor Fitsner on 06 Nov, 2014 10:28 PM

   

   Hm, that's xunit.console.clr4.x86 - .NET 4.0. Should be fine.

   I'm wondering if you could do a very simple test repo with just a single xUnit test case verifying encoding? This would greatly isolate the issue.

   • Edit

10. 10 Posted by Pure Krome on 06 Nov, 2014 10:43 PM

    

    yep. i'll try and create a new repo etc.

    • Edit

11. 11 Posted by Pure Krome on 07 Nov, 2014 12:18 AM

    

    Ok done. Error still occurs (yay!).

    Unit test results: https://ci.appveyor.com/project/PureKrome/appveyorunittest/build/1.0.1
    Fork this repo: https://github.com/PureKrome/AppVeyorUnitTest

    go nuts and figure this out :)

    NOTES: Class library has the 1 unit test method. Project is 4.5.

    • Edit

12. Support Staff 12 Posted by Feodor Fitsner on 07 Nov, 2014 12:20 AM

    

    Snap! :) will do, thanks for the repo.

    -Feodor

    • Edit

13. Support Staff 13 Posted by Feodor Fitsner on 13 Nov, 2014 05:30 AM

    

    It's definitely xUnit!

    I've played around test repo: https://ci.appveyor.com/project/FeodorFitsner/appveyorunittest

    Downloaded "clean" xunit runners from NuGet - same result, called console app - works correctly.

    Also cloned the repo to my dev machine - same result abcde%20*.(.). when running xunit tests though I have .NET 4.5.2 installed.

    I don't know if it makes sense to dig further or maybe better asking on their forums.

    • Edit

14. 14 Posted by Pure Krome on 13 Nov, 2014 06:14 AM

    

    Oh wow - interesting :) I might see if I can grab the XUnit team into this thread then...

    • Edit

15. 15 Posted by Pure Krome on 13 Nov, 2014 06:19 AM

    

    What's also interesting here @Feodor is that I don't get the xunit error when I'm running it locally on my dev .. but i've only got .NET 4.5.1 installed! I've pinged @BradWilson to see if he might be able to help.

    /me prays to the coding overlords...

    • Edit

16. 16 Posted by Brad Wilson on 13 Nov, 2014 05:00 PM

    

    There have been breaking changes in the URL encoding library across different versions of .NET.

    When writing unit tests that validate URL encoding, the "expected" value should be something that's been URL encoded by the system, rather than a hard-coded string.

    • Edit

17. 17 Posted by Pure Krome on 13 Nov, 2014 11:05 PM

    

    Hi Brad - thanks heaps for popping into this thread! (really appreciate it!)

    There have been breaking changes in the URL encoding library across different versions of .NET.

    The breaking change to the URL encoding was at version 4.5. (I think this is the link: http://msdn.microsoft.com/en-us/library/hh367887(v=vs.110).aspx ... and search for Uri.EscapeDataString).

    This thread is talking about .NET 4.5, .NET 4.5.1 and also .NET 4.5.2

    When writing unit tests that validate URL encoding, the "expected" value should be something that's been URL encoded by the system, rather than a hard-coded string.

    Um - that's what I thought we're doing in our fully-repo'd tests?

    @Feodor has a perfect repo that is highlighting that a CONSOLE APPLICATION (running under 4.5.2) works 100% perfectly (as the docs have suggested). When we use the xUnit test running, we are -not- getting the expected behaviour. This suggests that the xUnit runnner is not working as expected?

    What's weird is that the xunit runner on my dev machine (win 8.1, .NET 4.5.1 & R# .. which I used to 'grab' the running using the R# Extensions Manager) isn't doing this :/

    We don't get it :/

    • Edit

18. 18 Posted by Pure Krome on 20 Nov, 2014 03:40 AM

    

    @Feodor - I think we've lost Brad :(

    Yeah, it's a work around to do:

    var expected = Uri.EscapeDataString("abcde *.(.).");
    Assert.Equal(expected, result);
    

    but I feel that this is just covering up the real issue here. If anything, hiding the problem? @Feodor - do you agree with me, here?

    • Edit

19. Support Staff 19 Posted by Feodor Fitsner on 20 Nov, 2014 04:15 AM

    

    Yes, there is some problem.

    • Edit

20. 20 Posted by Pure Krome on 11 Jul, 2016 03:51 AM

    

    Closing issue.

    • Edit

21. Pure Krome closed this discussion on 11 Jul, 2016 03:51 AM.