Home → Problems →

Constant build failures

• Edit

odinserj

05 Sep, 2014 10:29 AM

Hello Feodor. I'm experiencing constant build failures due to different reasons. All of these problems started to appear approximately in july (perhaps after Scott Hanselman's effect). Before that build failures always reflected problems with my project itself.

1. Problems with uploading of xUnit test runner results:

Exception has been thrown by the target of an invocation. ---> System.Net.WebException: The operation has timed out
290  
291  Server stack trace: 
292     at System.Net.WebClient.UploadDataInternal(Uri address, String method, Byte[] data, WebRequest& request)
293     at System.Net.WebClient.UploadData(Uri address, String method, Byte[] data)
294     at Appveyor.BuildAgent.Api.BuildServices.UpdateTest(String testName, String testFramework, String fileName, TestOutcome outcome, Nullable`1 durationMilliseconds, String errorMessage, String errorStackTrace, String stdOut, String stdErr) in c:\Projects\Appveyor\appveyor\Appveyor.BuildAgent.Api\BuildServices.cs:line 122
295     at Xunit.ConsoleClient.AppveyorRunnerCallback.TestPassed(TestMethod method, TestPassedResult result)
296     at Xunit.ConsoleClient.RunnerCallback.TestFinished(TestMethod method)
297     at Xunit.ExecutorWrapper.XmlNodeCallbackHandlerWithICallbackEventHandler.RaiseCallbackEvent(String result)

1. Build job fails after 30 minutes timeout. However, average build time for successful builds is about 8 minutes. I can't understand what happens - sometimes build hangs on integration test, sometimes on unit tests, sometimes it does not hang and completes successfully. As I said earlier, there were no problems in April (when I started to use this awesome CI Server), May and June. But after these monthes, there were too many build failures caused by AppVeyor itself.

Thanks in advance, here is the link - https://ci.appveyor.com/project/odinserj/hangfire-525

1. Support Staff 1 Posted by Feodor Fitsner on 05 Sep, 2014 07:34 PM

   

   Hi Sergey,

   Right, this is not good. Thanks for letting me know - I'm going to see what's going on. Our #1 priorities right now are stability and performance!

   • Edit

2. Support Staff 2 Posted by Feodor Fitsner on 11 Sep, 2014 03:29 AM

   

   Hi Sergey,

   I spent some time this week trying to identify the cause of failing builds of Hangifre.

   At the first place I though it might be something with agent API sporadically failing to send data to AppVeyor. However, running simple tests with a lot of test cases went smoothly.

   Then I noticed that xUnit process consumes enormous amount of memory while running tests in Hangfire.Core.Tests.dll which was strange as these tests are basically in-memory tests and don't talk to SQL or Redis or MSMQ. On my local machine xUnit process took roughly 900 MB by the end (see attached screenshot).

   More interesting, when running the same tests in AppVeyor environment and sending real-time test results memory consumption is more severe - xUnit process takes all available memory and CPU becomes 100% slowing the build down to zero - that's why it hits 30 minutes cap.

   I gave it a try with ANTS Memory Profiler to see what's going on. I think it's Moq and particularly Castle proxies are eating the memory. See another screenshot attached.

   Hope that helps.

   • xunit-perf-1.png 13.7 KB
   • xunit-moq.png 93.3 KB
   • Edit

3. 3 Posted by odinserj on 12 Sep, 2014 02:54 PM

   

   Hi, Feodor! I'm really impressed with your investigation. It is unbelievable for me to receive such feedback on a support issue, especially from a busy man like you. You showed me the quality and I very appreciate it.

   I'm very ashamed that I didn't do it myself, and you are right, the problem was related to Moq library's mock objects. I was not aware that it creates a new instance for every mocked method invocation instead of counting invocations in a grouped manner.

   After fixing this all is working fine. Hope my project will not eat project VMs anymore.

   • Before.png 50.1 KB
   • After.png 42.1 KB
   • Edit

4. odinserj closed this discussion on 12 Sep, 2014 03:02 PM.