Home → Questions →

Splitting large project - Usage of parallel testing

• Edit

michael

01 Feb, 2017 03:42 PM

Hi there,

I am trying to wrap my head around the concept of parallel testing with Appveyor and I am having trouble understanding how this is supposed to work, at least in my case.

One of our job is building a setup with Wix and executing all the tests of our suite (4000 ish). Some tests run for over 20 sec and the overall build time is 36 min. I was hoping that by splitting them into different categories (to enable parallel testing mode), I could optimize the overall build time.

It seems however that parallel testing runs the FULL job multiple time (a bit like a build matrix) by simply filtering out the tests that should not be executed. Our scripts however needs to install a bunch of 3rd Party dependencies before running the tests. Also our build creates a setup file. It should obviously not be created for each job. Instead it should be created only once.

Question1:
Is there a way to simply install all the dependencies (nuget packages, 3rd party tool via install build steps etc) once and run the tests in different instances of nunit (e.g. multiple processes) instead of having different jobs started?

Question2:
If this is not possible, is there a way to specify that some steps should be performed ONLY ONCE (for example setup creation).

Thanks for your feedback,

Bests,
Michael

1. 1 Posted by Ilya Finkelshte... on 02 Feb, 2017 01:49 AM

   

   Hi Michael,

   Your understanding is right, parallel tests are separate build jobs. As mentioned here, "You can split tests into jobs on the Free plan, but jobs will run in series as the Free plan allows 1 concurrent job per account."

   You can use NUnit parallel testing, with your own test script. Note that build VMs have only 2 CPU cores, so I am not sure if more than 2 parallel test process will make any improvement though. Also parallel testing is usually error prone as some tests may have common dependencies in the environment and if they say modify the same IIS website in parallel, nobody can predict anything.

   Another approach is to run single build which will create and upload artifacts, and then start (with API) another build which will download artifacts and run parallel tests though this might require you to have more parallel buds. If you want to implement this, we can help with scripting.

   Yet another approach is our new private build clouds solution, which is still expensive as you have to pay for Azure subscription. We also support GCE, AWS, Hyper-V and Docker clouds, this is just not documented yet. With this solution you can instantiate VM with bigger number of cores.

   • Edit

2. 2 Posted by michael on 06 Feb, 2017 04:28 PM

   

   Hi Ilya,

   Thanks for the explanation. Private build seems like a great solution that we will investigate in the future.

   Your approach with starting another build with the API sounds very promising. Do you have any example of how this could be done?

   For the moment, we will stick with long running jobs to be run nightly

   Bests,
   Michael

   • Edit

3. 3 Posted by Ilya Finkelshte... on 06 Feb, 2017 07:31 PM

   

   Hi Michael,

   • How to start build job. Note that you can pass environment variables, which you can use to tell build job what group of tests to run. This script can be run as part of "main" job which created all artifacts.

   • download build artifacts (or advanced version). Those scripts you can run on "child" jobs to get artifacts and start testing them

   • Another problem you may optionally need to solve is to have single source of failure. For that you can monitor "child" jobs from "main" and fail main job if at least one "child" failed. For that you may need to

     • remember build version when you start it:

       $newBuild = Invoke-RestMethod -Uri 'https://ci.appveyor.com/api/builds' -Headers $headers  -Body $body -Method POST
       $version = $newBuild.version
       

     • Test build status for every version (as you will start several builds) and throw if at least one failed:

       (Invoke-RestMethod -Uri "https://ci.appveyor.com/api/projects/<your_account>/<your_project_slug>/build/$version" -Headers $headers -Method GET).build.status
       

       You can examine this status in the loop on finish of "main" job.

   Please let us know if something looks confusing, and feel free to send your scripts to review.

   Ilya.

   • Edit

4. Ilya Finkelshteyn closed this discussion on 25 Aug, 2018 02:14 AM.