Home → Questions →

Only deploy after all parallel testing branches succeed

• Edit

mail

31 Oct, 2016 12:05 PM

Hi, I'm trying out parallel testing for one of my projects using two categories "A" and "B". I also have a custom deployment script under deploy_script.

What I've found so far is the deploy script is run twice, once on each category "A" and "B", and it is run as soon as the tests for that job have completed. I've tried moving the deploy script into on_success and after_test but I get the same behaviour.

What I would like is for the deploy script to be run once and only after all parallel testing branches have succeeded. The scenario is straightforward... I would like to parallelise my testing to speed up the build, but I still only want to deploy if all tests pass. Is this possible to configure? Thanks!

1. 1 Posted by Ilya Finkelshte... on 31 Oct, 2016 05:05 PM

   

   Hi James,

   Could you please send us your appveyor.yml file?

   --ilya.

   • Edit

2. 2 Posted by mail on 31 Oct, 2016 05:17 PM

   

   Hi Ilya, I've sent it via email.

   • Edit

3. 3 Posted by Ilya Finkelshte... on 31 Oct, 2016 09:57 PM

   

   Hi James,

   Right, parallel testing creates separate build jobs, and each job has its own deployment. I see that following workaround for your situation. Although build has multiple jobs, after all jobs done, notification is being sent. So you can do the following:

   1. Create deployment environment which can be used for asynchronous deployment

   2. Create "Generic Webhook" Azure function which will start deployment from environment created in previous step.

   3. Create Webhook notification in your build which will cal function created in previous step.

   It is little bit involved, but if you are interested in this approach we can create lab and provide you with more detailed description.

   --ilya.

   • Edit

4. 4 Posted by mail on 01 Nov, 2016 03:37 PM

   

   Hi Ilya, thanks for the information. I am interested in finding out more if you could provide a more detailed description / example.

   Thanks,
   James.

   • Edit

5. 5 Posted by mail on 08 Nov, 2016 10:25 AM

   

   Hi Ilya, any update on this?

   • Edit

6. 6 Posted by Ilya Finkelshte... on 08 Nov, 2016 10:28 AM

   

   Sorry, was under water with other work, give me please a few days OK?

   • Edit

7. 7 Posted by mail on 11 Nov, 2016 03:12 PM

   

   Hi Ilya, if doing it this way is overly complicated then a potentially simpler solution would be to have a build VM with more cores available, and then we can run more tests in parallel on a single VM.

   I guess it depends on if your infrastructure is able to allocate more powerful VMs to a particular customer.

   Finally we do have a fallback option which doesn't involve using AppVeyor for the testing processor intensive tests, so don't worry if there aren't any good ways of doing this in AppVeyor at the moment. It is just a bit more work to implement which is why I was trying the AppVeyor way first.

   Cheers,
   James.

   • Edit

8. 8 Posted by Ilya Finkelshte... on 12 Nov, 2016 03:36 PM

   

   Hi James, sorry for delay, it was difficult week.

   Another customer asked very similar question, the only difference is that he needs database creation happens before all parallel testing. For some reason I proposed simpler solution to him and then I realized that this maybe good for you too!

   Please check this out:

   • Create separate AppVeyor project which will do deployment. This post can be useful.

   • One (and only one) of test jobs calls starts this second deployment project with API. Job can distinguish itself by job name which contains specific test category. This part is little fragile as you may rename test category without updating the script.

   • All jobs wait for deployment project job to finish OK. To ensure that deployment project build is exactly what we just started we will start it for current test build commit ID.

   • We have to delete webhook for deployment project on GitHub, so all changes will trigger only real test project to build.

   • We have to set deployment project to use separate YAML file, otherwise we will fall into infinite loop.

   Last two points are important to avoid unpredictable recursive behavior!

   I made a lab for DB creation scenario. I think it can be easily converted for deployment scenario, by moving logic to on_finish.

   https://github.com/IlyaFinkelshteyn/ParallelTests

   Important files are:
   • appveyor.yml

   • createDb.yml (https://ci.appveyor.com/project/IlyaFinkelshteyn/paralleltests-lo9hm set to use this file in UI settings)

   • CreateDb.ps1

   My test build logs:
   https://ci.appveyor.com/project/IlyaFinkelshteyn/paralleltests
   https://ci.appveyor.com/project/IlyaFinkelshteyn/paralleltests-lo9hm

   Please let me know if this can work for you.

   --ilya.

   • Edit

9. 9 Posted by mail on 14 Nov, 2016 12:29 PM

   

   Hi Ilya, many thanks for the example. I can see how I could adapt it for my needs, either by spawning test jobs or by configuring the jobs in the `appveyor.yml` and having one job wait for the others to complete.

   While I consider whether we want to go down this route, could you tell me if it's possible to have a build VM with more cores?

   Thanks,
   James.

   • Edit

10. 10 Posted by Ilya Finkelshte... on 15 Nov, 2016 07:33 AM

    

    Hi James,

    I also was thinking about your solution and I see the following approach:

    • Every job tries to start that second deployment project. Start script should be idempotent for job to run once. Basically it should check if it is already started.
    • Second deployment project know all test job names and wait for all them to finish successfully. After that it starts it main function, which is in this case deploy. Better for this function also to be idempotent in case it was accidentally started twice.

    Sorry I forgot to answer your question about cores. It is not actually cores but real separate VMs each of them instantiated for specific job (and yes parallel testing does full build job for every category). Unfortunately this cost money. And even this third deployment project will "eat" additional concurrent job which maybe discouraging. So maybe for you, if you are pretty confident that they finish at about the same time, it is easy to upgrade to plan with 2 concurrent jobs.

    We are working to add more orchestration, which will be more decoupled, event-based and not so conflicting with pricing model in near future.

    Thank you,
    Ilya.

    • Edit

11. 11 Posted by mail on 15 Nov, 2016 10:07 AM

    

    Hi Ilya, sorry, I think you misunderstood my question about more cores.

    I'll give the question some context. Currently the build of this project takes about an hour to complete, and the time is dominated by testing. The problem is I want to add more tests, and shorten the build time. To do this all I need is more grunt; more cores.

    I thought initially the I could just change to my `appveyor.yml` to use parallel testing, which is what started this thread, and we've now established that this involves a little more complexity. Which is fine.

    So my next question was: can we skip the additional complexity by simply add more cores to my build VM?

    Because if you could give me a VM with four cores instead of two, then I only have to change one line in my tests to `[assembly: LevelOfParallelism(4)]` to take advantage of those extra cores. Which I'm sure you'll agree is a much simpler solution :)

    So my question is really just are you able to provide build VMs with more than two cores?

    Cheers,
    James.

    • Edit

12. 12 Posted by Ilya Finkelshte... on 15 Nov, 2016 10:13 AM

    

    Ah I see. We are releasing "Private build clouds" feature soon. You will be able to connect your own Azure or Hyper-V (later others) infrastructure to AppVeyor and control VMs you want to instantiate. No ETA yet, but we are already testing it in staging environment. Would it feet your needs?

    --ilya.

    • Edit

13. 13 Posted by mail on 15 Nov, 2016 05:35 PM

    

    Ok, thanks for the information. That might work, depending on cost, complexity, etc.

    Cheers,
    James.

    • Edit

14. 14 Posted by Ilya Finkelshte... on 15 Nov, 2016 06:18 PM

    

    OK will update you soon. Hope you do not route our newsletters directly to spam folder :)

    • Edit

15. 15 Posted by mail on 21 Nov, 2016 10:30 AM

    

    Hi Ilya,

    I got around to looking at this again over the weekend, and came up with quite a simple solution based on the example you gave that doesn't involve spawning additional jobs:

    • After the first job has finished its tests, it waits for the other jobs to complete by looping through the $project.build.jobs array returned from your API (ignoring itself). If any fail, it throws an error. If they all succeed then it runs the deployment.

    • All the other jobs simply run their tests as normal, then finish.

    It seems to work well so far. Thank you very much for your help.

    Cheers,
    James.

    • Edit

16. 16 Posted by Ilya Finkelshte... on 21 Nov, 2016 09:13 PM

    

    Actually thank you for this idea!

    • Edit

17. Ilya Finkelshteyn closed this discussion on 25 Aug, 2018 02:09 AM.