Build triggered by multiple repositories
Hi,
I'm trying to setup CI builds for our main product. Instead of one single repository we have 20 different repositories, some hosted on Github and some on Bitbucket. Ideally a change in any of these repositories should trigger a complete build.
Unless I'm mistaken there is no built-in support for this type 'multi-repository-projects' on Appveyor so what i did now:
- create a new repository 'buildtrigger'
- add an appveyor,yml to the repository which clones all repositories and builds
- create an Appveyor project for the repository
- configure all repositories to call Appveyor's webhook
Tested this and manual builds can be started, and pushes to the 'buildtrigger' repository initiate a build as well.
Pushes to the other repositories however resulted in a response "Push request and project repositories do not match." Another thread on this forum had the solution to use https://ci.appveyor.com/api/git/webhook?id= urls instead of https://ci.appveyor.com/api/github/webhook?id= or https://ci.appveyor.com/api/bitbucket/webhook?id= which I was using. Now pushes to other repositories start a build as well, but the build doesn't completely run: only the clone step executes. The output is like:
Build started
git clone -q --branch=master https://bitbucket.org/<team>/buildtrigger.git C:\projects\buildtrigger
Discovering tests...OK
Build success
and that's it. I assume there is some mismatch in the payload as it is for another repository.
Can I do anything about this? Is there another solution to get builds triggered by multiple repositories?
Comments are currently closed for this discussion. You can start a new one.
Keyboard shortcuts
Generic
| ? | Show this help |
|---|---|
| ESC | Blurs the current field |
Comment Form
| r | Focus the comment reply box |
|---|---|
| ^ + ↩ | Submit the comment |
You can use Command ⌘ instead of Control ^ on Mac
1 Posted by Ilya Finkelshte... on 20 Dec, 2016 09:59 AM
Hello,
I did not try this approach, but I believe you are getting this "only clone" build because AppVeyor does not even try to download
appveyor.ymlfrom classic Git repository as classic Git does not support download individual files.You have number of options here:
Hope this helps.
Ilya.
2 Posted by Stijn on 20 Dec, 2016 10:36 AM
Hello,
thanks for the reply. Can you clarify some points?
> AppVeyor does not even try to download appveyor.yml from classic Git repository
> Use UI (not sure you like it)
> Use not fully documented but working download YAML over HTTP feature
I'm not sure what you mean here: the 'buildtrigger' repository is hosted on bitbucket and does contain appveyor.yml. So after the log shows it's cloned there should be an appveyor.yml, no?
Though your suggested solutions suggest otherwise; care to shed some light on how this works internally then?
3 Posted by Ilya Finkelshte... on 20 Dec, 2016 10:50 AM
Full clone happens on build VM when build configuration/scenario already settled down, and even if appveyor.yml exist in that cloned folder it is already too late to read build configuration/scenario from it. Clone step itself is part of already formed scenario.
Configuration/scenario is being formed on central servers, before build VM picked up a build job. At that moment full clone does not happen, only individual YAML file is being downloaded, but only if repository type supports it. Classic Git does not. If you use
gitinstead ofgithuborbitbucketin webhook URL, appveyor thinks that it is classic Git and does not try to download this file, and use what you set in UI for configuration.Hope this makes sense. But please ask additional questions if needed :)
4 Posted by Ilya Finkelshte... on 20 Dec, 2016 10:53 AM
Also I noticed that maybe markdown link did not appear in your email - this https://github.com/appveyor/ci/issues/1089 is feature I recommend you to try.
5 Posted by Stijn on 20 Dec, 2016 11:02 AM
Thanks, that was an insightful read.
The meat of the actual build is all in seperate scripts hence the actual Appveyor YAML is only a couple of lines so I'll try the UI first. Otherwise the YAML over http seems indeed the way to go then. I'll let you know things work out!
6 Posted by Stijn on 20 Dec, 2016 01:26 PM
Ok tried with the settings in the UI and builds can be triggered from anywhere using the classic Git webhook. Thanks again for providing all info so quickly.
One last question: is there a way to access the webhook payload or another way to figure out which repository's push/commit initiated the build?
7 Posted by Ilya Finkelshte... on 20 Dec, 2016 06:58 PM
You are welcome!
I believe you can use APPVEYOR_REPO_NAME environment variable, or other APPVEYOR_REPO_* variables described here https://www.appveyor.com/docs/environment-variables/.
Please let us know if this works for you.
Ilya.
8 Posted by Stijn on 21 Dec, 2016 10:24 AM
Yeah i was hoping the APPVEYOR_REPO_ variables would work, but for builds triggerred by the git webhook they still contain the repo/commit info from the last commit of the repository itself, not from the commit which triggered the webhook.
Unless you know of another way to access the webhook's payload I'm going to resort to using git log to figure out the last commit instead.
9 Posted by Ilya Finkelshte... on 22 Dec, 2016 01:27 AM
Yeah, sorry I was wrong, those variables will not work in this scenario. If you have working approach how to use git log for this, it would be great if you drop code sample here, which might help others.
Also another approach I proposed before (create projects on AppVeyor for every build, but those projects will do nothing, but start main build with API (example)) might work. You can pass those variables while main calling build with API from each of those project.
However this maybe too expensive to re-do your work if you almost done with current approach and have working solutions based on git logs...
Ilya.
10 Posted by Stijn Verstraet... on 22 Dec, 2016 01:37 PM
We also have a bunch of projects still in private repositories so the project-per-repo approach won’t do it and I think in the end the current method is somewhat less work when adding repositories in the future, we’ll see.
The git log approach is working though it took me a while to figure out how to clone as shallow as possible and still be able to get the last commit for any branch. But it’s not too bad now: only submodule shallow clone isn’t working, but still clone time is less than half in comparision with a full clone.
The process goes like this:
- get file which lists the repositories
- for each of those do git clone –depth=1 –no-single-branch
- use git log –n1 –all to get the newest commit (*)
- checkout latest commit
- build
(*) we’re a very small team and currently the CI is just for validation, releases are built locally on-demand almost daily. As such the stratgey for which branch to build from each repo is simply ‘use the newest commit’ which normally translates to master branch for most repos and feature braches for stuff still under review. It’s not hard to modify the checkout process to use specific branches, but ‘standard’ flow using pull-requests with multiple repos will be pretty hard: then you end up with matrix builds or whatever it’s called in TeamCity, and that is just messy to figure out what causes a build failure.
Full code, still work in progress, no guarantees etc; Appveyor settings source this file then calls the RunIt function:
# Convert string with spaces as seperator into an RSA private key file.
# The string is normally acquired as an Appveyor secure environment variable.
# See https://www.appveyor.com/docs/how-to/private-git-sub-modules/
function Write-PrivateRsaKeyFile() {
param(
[parameter(Mandatory)] [String] $spacedKey,
[parameter(Mandatory)] [String] $file
)
$fileContent = "-----BEGIN RSA PRIVATE KEY-----`n"
$fileContent += "$spacedKey".Replace(' ', "`n")
$fileContent += "`n-----END RSA PRIVATE KEY-----`n"
Set-Content $file $fileContent
}
# Write hosts section of an SSH config file.
# BitBucket doesn't allow the same SSH key for different users/groups so
# we define different host aliases with each their unique key file.
# When cloning the correct alias must be specified.
function Write-SshConfig() {
param(
[parameter(Mandatory)] [Object[]] $hosts,
[parameter(Mandatory)] [String] $file
)
$fileContent = ""
foreach($item in $hosts) {
$fileContent += "Host " + $item[0] + "`n"
$fileContent += " User git`n"
$fileContent += " Hostname " + $item[1] + "`n"
$fileContent += " PreferredAuthentications publickey`n"
$fileContent += " IdentityFile " + $item[2] + "`n"
}
Set-Content $file $fileContent
}
# Download and extract zip file
function Extract-FromWeb {
param (
[parameter(Mandatory)] [String] $address,
[parameter(Mandatory)] [String] $zipFile,
[parameter(Mandatory)] [String] $destDir
)
iwr $address -OutFile $zipFile
$shellApplication = new-object -com shell.application
$zipPackage = $shellApplication.NameSpace($zipFile)
mkdir $destDir
$destinationFolder = $shellApplication.NameSpace($destDir)
$destinationFolder.CopyHere($zipPackage.Items(), 20)
rm $zipFile
}
# Throw if last external command had an error
function Check-LastExitCode {
param (
[parameter()] [scriptblock] $CleanupScript=$null
)
if ($LastExitCode -eq 0) {
return
}
if ($CleanupScript) {
& $CleanupScript
}
$msg = @"
External command returned exit code $LastExitCode
Callstack:$($(Get-PSCallStack)[1] | Out-String)
"@
throw $msg
}
# Execute external command and throw if it had an error
function Exec {
param (
[parameter(Mandatory)] [scriptblock] $command
)
& $command
Check-LastExitCode
}
# Call git without having to cd to the directory.
function Invoke-Git {
param (
[parameter(Mandatory)] [String] $workingDir,
[parameter(Mandatory)] [String[]] $gitCommand
)
if($gitCommand.Length -gt 0) {
$subCommand = $gitCommand[0]
} else {
$subCommand = $False
}
if($subCommand -eq 'clone') {
Exec { git $gitCommand $workingDir }
} elseif($subCommand -eq 'submodule' -Or $subCommand -eq 'stash') {
$currentDir = $(Get-Location)
cd $workingDir
try {
Exec { git $gitCommand }
} finally {
cd $currentDir
}
} else {
Exec { git --git-dir=$workingDir\.git --work-tree=$workingDir $gitCommand }
}
}
# Read mr-style repository config file.
function Get-MrRepos {
param (
[ValidateScript({Test-Path $_})] [String] $mrconfig
)
$baseDir = Split-Path (Resolve-Path $mrconfig)
$getBranch = {
param($regexMatch)
$branch = ( $regexMatch.Context.PostContext | sls -pattern 'checkout\s+?(\S+)?' | %{ $regexMatch.Matches[ 0 ].Groups[ 1 ].Value } )
if( -not $branch ) { 'master' } else { $branch }
}
return sls $mrconfig -Pattern "^checkout\s+=\s+git\s+clone\s+'?(\S+)'?\s+'?(\S+)'?" -Context ( 0, 1 ) |
% { @{ 'remote' = $_.Matches[ 0 ].Groups[ 1 ].Value;
'directory' = ( Join-Path $baseDir $_.Matches[ 0 ].Groups[ 2 ].Value );
'branch' = ( & $getBranch $_ ) } }
}
$bitBucketNeuroAlias = 'bitbucketneuro'
$bitBucketIgAlias = 'bitbucketig'
# Configure ssh settings for cloning from Bitbucket and Github.
# Keys must be stored in $env:bbKeyNeuro and $env:bbKeyIg
function Install-SshConfig() {
param (
[parameter(Mandatory)] [String] $sshDir
)
$rsaKeyNeuro = (Join-Path $sshDir 'id_rsa_neuro')
$rsaKeyIg = (Join-Path $sshDir 'id_rsa_ig')
Write-PrivateRsaKeyFile "$env:bbKeyNeuro" $rsaKeyNeuro
Write-PrivateRsaKeyFile "$env:bbKeyIg" $rsaKeyIg
Write-SshConfig -hosts (($bitBucketNeuroAlias, 'bitbucket.org', $rsaKeyNeuro),
($bitBucketIgAlias, 'bitbucket.org', $rsaKeyIg),
('github.com', 'github.com', $rsaKeyNeuro)) -file (Join-Path $sshDir 'config')
}
# Modify host in address according to ssh config from Install-SshConfig.
# replace actual bitbucket hostnames with aliases pointing to specific ssh keys.
function Convert-CloneAddress() {
param (
[parameter(Mandatory)] [String] $cloneAddress
)
if(-not $skipSshConfig) {
$cloneAddress = $cloneAddress.Replace('bitbucket.org:neurofys', $bitBucketNeuroAlias + ':neurofys')
$cloneAddress = $cloneAddress.Replace('bitbucket.org:ignitron', $bitBucketIgAlias + ':ignitron')
}
$cloneAddress
}
# Download binary ffmpeg distribution and put it in the expected location
function Get-FFMpegBinaries() {
param (
[ValidateScript({Test-Path $_})] [String] $projectDir,
[String] $version = '3.2.2'
)
$ffmpegX86 = "ffmpeg-$version-win32-dev"
$ffmpegX64 = "ffmpeg-$version-win64-dev"
$zipFile = (Join-Path $projectDir 'ffmpeg.zip')
Extract-FromWeb "https://ffmpeg.zeranoe.com/builds/win32/dev/$ffmpegX86.zip" $zipFile (Join-Path $projectDir 'ffmpeg_x86')
Extract-FromWeb "https://ffmpeg.zeranoe.com/builds/win64/dev/$ffmpegX64.zip" $zipFile (Join-Path $projectDir 'ffmpeg_x64')
cp -Recurse (Join-Path $projectDir "ffmpeg_x86\$ffmpegX86\include") (Join-Path $projectDir '_include')
cp -Recurse (Join-Path $projectDir "ffmpeg_x86\$ffmpegX86\lib") (Join-Path $projectDir '_lib\ffmpeg_x86')
cp -Recurse (Join-Path $projectDir "ffmpeg_x64\$ffmpegX64\lib") (Join-Path $projectDir '_lib\ffmpeg_x64')
}
# Download the .mrconfig file
function Get-MrConfig() {
param (
[String] $mrConfigCloneDir,
[String] $mrConfigFile
)
Invoke-Git $mrConfigCloneDir -gitCommand clone, -q, --depth=1, (Convert-CloneAddress git@bitbucket.org:neurofys/mrconfig)
cp (Join-Path $mrConfigCloneDir '.mrconfig') $mrConfigFile
}
function RunIt() {
if(-not $skipSshConfig) {
Install-SshConfig $sshDir
}
if(-not (Test-Path $projectDir)) {
mkdir $projectDir
}
Get-MrConfig (Join-Path $projectDir 'mrconfig') $mrConfigFile
Get-MrRepos $mrConfigFile | % {
Write-Host '[mr] ' $_.directory
if( -not (Test-Path $_.directory) ) {
# Note appveyor's git doesn't yet have '--shallow-submodules'
Invoke-Git $_.directory -gitCommand clone, -q, --depth=1, --no-single-branch, (Convert-CloneAddress $_.remote)
} else {
Invoke-Git $_.directory -gitCommand pull, --rebase
}
$lastCommit = (Invoke-Git $_.directory 'log -n1 --all --format="%h %d"'.Split(' '))
Write-Host ('[{0}]' -f $lastCommit)
Invoke-Git $_.directory -gitCommand checkout, -qf, $lastCommit.Split(' ')[ 0 ]
if( Test-Path (Join-Path $_.directory '.gitmodules') ) {
# Still need to figure out how to shallow clone these
Invoke-Git $_.directory -gitCommand submodule, update, -q, --init, --recursive
}
}
Get-FFMpegBinaries $projectDir
Exec { msbuild (Join-Path $projectDir 'build/prepareworkspace.targets') '/p:NoFFMPEG=True' }
Exec { msbuild (Join-Path $projectDir 'build/createrelease.targets') "/t:CreateVariant" "/p:DistDir=$distDir\x86\;NoNTX=True;NoVISA=True;Configuration=Release;Platform=Win32" }
Exec { msbuild (Join-Path $projectDir 'build/createrelease.targets') "/t:CreateVariant" "/p:DistDir=$distDir\x64\;NoNTX=True;NoVISA=True;Configuration=Release;Platform=x64" }
}
$sshDir = 'c:\users\appveyor\.ssh'
$projectDir = 'c:\projects\tns'
$distDir = Join-Path $projectDir '_dist'
$mrConfigFile = Join-Path $projectDir '.mrconfig'
11 Posted by Ilya Finkelshte... on 24 Dec, 2016 01:41 AM
Wow, thanks a lot!
Ilya Finkelshteyn closed this discussion on 25 Aug, 2018 02:10 AM.