Home → Problems →

Accessing another repository during a build

• Edit

Dan Jones

18 Jul, 2014 03:05 PM

Hi,

We had a setup on team city which admittedly was a slight hack but worked quite well. I'd like to just open a discussion about how to do the same thing in AppVeyor or potentially come up with a better one.

We build our main repo and as a post build step we, based on the resulting built code, have some ps scripts which scaffold out some new files. We then want those files to be pushed to a separate private github repository which when pushed to builds a nuget package. The last step about the package is unimportant.

So I have tried all manner of things to get the after success build script to work. Here is the basic flow

Build project as normal doing a nuget restore
Scaffold out some new files
clone another private repo
Move the generated files into that working directory
git push up to the cloned repo

I have tried ssh and http cloning into a specific directory during the build but I always get what seems like a generic git error (git cloniung into xxx...RemoteException NativeCommandError)
And sometimes "xxx...is not a git Repository".

Quite sure it's security - with certain versions of the script it explicitly says this but have tried a host of things - user/password in the Url as variables, adding github repo ssh key on the fly (saw in another post), as I say also using https and ssh but can't get past this point.
I imagine there would be a much better way but I was porting the same script from team city and this is essentially the workflow we require.

Here is a cut down version of the build script

& git clone --depth=1 --branch=master git@github.com:adazzle/xxxxxxxx.git C:\projects\xxxxxxxx --fails here every time
& cd $env:APPVEYOR_BUILD_FOLDER
---some script that generates files - this always completes----
copy-item args
& cd $env:translations_dir
& git add *file spec*
& git commit -am "commit"
& git push origin master

Thanks

Dan

1. Support Staff 1 Posted by Feodor Fitsner on 18 Jul, 2014 03:28 PM

   

   Hi Dan,

   That "RemoteException" is a nasty PowerShell "feature" converting StdErr output to the exception. In this case git command writes progress into StdErr. To solve this in PS just get rid of StdErr output by appending 2> $null, for example:

   git clone ... 2> $null
   

   Regarding another repo authentication - take a look at personal access tokens:

   • https://help.github.com/articles/creating-an-access-token-for-comma...
   • https://help.github.com/articles/git-automation-with-oauth-tokens
   • Edit

2. 2 Posted by Dan Jones on 18 Jul, 2014 03:58 PM

   

   That's very helpful thanks Feodor,

   regarding the tokens - how would I use that in a build script avoiding the prompts? The links mention caching with credential helper which we do locally on our dev machines - but in this environment obviously isn't an option. So once I have an access token do I use that in place of the user name in the clone url?

   • Edit

3. Support Staff 3 Posted by Feodor Fitsner on 18 Jul, 2014 04:05 PM

   

   Exactly, the token works like a username + empty password. You can use it in the URL while cloning: https://github.com/blog/1270-easier-builds-and-deployments-using-gi...

   To avoid showing the token in the build log put it in environment variable.

   • Edit

4. 4 Posted by Dan Jones on 18 Jul, 2014 04:34 PM

   

   Hi Feodor,
   So I tried this:
   & git clone -n --depth=1 --branch=master https://$env:github_token:x-oauth-basic@github.com/adazzle/translations.git

   github_token being an environment variable and the output is attached.
   I took off the 2>$null as otherwise the build just sits there not erroring, but also not continuing.

   Thanks for your help so far.

   • Clone_Error.png 9.96 KB
   • Edit

5. Support Staff 5 Posted by Feodor Fitsner on 18 Jul, 2014 05:36 PM

   

   I know it looks ugly :), but append 2>&1 | % { $_.ToString() } to the git command.

   • Edit

6. 6 Posted by Dan Jones on 21 Jul, 2014 09:02 AM

   

   Hi Feodor,

   Thanks again for your help so far. I appended that to the git command and the error no longer appears. However it never get's past that point just reading
   "Cloning into directory..." Last build was cancelled after 64 hours! so I'm sure it wasn't doing anything. Should I be approaching this differently? Thanks again.
   Dan

   • Edit

7. Support Staff 7 Posted by Feodor Fitsner on 21 Jul, 2014 09:05 AM

   

   I'm sure the problem in git clone command asking to enter credentials. Awaiting input is the most common cause of hanging builds. If that repository is private how do you authenticate?

   • Edit

8. 8 Posted by Dan Jones on 21 Jul, 2014 10:18 AM

   

   Yeah ok that sounds very likely. So the git command is currently this:

   & git clone -n --depth=1 --branch=master https://$env:github_token:x-oauth-basic@github.com/adazzle/translations.git C:\projects\translations 2>&1 | % { $_.ToString() }

   $env:github_token is a variable setup using directions from a previous post and this link https://github.com/blog/1270-easier-builds-and-deployments-using-git-over-https-and-oauth

   I'll double check over the setup to make sure I haven't missed anything. Something isn't quite right with it though as it says in GitHub that the token has never been used

   • Edit

9. Support Staff 9 Posted by Feodor Fitsner on 21 Jul, 2014 10:30 AM

   

   Should be:

   & git clone -n --depth=1 --branch=master https://$($env:github_token):x-oauth-basic@github.com/adazzle/translations.git C:\projects\translations 2>&1 | % { $_.ToString() }
   

   Key change here is $($env:github_token)

   • Edit

10. 10 Posted by Dan Jones on 21 Jul, 2014 11:03 AM

    

    Fantastic - it is now cloning. Thank you very much for all your help!
    Dan

    • Edit

11. Ilya Finkelshteyn closed this discussion on 25 Aug, 2018 01:46 AM.