Home → Problems →

git clone hanging with multiple jobs

• Edit

jezz.santos

28 Dec, 2015 08:15 PM

(Only started seeing this problem when we went to multiple parallel jobs.)

I have 16 parallel jobs running on a single build, and quite randomly, one or more of the jobs will hang after the following line in the console:

git clone -q --branch=master git@github.com:mindkin/roamm.git C:\projects\<projecuslug>
(see attachment)

One suggestion is to use the 'shallow clone' feature. We did try that but the build failed because we are issuing subsequent commands to update git submodules.
We are pulling a private github repo with private submodules, and already use the documented workaround (http://www.appveyor.com/docs/how-to/private-git-sub-modules) to do that.
So 'shallow clone' will not work for us in this case.

Can you suggest any other options?
Perhaps create a script to do the shallow copy, and then shallow copy submodules into child folders?
What do you suggest?

• Capture.JPG 30.4 KB

1. 1 Posted by jezz.santos on 28 Dec, 2015 08:19 PM

   

   This is the script we are running after the: git clone

   (when it works, of course!)

   • Capture.JPG 48 KB
   • Edit

2. Support Staff 2 Posted by Feodor Fitsner on 28 Dec, 2015 08:36 PM

   

   I see you've enabled RDP. Did you have a chance to investigate what's going on the server when it hangs?

   -Feodor

   • Edit

3. 3 Posted by jezz.santos on 28 Dec, 2015 08:53 PM

   

   No, I haven't actually looked on the VM, when this problem happens.
   (we set up the RDP for the other problem we were having)

   Not sure what I would look for except a git process perhaps running?

   What would you look for?

   • Edit

4. Support Staff 4 Posted by Feodor Fitsner on 28 Dec, 2015 10:33 PM

   

   Well, git process is definitely hanging there. I would check how much of the repo was actually cloned: not started at all, half, entire repo.

   The problem with hanging git is interesting as I'm not sure whether it's client-side problem (Git) or server-side (GitHub). It doesn't look like connectivity issue though as I probably server couldn't hold a connection for so long.

   One of the way to "fix" that could be re-try. We could have an environment variable specifying the timeout for "git clone" process. If it wasn't able to complete within that timeout we kill the process, remove clone folder and try again. Also, we can add another variable enabling verbose output for "git clone" command. Sounds like a plan?

   • Edit

5. 5 Posted by jezz.santos on 28 Dec, 2015 10:35 PM

   

   Sounds like a great plan. Especially having a retry?

   What do you need me to do?

   • Edit

6. Support Staff 6 Posted by Feodor Fitsner on 28 Dec, 2015 10:37 PM

   

   There is nothing to do from your side for now. I'm going to implement that and deploy update to build workers shortly (day-two).

   • Edit

7. 7 Posted by jezz.santos on 28 Dec, 2015 10:39 PM

   

   awesome, let me know.

   BTW: Feel free to kick our build to test anytime.
   https://ci.appveyor.com/project/mindkin/roamm-c36ta

   • Edit

8. 8 Posted by jezz.santos on 28 Dec, 2015 11:38 PM

   

   BTW: was testing out whether using clone_depth would help.
   Turns out that it can still hang (see attachment), so retry looks like the only way

   • Capture.JPG 32.8 KB
   • Edit

9. Support Staff 9 Posted by Feodor Fitsner on 28 Dec, 2015 11:46 PM

   

   I'm wondering it cloning by HTTPS (not SSH) could do a better job. Are you referring sub-modules via their SSH URLs?

   • Edit

10. 10 Posted by jezz.santos on 28 Dec, 2015 11:53 PM

    

    Not sure how to tell?

    the submodule reference is like this: Subproject commit e9dffff71ebe68b2ab1167433a2e476bd22121e1

    (Remember, they are all private repos, the main one and the submodules)

    • Edit

11. Support Staff 11 Posted by Feodor Fitsner on 28 Dec, 2015 11:55 PM

    

    You can check in hidden .gitmodules file.

    • Edit

12. 12 Posted by jezz.santos on 28 Dec, 2015 11:59 PM

    

    Ah yeah, thats right: here it is:

    [submodule "lib/crib"]
        path = lib/crib
        url = git@github.com:mindkin/crib.git
    

    so yes, SSH

    • Edit

13. 13 Posted by jezz.santos on 29 Dec, 2015 01:27 AM

    

    I don't think I can do submodules that are private with HTTP access?
    That's why we had to go to SSH in the first place.

    I'd love to get off SSH, it is a PITA.

    • Edit

14. Support Staff 14 Posted by Feodor Fitsner on 29 Dec, 2015 03:30 AM

    

    Before going rude "retry" way I'd like to try HTTPS first. I feel that HTTPS might be the right way to do. It's possible to fetch modules via HTTPS too - I'll tell you how. I'm going to implement "force_github_https" flag that will make worker to clone private repo via HTTPS. Let you know when it's online.

    Btw, is your GitHub user authorized in AppVeyor has at least read access to submodule repositories?

    • Edit

15. 15 Posted by jezz.santos on 29 Dec, 2015 04:13 AM

    

    yes, github user has read access to all private repos.

    • Edit

16. 16 Posted by jezz.santos on 29 Dec, 2015 04:19 AM

    

    I can't remember the details now, but I understood that the reason we have moved to SSH is because github does not allow the same access key to be used on more than one repo.
    You probably know that.

    • Edit

17. Support Staff 17 Posted by Feodor Fitsner on 30 Dec, 2015 04:59 AM

    

    Hi Jezz,

    An update that allows cloning private GitHub repos via HTTPS has been deployed.
    It can be enabled on General tab of project settings or in appveyor.yml:

    force_https_clone: true
    

    Now how to fetch submodules via https. The idea is to change their repo URLs before doing update. This can be done by adding the following into build:

    git submodule init
    git config submodule.lib/crib.url https://github.com/mindkin/crib.git
    git config submodule.another-module.url ...
    ...
    git submodule update
    

    Let me know how that worked.

    • Edit

18. 18 Posted by jezz.santos on 30 Dec, 2015 09:33 AM

    

    Wow, how do you do it man?

    OK, not confident I am going to do this right, and add the right bits, and know what to remove.
    I have attached my current YML file.

    My repos and submodules are structured in a hierarchy like this:

    -> Roamm (https://github.com/mindkin/roamm) -> Crib (https://github.com/mindkin/crib) -> Gene (https://github.com/mindkin/gene)

    Would you be able to instruct me, or edit the attached file to be how you intent this to work?

    • appveyor.yml 10.4 KB
    • Edit

19. Support Staff 19 Posted by Feodor Fitsner on 30 Dec, 2015 05:32 PM

    

    You should replace inistall section with something like:

    install:
    - git submodule init
    # here we should check .git/config file for registered modules and their paths
    # and update their URLs to HTTPS ones
    - git config submodule.<submodule-path>.url <submodule-repo-https-url>
    - ...
    - git submodule update
    

    The private RSA key is not required anymore as modules will be fetched via HTTPS and Basic authentication.

    • Edit

20. 20 Posted by jezz.santos on 30 Dec, 2015 08:17 PM

    

    I have replaced the install section with this:
    

    - git submodule init
    - git config submodule.lib/crib.url https://github.com/mindkin/crib.git
    - git config submodule.lib/crib/lib.gene.url https://github.com/mindkin/gene.git
    - git submodule update --depth=5 --recursive
    

    however, now it is not recursively cloning the Gene submodule.
    Crib is a submodule of Roamm
    Gene is a submodule of the Crib.
    • Edit

21. Support Staff 21 Posted by Feodor Fitsner on 31 Dec, 2015 03:54 AM

    

    Well, I don't have much experience with git submodules. Maybe you can fetch 1st level, update URLs then fetch second level, etc. Also, I think it's possible to update URLs right away in .gitmodules if you can figure out regex.

    • Edit

22. 22 Posted by jezz.santos on 01 Jan, 2016 02:08 AM

    

    Not sorry Feodor,

    I have now taken the actions to get rid of submodules and move to nuget packages, so this issue is kind of redundant now.
    I appreciate the extra work you did on getting submodules by HTTPS. I wonder how you did that, and I hope your hard work helps others too.

    I now have a new challenge, after moving to NuGet, which is how to create a single Azure Cloud Services from multiple services defined in multiple repos in multiple nugets. A problem that I won't bother you with at this stage.

    • Edit

23. jezz.santos closed this discussion on 01 Jan, 2016 02:08 AM.