git clone hanging with multiple jobs

jezz.santos's Avatar

jezz.santos

28 Dec, 2015 08:15 PM

(Only started seeing this problem when we went to multiple parallel jobs.)

I have 16 parallel jobs running on a single build, and quite randomly, one or more of the jobs will hang after the following line in the console:

git clone -q --branch=master git@github.com:mindkin/roamm.git C:\projects\<projecuslug>
(see attachment)

One suggestion is to use the 'shallow clone' feature. We did try that but the build failed because we are issuing subsequent commands to update git submodules.
We are pulling a private github repo with private submodules, and already use the documented workaround (http://www.appveyor.com/docs/how-to/private-git-sub-modules) to do that.
So 'shallow clone' will not work for us in this case.

Can you suggest any other options?
Perhaps create a script to do the shallow copy, and then shallow copy submodules into child folders?
What do you suggest?

  1. 1 Posted by jezz.santos on 28 Dec, 2015 08:19 PM

    jezz.santos's Avatar

    This is the script we are running after the: git clone

    (when it works, of course!)

  2. Support Staff 2 Posted by Feodor Fitsner on 28 Dec, 2015 08:36 PM

    Feodor Fitsner's Avatar

    I see you've enabled RDP. Did you have a chance to investigate what's going on the server when it hangs?

    -Feodor

  3. 3 Posted by jezz.santos on 28 Dec, 2015 08:53 PM

    jezz.santos's Avatar

    No, I haven't actually looked on the VM, when this problem happens.
    (we set up the RDP for the other problem we were having)

    Not sure what I would look for except a git process perhaps running?

    What would you look for?

  4. Support Staff 4 Posted by Feodor Fitsner on 28 Dec, 2015 10:33 PM

    Feodor Fitsner's Avatar

    Well, git process is definitely hanging there. I would check how much of the repo was actually cloned: not started at all, half, entire repo.

    The problem with hanging git is interesting as I'm not sure whether it's client-side problem (Git) or server-side (GitHub). It doesn't look like connectivity issue though as I probably server couldn't hold a connection for so long.

    One of the way to "fix" that could be re-try. We could have an environment variable specifying the timeout for "git clone" process. If it wasn't able to complete within that timeout we kill the process, remove clone folder and try again. Also, we can add another variable enabling verbose output for "git clone" command. Sounds like a plan?

  5. 5 Posted by jezz.santos on 28 Dec, 2015 10:35 PM

    jezz.santos's Avatar

    Sounds like a great plan. Especially having a retry?

    What do you need me to do?

  6. Support Staff 6 Posted by Feodor Fitsner on 28 Dec, 2015 10:37 PM

    Feodor Fitsner's Avatar

    There is nothing to do from your side for now. I'm going to implement that and deploy update to build workers shortly (day-two).

  7. 7 Posted by jezz.santos on 28 Dec, 2015 10:39 PM

    jezz.santos's Avatar

    awesome, let me know.

    BTW: Feel free to kick our build to test anytime.
    https://ci.appveyor.com/project/mindkin/roamm-c36ta

  8. 8 Posted by jezz.santos on 28 Dec, 2015 11:38 PM

    jezz.santos's Avatar

    BTW: was testing out whether using clone_depth would help.
    Turns out that it can still hang (see attachment), so retry looks like the only way

  9. Support Staff 9 Posted by Feodor Fitsner on 28 Dec, 2015 11:46 PM

    Feodor Fitsner's Avatar

    I'm wondering it cloning by HTTPS (not SSH) could do a better job. Are you referring sub-modules via their SSH URLs?

  10. 10 Posted by jezz.santos on 28 Dec, 2015 11:53 PM

    jezz.santos's Avatar

    Not sure how to tell?

    the submodule reference is like this: Subproject commit e9dffff71ebe68b2ab1167433a2e476bd22121e1

    (Remember, they are all private repos, the main one and the submodules)

  11. Support Staff 11 Posted by Feodor Fitsner on 28 Dec, 2015 11:55 PM

    Feodor Fitsner's Avatar

    You can check in hidden .gitmodules file.

  12. 12 Posted by jezz.santos on 28 Dec, 2015 11:59 PM

    jezz.santos's Avatar

    Ah yeah, thats right: here it is:

    [submodule "lib/crib"]
        path = lib/crib
        url = git@github.com:mindkin/crib.git
    

    so yes, SSH

  13. 13 Posted by jezz.santos on 29 Dec, 2015 01:27 AM

    jezz.santos's Avatar

    I don't think I can do submodules that are private with HTTP access?
    That's why we had to go to SSH in the first place.

    I'd love to get off SSH, it is a PITA.

  14. Support Staff 14 Posted by Feodor Fitsner on 29 Dec, 2015 03:30 AM

    Feodor Fitsner's Avatar

    Before going rude "retry" way I'd like to try HTTPS first. I feel that HTTPS might be the right way to do. It's possible to fetch modules via HTTPS too - I'll tell you how. I'm going to implement "force_github_https" flag that will make worker to clone private repo via HTTPS. Let you know when it's online.

    Btw, is your GitHub user authorized in AppVeyor has at least read access to submodule repositories?

  15. 15 Posted by jezz.santos on 29 Dec, 2015 04:13 AM

    jezz.santos's Avatar

    yes, github user has read access to all private repos.

  16. 16 Posted by jezz.santos on 29 Dec, 2015 04:19 AM

    jezz.santos's Avatar

    I can't remember the details now, but I understood that the reason we have moved to SSH is because github does not allow the same access key to be used on more than one repo.
    You probably know that.

  17. Support Staff 17 Posted by Feodor Fitsner on 30 Dec, 2015 04:59 AM

    Feodor Fitsner's Avatar

    Hi Jezz,

    An update that allows cloning private GitHub repos via HTTPS has been deployed.
    It can be enabled on General tab of project settings or in appveyor.yml:

    force_https_clone: true
    

    Now how to fetch submodules via https. The idea is to change their repo URLs before doing update. This can be done by adding the following into build:

    git submodule init
    git config submodule.lib/crib.url https://github.com/mindkin/crib.git
    git config submodule.another-module.url ...
    ...
    git submodule update
    

    Let me know how that worked.

  18. 18 Posted by jezz.santos on 30 Dec, 2015 09:33 AM

    jezz.santos's Avatar

    Wow, how do you do it man?

    OK, not confident I am going to do this right, and add the right bits, and know what to remove.
    I have attached my current YML file.

    My repos and submodules are structured in a hierarchy like this:

    -> Roamm (https://github.com/mindkin/roamm) -> Crib (https://github.com/mindkin/crib) -> Gene (https://github.com/mindkin/gene)

    Would you be able to instruct me, or edit the attached file to be how you intent this to work?

  19. Support Staff 19 Posted by Feodor Fitsner on 30 Dec, 2015 05:32 PM

    Feodor Fitsner's Avatar

    You should replace inistall section with something like:

    install:
    - git submodule init
    # here we should check .git/config file for registered modules and their paths
    # and update their URLs to HTTPS ones
    - git config submodule.<submodule-path>.url <submodule-repo-https-url>
    - ...
    - git submodule update
    

    The private RSA key is not required anymore as modules will be fetched via HTTPS and Basic authentication.

  20. 20 Posted by jezz.santos on 30 Dec, 2015 08:17 PM

    jezz.santos's Avatar

    I have replaced the install section with this:

    - git submodule init
    - git config submodule.lib/crib.url https://github.com/mindkin/crib.git
    - git config submodule.lib/crib/lib.gene.url https://github.com/mindkin/gene.git
    - git submodule update --depth=5 --recursive
    
    however, now it is not recursively cloning the Gene submodule.
    Crib is a submodule of Roamm
    Gene is a submodule of the Crib.
  21. Support Staff 21 Posted by Feodor Fitsner on 31 Dec, 2015 03:54 AM

    Feodor Fitsner's Avatar

    Well, I don't have much experience with git submodules. Maybe you can fetch 1st level, update URLs then fetch second level, etc. Also, I think it's possible to update URLs right away in .gitmodules if you can figure out regex.

  22. 22 Posted by jezz.santos on 01 Jan, 2016 02:08 AM

    jezz.santos's Avatar

    Not sorry Feodor,

    I have now taken the actions to get rid of submodules and move to nuget packages, so this issue is kind of redundant now.
    I appreciate the extra work you did on getting submodules by HTTPS. I wonder how you did that, and I hope your hard work helps others too.

    I now have a new challenge, after moving to NuGet, which is how to create a single Azure Cloud Services from multiple services defined in multiple repos in multiple nugets. A problem that I won't bother you with at this stage.

  23. jezz.santos closed this discussion on 01 Jan, 2016 02:08 AM.

Comments are currently closed for this discussion. You can start a new one.

Keyboard shortcuts

Generic

? Show this help
ESC Blurs the current field

Comment Form

r Focus the comment reply box
^ + ↩ Submit the comment

You can use Command ⌘ instead of Control ^ on Mac