Vagrant not working inside AppVeyor

ukio's Avatar

ukio

20 Dec, 2014 11:05 PM

Hi everybody,

I'm trying to run some integration tests, which involves spinning up a VirtuaBox VM inside an AppVeyor Build Worker via Vagrant.

However, whenever it tries to bring up the VM (via `vagrant up`) it will never come up properly. It runs into a timeout while trying to connect to that VM. See here:
https://ci.appveyor.com/project/ukio/sample-toplevel-cookbook/build/34-master

I suspect this is because Vagrant wants to create an new network adapter which (at least on my Win7 laptop) causes the UAC prompt to come up. I'm afraid it stalls here becuase the prompt is never granted elevated permissions for creating the network adapters.

Any ideas how to confirm and / or fix this?

Thanks and kind regards,
Torben

  1. Support Staff 1 Posted by Feodor Fitsner on 20 Dec, 2014 11:15 PM

    Feodor Fitsner's Avatar

    Well, I don't think it's UAC - in fact, it's disabled on build workers.

    Try using "unstable" OS (image) - it's running build agent interactive - maybe this could help. You can select it on Environment tab of project settings.

  2. 2 Posted by ukio on 21 Dec, 2014 12:00 AM

    ukio's Avatar

    Hi Feodor, thanks for the quick response!

    Yes, I believe it's disabled. Also tried explicitly disabling UAC and / or the UAC prompt (from http://ss64.com/nt/syntax-uac.html), but it did not help though...

    Will try with the "unstable" OS (image) now...

  3. 3 Posted by ukio on 21 Dec, 2014 01:26 AM

    ukio's Avatar

    FYI - I could verify that it works in general (via minimalistic vagrant file):
    https://ci.appveyor.com/project/ukio/sample-toplevel-cookbook/build/46-appveyor

    So it must be something else. I'll start with increasing the timeout a bit...

  4. Support Staff 4 Posted by Feodor Fitsner on 21 Dec, 2014 01:30 AM

    Feodor Fitsner's Avatar

    That's great, keep us posted!

    -Feodor

  5. 5 Posted by ukio on 27 Dec, 2014 10:11 PM

    ukio's Avatar

    Experiences so far:

    • increased the timeout for bringing up the vm to 20 minutes
    • running builds on x64 Windows Server 2012 only
    • the time for bringing up the VM varies quite a lot. I got the vm up and running
      • on hyperv: somewhere between 6 and 16 minutes
      • on azure: somewhere between 16 and 26 minutes
    • so far I only got 32-bit VMs running. The 64-bit VMs always timed out.
    • still randomly running into timeouts while bringing up the VMs (mostly on hyperv, azure feels more stable)
    • tried to cache the VirtualBox installation, but that doesn't work out -> you need to run the installer to get the additional network adapters and probably lots of other things

    So, current status: still trying to get a series of green stable builds reproducably and reliably

  6. 6 Posted by ukio on 27 Dec, 2014 10:49 PM

    ukio's Avatar

    Results with the hyperv workers are getting better. Maybe it was a bad idea trying to cache the Vagrant installation as well, so I removed it from the cache, and promptly got a new build time record:

  7. 7 Posted by ukio on 27 Dec, 2014 11:16 PM

    ukio's Avatar

    ...and the next build on hyperv timed out again :-( https://ci.appveyor.com/project/ukio/vagrant-appveyor-testing/build...

    There was no code / appveyor.yml change involved. So what else could the differences be?

    Actually - the hyperv workers should be all exactly the same, shouldn't they?

  8. 8 Posted by ukio on 28 Dec, 2014 12:00 AM

    ukio's Avatar

    New record:

  9. Support Staff 9 Posted by Feodor Fitsner on 28 Dec, 2014 12:03 AM

    Feodor Fitsner's Avatar

    Not bad at all!

  10. 10 Posted by ukio on 28 Dec, 2014 12:46 AM

    ukio's Avatar

    This one looks weird. I reverted back to the trusty32 box to check if it were the culprit. Got a failed build after 22 minutes again, but this time with a different error message:
    https://ci.appveyor.com/project/ukio/vagrant-appveyor-testing/build...

    ...
        default: Warning: Connection timeout. Retrying... 
    153    default: Warning: Connection timeout. Retrying... 
    154    default: Warning: Connection timeout. Retrying... 
    155Vagrant can't use the requested machine because it is locked! This
    156means that another Vagrant process is currently reading or modifying
    157the machine. Please wait for that Vagrant process to end and try
    158again. Details about the machine are shown below:
    159 
    160Name: default
    161Provider: virtualbox 
    162Command exited with code 1
    

    The error indicates some kind of shared state, but it's also pretty exactly at the configured boot timeout after 20 minutes.

  11. 11 Posted by ukio on 29 Dec, 2014 11:41 PM

    ukio's Avatar

    Hi Feodor,

    I have no luck with the ubuntu 14.04 base boxes on hyper-v build workers. On the azure workers everything works well (it just takes a bit longer).

    Tried with the "ubuntu/trusty32" as well as the "chef/ubuntu-14.04-i386" base box. In both cases it randomly (looks like random at least) fails to connect via SSH within the given timeout of 15 minutes.

    Compare the latest build results from hyper-v (mostly instable):
    https://ci.appveyor.com/project/ukio/vagrant-appveyor-testing/history
    ...with the ones from azure (stable): https://ci.appveyor.com/project/tknerr/vagrant-appveyor-testing/his...

    What could be the differences between the azure and hyper-v workers? Anything ssh related maybe?

    I added some client-side ssh debugging if the vagrant up command fails. However it only reveals a ssh_exchange_identification: read: Connection reset by peer error, which I can't debug further client side.

    It would probably help to debug this via RDP. I am now starting the VMs in gui mode but no idea how to access the build worker via RDP if that is possible at all.

    Do you have any futher means on how to debug this?

    I don't have any meta information about the build worker IDs who are executing the builds -- maybe that would give another hint (e.g. reveal that it always fails / succeeds on a specific hyper-v worker).

    Other suspects:
    * firewall? * disabled the windows firewall but did not help * checked inside the ubuntu/trusty32 vm: ufw firewall is inactive and neither fail2ban nor denyhosts is installed, so that should not be the reason * reverse dns slowness? * unlikely since the appveyor build workers should use 8.8.8.8 which is pretty stable * however, UseDNS is not explictly disabled in /etc/ssh/sshd_config and the default is yes, so it might have to do with it

    sigh...

  12. 12 Posted by ukio on 30 Dec, 2014 12:54 AM

    ukio's Avatar

    @Feodor can you see this in the VirtualBox GUI? https://ask.openstack.org/en/question/28297/cloud-init-nonet-waitin...

    Not sure why it work then on the azure workers though...

  13. Support Staff 13 Posted by Feodor Fitsner on 30 Dec, 2014 02:27 AM

    Feodor Fitsner's Avatar

    You mean it might be DHCP problem? There is no DHCP on Hyper-V environment.

    -Feodor

  14. 14 Posted by ukio on 30 Dec, 2014 12:09 PM

    ukio's Avatar

    No, not exactly a DHCP problem, but something about the network adapters in ubuntu 14.04, or just a desperate guess because I see no other chance to debug this.

    So weird it works sometimes and sometimes not, but I can't see why.

    Is there any possibility to RDP into a build worker? Or at least getting screenshots of the VirtualBox GUI anyhow?

  15. Support Staff 15 Posted by Feodor Fitsner on 31 Dec, 2014 03:11 AM

    Feodor Fitsner's Avatar

    Theoretically, you can RDP into build worker instance.

    You should do three steps:
    1. Get know instance IP from IP-enabled NIC (say using PowerShell).
    2. Update appveyor account password, for example:

    $objUser = [ADSI]("WinNT://$($env:computername)/appveyor")
    $objUser.SetPassword('yourpassword')
    
    3. Allow RDP on firewall.

    Be aware your session will be limited with total time of build (40 minutes). Let me know how it goes.

  16. 16 Posted by Stefan Scherer on 25 Nov, 2015 09:49 PM

    Stefan Scherer's Avatar

    Great work to try Vagrant on AppVeyor.

    But is it also possible to run 64bit VM's in AppVeyor?
    AFAIK you would need VT-X enabled (nested hypervisor) in the AppVeyor worker instance. Can this be enabled in AppVeyor somewhere?

  17. 17 Posted by andrew.pashkin on 27 Feb, 2016 01:54 AM

    andrew.pashkin's Avatar

    Also stumbled upon this issue.

    Regarding RDB - there is an article in in the official docs about that:
    https://www.appveyor.com/docs/how-to/rdp-to-build-worker

  18. 18 Posted by andrew.pashkin on 27 Feb, 2016 11:05 AM

    andrew.pashkin's Avatar

    I was able to resolve my problem, here is the successful build:
    https://ci.appveyor.com/project/AndrewPashkin/vagrant-appveyor-test...

    In short, I've added such configuration to my Vagrantfile:

    config.vm.provider "virtualbox" do |vb|
        vb.customize ["modifyvm", :id, "--nictype1", "Am79C973"]
    end
    
    How did I came to the solution:
    1. First of all, I used this method, to access a build workers via RDP:
      https://www.appveyor.com/docs/how-to/rdp-to-build-worker
    2. The other curical point was in enabling GUI for a VM:

      config.vm.provider "virtualbox" do |vb|
          vb.gui = true
      end
      
    3. Finally, I was able to see, what happens inside a VM and spot an error
      (see attached screenshot). Also, I saw this comment earlier.

  19. 19 Posted by Stefan Scherer on 27 Feb, 2016 11:28 AM

    Stefan Scherer's Avatar

    Hi Andrew,

    This is awesome! Thanks for sharing nictype1=Am79C973
    I'll try this.

    Stefan

  20. 20 Posted by andrew.pashkin on 27 Feb, 2016 12:33 PM

    andrew.pashkin's Avatar

    It is also impossible to launch 64-bit boxes. See attached image with the error.

  21. Ilya Finkelshteyn closed this discussion on 25 Aug, 2018 02:04 AM.

Comments are currently closed for this discussion. You can start a new one.

Keyboard shortcuts

Generic

? Show this help
ESC Blurs the current field

Comment Form

r Focus the comment reply box
^ + ↩ Submit the comment

You can use Command ⌘ instead of Control ^ on Mac

Recent Discussions

05 Dec, 2019 09:12 PM
04 Dec, 2019 07:46 PM
04 Dec, 2019 12:01 AM
04 Dec, 2019 12:00 AM
03 Dec, 2019 11:00 PM