tag:help.appveyor.com,2012-11-13:/discussions/questions/33953-docker-linux-containers-in-windows-docker-workersAppVeyor: Discussion 2019-07-02T21:01:03Ztag:help.appveyor.com,2012-11-13:Comment/468922742019-01-30T14:17:49Z2019-01-30T14:17:50ZDocker: Linux containers in Windows docker workers<div><p>Our produce is a <em>WIndows</em> product that uses <em>Linux</em> containers. We'd love to test with Appveyor.</p>
<p>I see in <a href="https://help.appveyor.com/discussions/questions/2652-windows-server-2016#comment_42637798">https://help.appveyor.com/discussions/questions/2652-windows-server...</a> that linux containers were going to be offered, but I still don't see documentation on this. There are also references at <a href="https://help.appveyor.com/discussions/questions/8194-linux-docker-containers">https://help.appveyor.com/discussions/questions/8194-linux-docker-c...</a></p>
<p>Is this something Appveyor is going to be able to do? The last time I tried it was not possible, only WIndows containers were usable on Windows workers.</p></div>randytag:help.appveyor.com,2012-11-13:Comment/468922742019-01-30T20:06:09Z2019-01-30T20:06:09ZDocker: Linux containers in Windows docker workers<div><p>We actually provide the solution during last year, please check <a href="https://ci.appveyor.com/project/appveyor-tests/docker-ce/builds/21784007">this</a> self-descriptive build. It requires more expensive build VMs (check "Quad" VMs on the billing page).But right now we are forced to re-design it because of many issues with have with Docker for Windows and nested virtualization. We plan to provide an alternative based on Windows Server 2019 and Docker EE. If you are interested in trying this new approach, let us know, we will include you in pilot, which should start in coming weeks.</p></div>Ilya Finkelshteyntag:help.appveyor.com,2012-11-13:Comment/468922742019-01-30T20:31:04Z2019-01-30T20:31:04ZDocker: Linux containers in Windows docker workers<div><p>I thank you very much for the answer but am a little confused. You say that you already provide the solution, but also that it's not yet available?</p>
<p>Please include us in the pilot!<br>
We'd be happy to pay a reasonable monthly for this, it would be great. However, I don't see anything about Quad VM on the pricing page, <a href="https://www.appveyor.com/pricing/">https://www.appveyor.com/pricing/</a>, can you be more specific?</p></div>randytag:help.appveyor.com,2012-11-13:Comment/468922742019-01-30T20:51:36Z2019-01-30T20:51:36ZDocker: Linux containers in Windows docker workers<div><p>It is available but we plan to discontinue it in favor of Windows Server 2019 based solution. Pricing page contains only very basic options, check account settings > billing in AppVeyor portal. E.g. <code>https://ci.appveyor.com/account/<your_account>/billing</code>. It has much more options.</p></div>Ilya Finkelshteyntag:help.appveyor.com,2012-11-13:Comment/468922742019-01-30T21:23:08Z2019-01-30T21:23:08ZDocker: Linux containers in Windows docker workers<div><p>And thank you again for the quick response.</p>
<p>Unfortunately, I seem to have opened this through github (it's open source, again we're happy to pay). So since I opened it through github there's no info on the billing page, see attached screenshot.</p>
<p>Feel free to change the account, or tell me what to do, thanks.</p></div>randytag:help.appveyor.com,2012-11-13:Comment/468922742019-01-30T22:45:34Z2019-01-30T22:45:34ZDocker: Linux containers in Windows docker workers<div><p>I see, honestly we hesitate to provide this solution at the moment. We plan to move all Linux Container customers to new Windows Server 2019 VMs asap. We are right now preparing this. Is it OK for you to wait for couple of weeks?</p></div>Ilya Finkelshteyntag:help.appveyor.com,2012-11-13:Comment/468922742019-01-30T22:58:45Z2019-01-30T22:58:45ZDocker: Linux containers in Windows docker workers<div><p>Yes, it's OK. We would dearly love to move our testing over to appveyor instead of managing machines on buildkite. You have me on the list for notification? Thanks!</p></div>randytag:help.appveyor.com,2012-11-13:Comment/468922742019-01-31T19:43:35Z2019-01-31T19:43:35ZDocker: Linux containers in Windows docker workers<div><p>Sure.</p>
<p>Please subscribe to technical updates at <a href="https://ci.appveyor.com/notifications">https://ci.appveyor.com/notifications</a> and follow us on Twitter.</p>
<p>If we decide to not advertise trial of those VMs with those channels, we will send you information separately. But subscribing to technical updates is useful anyway.</p>
<p>Ilya.</p></div>Ilya Finkelshteyntag:help.appveyor.com,2012-11-13:Comment/468922742019-02-15T01:22:00Z2019-02-15T01:22:00ZDocker: Linux containers in Windows docker workers<div><p>New solution is ready for testing and your enabled for your account.</p>
<p>Please see sample build: <a href="https://ci.appveyor.com/project/appveyor-tests/win2k19-azure/builds/22386598">https://ci.appveyor.com/project/appveyor-tests/win2k19-azure/builds...</a><br>
and respective YAML config: <a href="https://github.com/appveyor-tests/win2k19-azure/blob/master/appveyor.yml">https://github.com/appveyor-tests/win2k19-azure/blob/master/appveyo...</a></p>
<p>Note that build start is typically about 3 minutes (time to provision VM on Azure).</p>
<p>Please let us know how it works for you.</p></div>Ilya Finkelshteyntag:help.appveyor.com,2012-11-13:Comment/468922742019-02-15T01:26:43Z2019-02-15T01:26:43ZDocker: Linux containers in Windows docker workers<div><p>Note that <code>image: Windows Server 2019</code> is essential to run build on those VMs.</p></div>Ilya Finkelshteyntag:help.appveyor.com,2012-11-13:Comment/468922742019-02-15T16:21:31Z2019-02-15T16:21:31ZDocker: Linux containers in Windows docker workers<div><p>Thanks so much! We're looking forward to trying it out.</p></div>randytag:help.appveyor.com,2012-11-13:Comment/468922742019-02-18T14:52:08Z2019-02-18T14:52:08ZDocker: Linux containers in Windows docker workers<div><p>Could I get it added to our account as well?</p></div>mikejolleytag:help.appveyor.com,2012-11-13:Comment/468922742019-02-18T19:55:21Z2019-02-18T19:55:21ZDocker: Linux containers in Windows docker workers<div><p><strong>@mikejolley</strong>: please provide your account name (or simple link to the build) -- I cannot figure it out using email you use in the forum.</p></div>Ilya Finkelshteyntag:help.appveyor.com,2012-11-13:Comment/468922742019-02-18T21:14:16Z2019-02-18T21:14:16ZDocker: Linux containers in Windows docker workers<div><p><a href="https://ci.appveyor.com/project/AdvancedSystemsUnlimited/kykenkee/builds/22457239">https://ci.appveyor.com/project/AdvancedSystemsUnlimited/kykenkee/b...</a></p></div>mikejolleytag:help.appveyor.com,2012-11-13:Comment/468922742019-02-18T21:30:18Z2019-02-18T21:30:18ZDocker: Linux containers in Windows docker workers<div><p><code>Windows Server 2019</code> image is enabled for <code>AdvancedSystemsUnlimited</code> account. Couple of things to note:</p>
<ul>
<li>
<p>This image runs on Azure VMs, so build start time is 2-6 minutes (overhead of creating and starting VM on Azure, though we are working to make it closer to 2)</p>
</li>
<li>
<p>For your plan it will be <code>Standard_D2s_v3</code> VM (details <a href="https://docs.microsoft.com/en-us/azure/virtual-machines/windows/sizes-general#dsv3-series-sup1sup">here</a>) with 2 logical CPUs. 4-CPUs <code>Standard_D4s_v3</code> goes with "Quad VM" plan (available on the billing page).</p>
</li>
</ul>
<p>It is right now in preview, your feedback is greatly appreciated.</p></div>Ilya Finkelshteyntag:help.appveyor.com,2012-11-13:Comment/468922742019-02-20T00:17:02Z2019-02-20T00:17:54ZDocker: Linux containers in Windows docker workers<div><p>Thanks so much for your work on this @ilya - Earlier today I was in fact able to run a simple docker command using your example (<a href="https://github.com/appveyor-tests/win2k19-azure/blob/master/appveyor.yml">https://github.com/appveyor-tests/win2k19-azure/blob/master/appveyo...</a>) but now I am no longer able to; it seems that the docker command is not found in the PATH. I don't think this is due to any changes I've made but maybe I got lost somewhere. Build link: <a href="https://ci.appveyor.com/project/rfay/build-tools/builds/22500893">https://ci.appveyor.com/project/rfay/build-tools/builds/22500893</a></p>
<p>(If you'd rather have feedback somewhere else or an issue or whatever, just say the word. Thanks for being so responsive here.)</p></div>randytag:help.appveyor.com,2012-11-13:Comment/468922742019-02-20T00:33:33Z2019-02-20T00:33:33ZDocker: Linux containers in Windows docker workers<div><p>Thank you for pointing this out. Can you please try now?</p></div>Ilya Finkelshteyntag:help.appveyor.com,2012-11-13:Comment/468922742019-02-20T00:52:29Z2019-02-20T00:52:29ZDocker: Linux containers in Windows docker workers<div><p>Thanks, it seems to be behaving again: <a href="https://ci.appveyor.com/project/rfay/build-tools/builds/22501376">https://ci.appveyor.com/project/rfay/build-tools/builds/22501376</a></p></div>randytag:help.appveyor.com,2012-11-13:Comment/468922742019-03-05T15:17:28Z2019-03-05T15:17:52ZDocker: Linux containers in Windows docker workers<div><p>Thanks for the awesome help, I've mostly got things working.</p>
<p>One of our test scenarios uses NFS and has to create a volume with options, but I see:</p>
<blockquote>
<p>docker volume create --driver local --opt type=nfs junker99</p>
<p>Error response from daemon: create junker99: options are not supported on this platform</p>
</blockquote>
<p>I'm always a little confused about what docker behavior I'm seeing.<br>
Is it docker-ee?<br>
How does one find out about behaviors like this?<br>
Is there any workaround for this problem with volumes and options? That seems like a strange restriction, and so standard in the docker docs.</p></div>randytag:help.appveyor.com,2012-11-13:Comment/468922742019-03-05T16:25:31Z2019-03-05T16:25:31ZDocker: Linux containers in Windows docker workers<div><p>Network creation also seems not to work?</p>
<p>Tests get this trying to create a normal docker-compose network:</p>
<pre>
<code> error (404): could not find plugin bridge in v1 plugin registry: plugin not found</code>
</pre></div>randytag:help.appveyor.com,2012-11-13:Comment/468922742019-03-06T22:30:18Z2019-03-06T22:30:18ZDocker: Linux containers in Windows docker workers<div><p>Docker is installed exactly as described <a href="https://docs.microsoft.com/en-us/virtualization/windowscontainers/quick-start/quick-start-windows-server#install-docker">this MS doc</a>. As stated <a href="https://docs.docker.com/engine/reference/commandline/volume_create/">in Docker docs</a> in <strong>Driver-specific options</strong>: "The built-in <code>local</code> driver on Windows does not support any options".<br>
Can you please describe the environment where it worked for you?</p>
<p>Regarding network creation issue, can you please provide more detailed repro steps?</p></div>Ilya Finkelshteyntag:help.appveyor.com,2012-11-13:Comment/468922742019-03-07T03:04:13Z2019-03-07T03:04:13ZDocker: Linux containers in Windows docker workers<div><p>Thanks so much for your support on this.</p>
<p>This absolutely works on docker-ce (Docker for Windows):</p>
<pre>
<code>λ docker volume create --driver local --opt type=nfs junker99
junker99</code>
</pre>
<p>An example of docker network creation is just <code>docker network create junk</code></p>
<p>Here's what happens when you do that here: <a href="https://ci.appveyor.com/project/rfay/ddev-rgym1">https://ci.appveyor.com/project/rfay/ddev-rgym1</a></p>
<p><code>Error response from daemon: could not find plugin bridge in v1 plugin registry: plugin not found</code></p></div>randytag:help.appveyor.com,2012-11-13:Comment/468922742019-03-19T14:14:19Z2019-03-19T14:14:19ZDocker: Linux containers in Windows docker workers<div><p>I guess it may not be possible to do what needs to be done with Windows Server 2019. I'll give one more attempt by configuring a Windows Server 2019 with docker-ee myself to see if I can figure out what's going on here.</p>
<p>Could you please let me know which edition of Windows Server 2019 you're using, and any particular characteristics that I'd need to duplicate? Versions are at <a href="https://www.microsoft.com/en-us/evalcenter/evaluate-windows-server-2019">https://www.microsoft.com/en-us/evalcenter/evaluate-windows-server-...</a></p>
<p>Thanks!</p></div>randytag:help.appveyor.com,2012-11-13:Comment/468922742019-03-19T16:56:28Z2019-03-19T16:56:28ZDocker: Linux containers in Windows docker workers<div><p>We are working on the replacing it with Docker CE (backed by MobyLinuxVM) actually, which apparently have better functional for many scenarios, including your. Hope to release updated image this week.</p>
<p>We use Windows Server 2019 Datacenter provided by Azure, if you are still interested.</p></div>Ilya Finkelshteyntag:help.appveyor.com,2012-11-13:Comment/468922742019-03-19T17:00:36Z2019-03-19T17:00:36ZDocker: Linux containers in Windows docker workers<div><p>docker-ce would make me very happy. I'll look forward to the update.</p></div>randytag:help.appveyor.com,2012-11-13:Comment/468922742019-04-22T21:27:04Z2019-04-22T21:27:04ZDocker: Linux containers in Windows docker workers<div><p>It was not easy, but finally <code>Windows Server 2019</code> image runs Docker CE. Please see usage sample <a href="https://ci.appveyor.com/project/appveyor-tests/win2k19-azure/builds/24024378">here</a>. Let us know how it goes.</p></div>Ilya Finkelshteyntag:help.appveyor.com,2012-11-13:Comment/468922742019-04-23T01:01:48Z2019-04-23T01:01:48ZDocker: Linux containers in Windows docker workers<div><p>Yay, thanks for the hard work, look forward to trying it.</p></div>randytag:help.appveyor.com,2012-11-13:Comment/468922742019-04-23T20:05:29Z2019-04-23T20:05:29ZDocker: Linux containers in Windows docker workers<div><p>I'm not having any more luck, the behavior is the same. <code>docker network create</code> fails, <code>docker volume</code> with drive fails.</p>
<ul>
<li><a href="https://ci.appveyor.com/project/rfay/build-tools-w323a/builds/24052278">https://ci.appveyor.com/project/rfay/build-tools-w323a/builds/24052278</a></li>
<li><a href="https://github.com/drud/build-tools/blob/b37ef94e2a610ad40560e666423541430c8ba47b/.appveyor.yml">https://github.com/drud/build-tools/blob/b37ef94e2a610ad40560e66642...</a></li>
</ul>
<p>These work fine on docker-desktop (docker-ce) on Windows 10 Pro.</p>
<p>Thanks for all your help!</p></div>randytag:help.appveyor.com,2012-11-13:Comment/468922742019-04-23T20:27:21Z2019-04-23T20:27:21ZDocker: Linux containers in Windows docker workers<div><p>Maybe I should have stress more that usage sample I sent in previous message is essential. Use <code>Switch-DockerLinux</code> in PowerShell or <code>docker-switch-linux</code> in CMD before running those commands.</p>
<p>What you do now is the same as if you run those commands on your Windows 10 Pro without selecting <code>Switch to Linux Containers...</code> in the Docker UI.</p></div>Ilya Finkelshteyntag:help.appveyor.com,2012-11-13:Comment/468922742019-04-23T23:06:07Z2019-04-23T23:06:07ZDocker: Linux containers in Windows docker workers<div><p>Great progress, and thanks for your help. Now I'm to the point where I need the drive to be shared so it can be mounted. Is that something I can do or a config change you have to make?</p>
<p><a href="https://ci.appveyor.com/project/rfay/build-tools-w323a/builds/24055637">https://ci.appveyor.com/project/rfay/build-tools-w323a/builds/24055637</a></p></div>randytag:help.appveyor.com,2012-11-13:Comment/468922742019-04-24T00:05:42Z2019-04-24T00:05:42ZDocker: Linux containers in Windows docker workers<div><p>Again I would refer to the usage example. Drive D is shared exactly for this task.</p></div>Ilya Finkelshteyntag:help.appveyor.com,2012-11-13:Comment/468922742019-04-24T18:42:25Z2019-04-24T18:42:25ZDocker: Linux containers in Windows docker workers<div><p>Thanks for the coaching. I can get annoyed when people don't read the whole thing as well.</p>
<p>I'm not sure we'll be able to figure out how to adapt to not having the user directory shared, as that's the standard place for tempdir (which we share in tests), and just the natural place for everything.</p>
<p>Would it be possible to put the user directory on D?</p></div>randytag:help.appveyor.com,2012-11-13:Comment/468922742019-04-24T20:11:44Z2019-04-24T20:11:44ZDocker: Linux containers in Windows docker workers<div><p>No worries :) Try this:</p>
<pre>
<code>icacls %TMP% /grant everyone:f
mklink /J D:\user-tmp %TMP%
docker-switch-linux
docker run --rm -v D:/user-tmp:/user-tmp busybox ls /user-tmp</code>
</pre></div>Ilya Finkelshteyntag:help.appveyor.com,2012-11-13:Comment/468922742019-04-24T20:22:39Z2019-04-24T20:22:39ZDocker: Linux containers in Windows docker workers<div><p>I don't speak enough Windows in reality, but I think that's problematic because we use system-provided TMPDIR for tests, which is still going to be over on C: in the current situation right? So we'll be creating working repositories (in tests) in TMPDIR, and won't be able to mount them.</p>
<p>I'm pretty sure that if the homedir were on D it would all work out, not sure.</p>
<p>Alternately, if there's not a reason to avoid it, sharing C: would be natural and what most people would expect. Most people would expect both C: and D: (in this case) to be shared.</p></div>randytag:help.appveyor.com,2012-11-13:Comment/468922742019-04-24T20:38:00Z2019-04-24T20:38:00ZDocker: Linux containers in Windows docker workers<div><p>Yeah, we will consider sharing C as well. For now I still believe junction should work fro you. You can do <code>mklink /J D:\<any_folder_on_D> <any_folder_on_C></code> and then use that <code>D:\<any_folder_on_D></code> as a docker volume</p>
<p>But because share credentials are done with separate user (not <code>appveyor</code> under which build is running) you might need to change ACLs if (see <code>icacls</code> in sample) if this folder happens to be inside the <code>appveyor</code> user folder.</p></div>Ilya Finkelshteyntag:help.appveyor.com,2012-11-13:Comment/468922742019-04-24T21:17:42Z2019-04-24T21:17:42ZDocker: Linux containers in Windows docker workers<div><p>But... <code>C:\Users\testbot>echo %TEMP% C:\Users\testbot\AppData\Local\Temp</code></p>
<p>Doesn't that mean I'm going to end up creating directories in AppData\Local\Temp and not being able to mount them?</p></div>randytag:help.appveyor.com,2012-11-13:Comment/468922742019-04-24T21:22:11Z2019-04-24T21:22:11ZDocker: Linux containers in Windows docker workers<div><p>Please replace <code>TMP</code> with <code>TEMP</code> in the example I sent and try. I believe it should work for you.</p></div>Ilya Finkelshteyntag:help.appveyor.com,2012-11-13:Comment/468922742019-04-30T16:46:25Z2019-04-30T16:46:25ZDocker: Linux containers in Windows docker workers<div><p>Just a note that I haven't been successful with this yet, but probably haven't given it enough effort. The simple changes suggested did not get things happening on the D: drive. I'm pretty sure that if you end up moving the homedir to D: things will be OK, and absolutely certain that if you share the C: drive it will be OK. Or give us the option to change the shares.</p></div>randytag:help.appveyor.com,2012-11-13:Comment/468922742019-05-02T00:08:53Z2019-05-02T00:08:53ZDocker: Linux containers in Windows docker workers<div><p>I did some testing on a local WIndows 10 machine with 2 drives, and do not believe this strategy works. If you use <code>mklink /J</code> to create a shadow directory, Windows still knows where it really is, and docker still cares that the real thing isn't shared.</p>
<pre>
<code>F:\Users>dir
Volume in drive F is New Volume
Volume Serial Number is 84D6-41DB
Directory of F:\Users
05/01/2019 05:40 PM <DIR> .
05/01/2019 05:40 PM <DIR> ..
05/01/2019 05:40 PM <JUNCTION> rfay [c:\Users\rfay]</code>
</pre></div>randytag:help.appveyor.com,2012-11-13:Comment/468922742019-05-02T01:44:42Z2019-05-02T01:44:42ZDocker: Linux containers in Windows docker workers<div><p>OK, <a href="https://github.com/appveyor/ci/blob/master/scripts/docker-appveyor.psm1">here</a> is the way we do this (admittedly quite hacky). I think you can figure out how to deal with C: the same way :)</p></div>Ilya Finkelshteyn