Home → Problems →

Can't ssh from a Powershell deploy script

• Edit

brliron

21 Jun, 2024 10:48 AM

Hello,

I'm trying to use ssh from a Powershell deploy script, and I'm encountering 2 issues.

1. Without doing anything out of the ordinary, just trying to ssh to a server I control using this script:
   

   echo "Before ssh"
   ssh [email blocked]
   echo "After ssh"
   

   I have an error about another key for the same server being in the known_hosts file. This is the only time I try to ssh to this server during the build, the server shouldn't be in the known_hosts file - and even if it was, the host key didn't change recently.
   Build logs: https://ci.appveyor.com/project/brliron/thcrap-dev/builds/50066316
   Trying the same command from a Powershell shell on my local machine gives the expected result:
   

   [email blocked]: Permission denied (publickey).
   

   I have the same error if I try with another server - here, using brliron.fr, which points to another server I control. Build logs: https://ci.appveyor.com/project/brliron/thcrap-dev/builds/50063686
   For the sake of testing, I tried to print the content of the known_hosts file, and it doesn't help me understand this issue: https://ci.appveyor.com/project/brliron/thcrap-dev/builds/50063440

2. After that, I tried to remove the known_hosts file using this deploy script:
   

   rm ~.ssh\known_hosts
   echo "Before ssh"
   ssh [email blocked]
   echo "After ssh"
   

   But it just gets stuck on the ssh command. Our builds usually take 5-6 minutes, I cancelled this one after 35 minutes: https://ci.appveyor.com/project/brliron/thcrap-dev/builds/50066395 (and the "Before ssh" line is printed at 6:09).

1. 1 Posted by brliron on 21 Jun, 2024 11:43 AM

   

   I forgot to say that I'm using the "Visual Studio 2019" build worker image.

   • Edit

2. Support Staff 2 Posted by Owen McDonnell on 21 Jun, 2024 10:28 PM

   

   Is this a new issue, by which i mean was your configuration with an ssh call working before?
   I'm far from an SSH guru, but have you tried to add the host with an ssh-keyscan command?

   • Edit

3. 3 Posted by brliron on 23 Jun, 2024 08:50 AM

   

   I've never tried using ssh from AppVeyor before so I can't comment on whether it's a new issue or not.
   I tried adding the host by running ssh-keyscan on my local machine and adding the output to the known_hosts file with an echo command, it didn't change anything: https://ci.appveyor.com/project/brliron/thcrap-dev/builds/50063512 .

   On the other hand, I didn't try running the ssh-keyscan command directly from the deploy script because it should have the same output, but just to be sure, I'm going to try it anyway.

   After trying, it didn't help. Here is a build with ssh-keyscan kosuzu.thpatch.net >> ~\.ssh\known_hosts being run before the ssh command: https://ci.appveyor.com/project/brliron/thcrap-dev/builds/50073307
   And another one where I tried to remove the known_hosts file and then to add our server key with ssh-keyscan: https://ci.appveyor.com/project/brliron/thcrap-dev/builds/50073397 . In the same scenario, I also got a timeout from ssh instead of it being stuck: https://ci.appveyor.com/project/brliron/thcrap-dev/builds/50073383
   In both cases, the behavior is the same as without the ssh-keyscan command.

   • Edit

4. Support Staff 4 Posted by Owen McDonnell on 25 Jun, 2024 06:12 PM

   

   When you ssh to this server from your local machine, you are being prompted for credentials or you are using ssh keys?

   • Edit

5. 5 Posted by brliron on 25 Jun, 2024 10:20 PM

   

   The server is configured to use ssh keys for authentication. On the machine I use for local testing, I don't have any ssh key in ~/.ssh, so I'm getting this error (which is expected):
   

   [email blocked]: Permission denied (publickey).
   

   And specifying the key on the command line with ssh [email blocked] -i .\id_ed25519 results in this:
   

   PTY allocation request failed on channel 0
   Hello world!
   Connection to kosuzu.thpatch.net closed.
   

   (the SSH server is configured to execute a script when connecting with this user instead of opening a shell, which for now only displays "Hello world!" but will take care of the deployment at some point).

   I tried to use the ssh key from the deploy script on AppVeyor, without success. The behavior on AppVeyor is the same with or without specifying the ssh key (for example, this job https://ci.appveyor.com/project/brliron/thcrap-dev/builds/50063512 uses an ssh key and this job https://ci.appveyor.com/project/brliron/thcrap-dev/builds/50073307 doesn't).

   • Edit

6. Support Staff 6 Posted by Owen McDonnell on 26 Jun, 2024 10:49 PM

   

   Regarding your first example that uses the ssh key - when i run that locally I get the same Pseudo-terminal will not be allocated.. message. If i add -T flag (i.e. ssh -T [email blocked] -i .\id_ed25519) then the command runs without that error, instead returning a bash error about the echo command.
   You also mentioned a timeout on some build. That may be due to the ssh server attempting to detect an interactive terminal.

   • Edit

7. 7 Posted by brliron on 01 Jul, 2024 03:32 PM

   

   After a lot more testing and remembering that ssh -vvv is a thing, I think I found a way to fix the 2 problems I encountered - one of them would be on my end, and one of them would be on your end.

   The timeout

   That one is the one I need to fix on my end (if you're curious, you can read, if you don't care, you can skip to the next section "wrong host key"). This one seems to happen when I don't have a known_hosts file. I guess that it tries to display this message:
   

   The authenticity of host 'kosuzu.thpatch.net (2604:a880:400:d1::755:f001)' can't be established.
   ED25519 key fingerprint is SHA256:PxPrL5UYr/TP4e+FUgUzqvlu1dva9EJrGMCHkItQPFQ.
   This key is not known by any other names.
   Are you sure you want to continue connecting (yes/no/[fingerprint])?
   

   And then it gets stuck waiting for an answer. The weird thing is that this message isn't displayed in the AppVeyor build logs, but that's my only guess.
   But, you suggested me to add the server key to the file, which I tried, why didn't it fix this problem? Because the command I used for that, ssh-keyscan kosuzu.thpatch.net > ~\.ssh\known_hosts, saves the file in UTF-16, while ssh expects UTF-8. Because of that, the file gets ignored and it's like if it wasn't there.
   The fix is for me to use this command instead: ssh-keyscan kosuzu.thpatch.net | Set-Content -Encoding utf8 ~\.ssh\known_hosts

   Wrong key host

   The timeout issue happens after I remove the known_hosts file pre-installed on the instance. With the default known_hosts file, I had another error:
   

   @@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
   @    WARNING: REMOTE HOST IDENTIFICATION HAS CHANGED!     @
   @@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
   IT IS POSSIBLE THAT SOMEONE IS DOING SOMETHING NASTY!
   Someone could be eavesdropping on you right now (man-in-the-middle attack)!
   It is also possible that a host key has just been changed.
   The fingerprint for the ECDSA key sent by the remote host is
   SHA256:ubPkz2l+Gh592f/Axh/Zc/zfLkLNmceKcimdkDi3Hjk.
   Please contact your system administrator.
   Add correct host key in C:\Users\appveyor/.ssh/known_hosts to get rid of this message.
   Offending ECDSA key in C:\Users\appveyor/.ssh/known_hosts:597
   ECDSA host key for kosuzu.thpatch.net has changed and you have requested strict checking.
   Host key verification failed.
   

   I believe that this one is caused by an error in the pre-installed known_hosts file. A normal line in this file can look like these:
   

   github.com,20.26.156.214 ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIOMqqnkVzrm0SdG6UOoqKLsabgH5C9okWi0dh2l9GKJl
   bitbucket.org,3.26.128.* ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIIazEu89wgQZ4bqs3d63QSMzYVa0MuJ2e2gKTKqu+UUO
   

   But the 3 last lines are this:
   

   gitlab.com,* ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIAfuCHKVTjquxvt6CM6tdG4SLp1Btn/nOeHHE5UOzRdf
   gitlab.com,* ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCsj2bNKTBSpIYDEGk9KxsGh3mySTRgMtXL583qmBpzeQ+jqCMRgBqB98u3z++J1sKlXHWfM9dyhSevkMwSbhoR8XIq/U0tCNyokEi/ueaBMCvbcTHhO7FcwzY92WK4Yt0aGROY5qX2UKSeOvuP4D6TPqKF1onrSzH9bx9XUf2lEdWT/ia1NEKjunUqu1xOB/StKDHMoX4/OKyIzuS0q/T1zOATthvasJFoPrAjkohTyaDUz2LN5JoH839hViyEG82yB+MjcFV5MU3N1l1QL3cVUCh93xSaua1N85qivl+siMkPGbO5xR/En4iEY6K2XPASUEMaieWVNTRCtJ4S8H+9
   gitlab.com,* ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBFSMqzJeV9rUzU4kWitGjeR4PWSa29SPqJ1fVkhtj3Hw9xjLVXVYrU9QlYWrOLXBpQ6KWjbjTDTdDkoohFzgbEY=
   

   These lines were written assuming that the 1st field of each line was in the format "hostname,ip", using * to say "match any IP from gitlab.com". But the documentation (the section "SSH_KNOWN_HOSTS FILE FORMAT" in the sshd man page https://linux.die.net/man/8/sshd ) says this about this field:

   Hostnames is a comma-separated list of patterns (‘’ and ‘?’ act as wildcards); each pattern in turn is matched against the host name. In these lines, gitlab.com and `are just 2 patterns used to match the host server, and*` ends up matching every server, including the one I'm trying to connect to, which of course doesn't have the same host key as Gitlab's servers.

   The fix would be to change the pre-installed file on the AppVeyor instance, to either replace the 3 * wildcards with a pattern unique to Gitlab's IP addresses, or just to remove the wildcard and keep only gitlab.com in the Hostnames field.

   In the meanwhile, I can work around this issue on my end by removing the pre-installed known_hosts file.

   • Edit

8. Support Staff 8 Posted by Owen McDonnell on 04 Jul, 2024 08:44 PM

   

   Thanks for all the clarifications and glad to hear you've been able to work around it. This is not an area of expertise for me ; )
   I've added modification of the known_hosts file to our next image update. You can watch it here

   • Edit

9. Feodor Fitsner closed this discussion on 03 Sep, 2024 09:03 PM.