Problem with PowerShell Core 6.1.1 in Ubuntu and Ubuntu1804 images

Florian Feldhaus's Avatar

Florian Feldhaus

29 Dec, 2018 07:42 PM

After PowerShell Core was upgraded to 6.1.1 in the Ubuntu and Ubuntu1804 image a few users encountered a strange issue with Pester tests. The root cause seems to be, that PowerShell 6.1.1 in the Appveyor Ubuntu and Ubuntu1804 images treat Properties of Hashtables case sensitive (e.g. $test=@{result="result"} and then $test.result returns "result", but $test.Result does not return anything.

The issue does not occur when we install the same PowerShell Version on the official Docker 16.04 or 18.04 image using the steps described by Microsoft here:
https://docs.microsoft.com/en-us/powershell/scripting/install/installing-powershell-core-on-linux?view=powershell-6

The issue also does not occur on Windows or Mac OS X.

A detailed discussion of the issue can be found here:
https://github.com/pester/Pester/issues/1101

As the issue can always be reproduced with Appveyor images, but not with other Ubuntu images, can you describe exactly how PowerShell 6.1.1 is installed in the Appveyor images?

Can you check on your side to identify anything which could cause this behavior?

  1. 1 Posted by Florian Feldhau... on 29 Dec, 2018 08:09 PM

    Florian Feldhaus's Avatar

    The problem seems to be caused by PowerShell issue 7761
    https://github.com/PowerShell/PowerShell/issues/7761

    Until this is fixed in PowerShell or .NET Core, the issue could be worked around by changing the default LANG setting from en-US-POSIX to en-US.

  2. Support Staff 2 Posted by Wasa Pleshakov on 30 Dec, 2018 09:58 AM

    Wasa Pleshakov's Avatar

    Ubuntu and Ubuntu1804 images have LANG set to C.UTF-8 https://www.appveyor.com/docs/linux-images-software/#configuration

    But you're right, one have to export LANG=en_US.utf8 to fix issue

    appveyor@av-ad9a4be53e8b43db95ae78a7072992fb:~$ export LANG=en_US.utf8
    appveyor@av-ad9a4be53e8b43db95ae78a7072992fb:~$ pwsh
    PowerShell 6.1.1
    Copyright (c) Microsoft Corporation. All rights reserved.


    https://aka.ms/pscore6-docs Type 'help' to get help.


    PS /home/appveyor> $test=@{result="result"} PS /home/appveyor> $test.Result result PS /home/appveyor> $test.result result PS /home/appveyor>
  3. Support Staff 3 Posted by Wasa Pleshakov on 30 Dec, 2018 10:05 AM

    Wasa Pleshakov's Avatar

    Installation of Powershell on Appveyor's images is very straightforward:
    setup https://packages.microsoft.com/config/ubuntu/xenial/prod.list repository and apt-get install powershell

    $ apt-cache show powershell
    Package: powershell
    Priority: extra
    Section: shells
    Installed-Size: 154369
    Maintainer: PowerShell Team <[email blocked]>
    Architecture: amd64
    Version: 6.1.1-1.ubuntu.16.04
    Depends: libc6, libgcc1, libgssapi-krb5-2, liblttng-ust0, libstdc++6, zlib1g, libssl1.0.0, libicu55
    Filename: pool/main/p/powershell/powershell_6.1.1-1.ubuntu.16.04_amd64.deb
    Size: 58139140
    MD5sum: dffe2a52714cb3fbef26e47a76ad1453
    SHA1: 3ded88eb37969c1b901937ddf33982751d34a600
    SHA256: 2b6e8722c87131505da78487f0fb4b0e075fdeeb0c6cd8b5405441c1bed8a0c1
    SHA512: d6fe1b58594d82ef5a0d47646052964ee37f8ac20b78d2d07e90b1c9d25697f30c236a2c3a402302fac60aee20db5fc31f47ad64ca61d3ff3dac7ca0e19cbb23
    Description: PowerShell is an automation and configuration management platform.
     It consists of a cross-platform command-line shell and associated scripting language.
    Description-md5: 6ee533ea0564ca22e8caa699e30ba649
    Homepage: https://microsoft.com/powershell
    Vendor: Microsoft Corporation
    License: MIT License
    
    Obviosly, we use bionic repository at Ubuntu1804.
  4. 4 Posted by Florian Feldhau... on 30 Dec, 2018 09:14 PM

    Florian Feldhaus's Avatar

    Thanks for all the details. Can you help me get the LANG setting right so that my tests run with en_US.utf8? I tried to put the following into my appveyor.yml but the powershell session inside test_script does show the C.utf8 LANG settings:

    environment:
      LANG: en_US.utf8
    test_script:
      - ps: Get-Culture
    
    The Get-Culture shows in the logs that it is still running with the wrong LANG setting:
    Running "test_script" scripts
    Get-Culture
    Parent                         : en-US
    Name                           : en-US-POSIX
    

    The Get-Culture output should be

    Running "test_script" scripts
    Get-Culture
    Parent                         : en-US
    Name                           : en-US
    
  5. Support Staff 5 Posted by Wasa Pleshakov on 31 Dec, 2018 07:46 AM

    Wasa Pleshakov's Avatar

    You may set LANG environment variable on init stage of build like this:

    init:
      - sh: export LANG=en_US.utf8
    
    But then you have to run PowerShell scripts like this:
    - sh: pwsh -c 'get-culture'
    
    This is due to build agent creates 2 shells to execute your commands from appveyor.yml:
    - one is bash shell to execute - sh: commands - other is PowerShell shell to execute - ps: commands and there is no way to set environment variables of one shell from within another shell.
  6. 6 Posted by Florian Feldhau... on 31 Dec, 2018 12:03 PM

    Florian Feldhaus's Avatar

    Thanks for the suggested workaround, it worked for me, even though it requires to duplicate the test specification once for Linux and once for Windows. Also the log output is looking very strange, probably because the output contains unnecessary spaces which creates very long output lines. Can you have a look at it?
    https://ci.appveyor.com/project/ffeldhaus/s3-client/builds/21304749...

    Can you please consider allowing to change the default LANG setting in Linux so that it gets picked by all processes, including the ps process? As at least .NET Core and PowerShell have different behavior on different cultures (especially C or tr-TR or other non latin cultures), it would be very helpful if we could specify the culture (LANG) per test configuration to test different cultures.

  7. 7 Posted by Florian Feldhau... on 31 Dec, 2018 12:11 PM

    Florian Feldhaus's Avatar

    There are very strange side effects with regards to log and console output. Please check the console and log output of https://ci.appveyor.com/project/ffeldhaus/s3-client/builds/21304749 and compare it to earlier builds.

  8. Support Staff 8 Posted by Wasa Pleshakov on 02 Jan, 2019 09:01 PM

    Wasa Pleshakov's Avatar

    Can you please check next:

    init:
     - sh: echo LANG=en_US.UTF-8 >/etc/default/locale
    
  9. 9 Posted by Florian Feldhau... on 02 Jan, 2019 09:06 PM

    Florian Feldhaus's Avatar

    Does not work as the process does not have enough permissions:

    echo LANG= en_US.UTF-8 >/etc/default/locale
    bash: /etc/default/locale: Permission denied
    Command exited with code 1
    Build failed

  10. Support Staff 10 Posted by Wasa Pleshakov on 02 Jan, 2019 09:24 PM

    Wasa Pleshakov's Avatar

    I should foresee that
    please add sudo there:

    init:
     - sh: sudo echo LANG=en_US.UTF-8 >/etc/default/locale
    
  11. 11 Posted by Florian Feldhau... on 02 Jan, 2019 09:45 PM

    Florian Feldhaus's Avatar

    Same issue:

    Build started
    2Running "init" scripts
    3sudo echo LANG=en_US.UTF-8 >/etc/default/locale
    4bash: /etc/default/locale: Permission denied
    5Command exited with code 1
    6Build failed

    Von meinem iPhone gesendet

  12. Support Staff 12 Posted by Wasa Pleshakov on 03 Jan, 2019 12:24 AM

    Wasa Pleshakov's Avatar

    I'm sorry I gave you the wrong advice. Sudo with redirects is always so tricky.
    It should be executed like this (100% working version):

    init:
     - sh: sudo bash -c 'echo LANG=en_US.UTF-8 >/etc/default/locale'
    
  13. 13 Posted by Florian Feldhau... on 03 Jan, 2019 09:19 AM

    Florian Feldhaus's Avatar

    Hi Wasa,

    It is indeed working now, but does not have the expected result, the culture in PowerShell is still en-US-POSIX:

    Build started
    Running "init" scripts
    sudo bash -c 'echo LANG=en_US.UTF-8 >/etc/default/locale'

    Running "test_script" scripts
    Get-Culture
    Parent : en-US
    LCID : 4096
    KeyboardLayoutId : 1033
    Name : en-US-POSIX
    IetfLanguageTag : en-US-POSIX

  14. Support Staff 14 Posted by Wasa Pleshakov on 04 Jan, 2019 10:56 AM

    Wasa Pleshakov's Avatar

    Florian,

    We prepared the new version of Build Agent which will set LANG variable before starting shells. But due to the reorganization of servers, we can't roll it out right away. ETA is next week.

  15. Support Staff 15 Posted by Wasa Pleshakov on 08 Jan, 2019 06:09 PM

    Wasa Pleshakov's Avatar

    Florian, Can you please test next configuration:

    environment:
       LANG: en_US.utf-8
       APPVEYOR_BUILD_WORKER_CLOUD: gce-test
    
    This instructs Appveyor CI to schedule build on gce-test cloud which contains ubuntu images with the new version of Build Agent. And instruct Build Agent to set LANG variable before starting shells.

    PS. As soon we rollout the new images there will be no need to set APPVEYOR_BUILD_WORKER_CLOUD: gce-test.

  16. 16 Posted by Florian Feldhau... on 09 Jan, 2019 10:13 PM

    Florian Feldhaus's Avatar

    This didn't seem to have worked. PowerShell still reports the language as en-US-POSIX (e.g. C.utf8).

    Can you check somehow to see if it really was running on gce-test?

    This is the test:
    https://ci.appveyor.com/project/ffeldhaus/s3-client/builds/21505832

    And this the change in appveyor for the test:
    https://github.com/ffeldhaus/S3-Client/commit/eb3f3abd713593392c448...

  17. Support Staff 17 Posted by Feodor Fitsner on 09 Jan, 2019 11:55 PM

    Feodor Fitsner's Avatar

    Hi Florian,

    Could you give it another try please? We've enabled "cloud override" feature for your account.

  18. 18 Posted by Florian Feldhau... on 10 Jan, 2019 12:36 AM

    Florian Feldhaus's Avatar

    Now it finally worked! Languate in PowerShell is now en-US (e.g. en-US.utf8).
    Thanks a lot!

  19. Support Staff 19 Posted by Wasa Pleshakov on 16 Jan, 2019 07:15 PM

    Wasa Pleshakov's Avatar

    Florian,
    We updated Ubuntu images. Please remove line APPVEYOR_BUILD_WORKER_CLOUD: gce-test from your configuration. LANG: en_US.utf-8 should work anyway.

  20. Ilya Finkelshteyn closed this discussion on 18 Mar, 2019 09:00 PM.

Comments are currently closed for this discussion. You can start a new one.

Keyboard shortcuts

Generic

? Show this help
ESC Blurs the current field

Comment Form

r Focus the comment reply box
^ + ↩ Submit the comment

You can use Command ⌘ instead of Control ^ on Mac

Recent Discussions

21 Aug, 2019 11:06 PM
21 Aug, 2019 02:08 PM
21 Aug, 2019 01:37 PM
21 Aug, 2019 10:21 AM
21 Aug, 2019 09:10 AM

 

21 Aug, 2019 05:56 AM
21 Aug, 2019 03:08 AM
20 Aug, 2019 11:33 PM
20 Aug, 2019 08:07 PM
20 Aug, 2019 05:50 PM
20 Aug, 2019 12:35 PM