Service Fabric Cluster Cleanup hanging

Mauricio Guevara Souza's Avatar

Mauricio Guevara Souza

26 Jul, 2018 09:18 AM

Hello:

When I try to start Service Fabric local cluster, it hangs forever. I try to clean using "CleanCluster.ps1" script and i get "Removing cluster configuration..." and it gets stuck there. This was working before. I am using Visual Studio 2017 image.

  1. Support Staff 1 Posted by Ilya Finkelshte... on 26 Jul, 2018 10:31 AM

    Ilya Finkelshteyn's Avatar

    Can you add non-blocking RDP at init stage as described here, and connect to build worker when it stuck next time? Maybe you will see the reason it stuck. Let us know what you found.

  2. 2 Posted by Mauricio Guevar... on 26 Jul, 2018 10:44 AM

    Mauricio Guevara Souza's Avatar

    I did that and apparently the problem is that the Service Fabric Host Service is not starting. I tried to start it manually and it didnt start.

  3. Support Staff 3 Posted by Ilya Finkelshte... on 26 Jul, 2018 10:47 AM

    Ilya Finkelshteyn's Avatar

    Did you check Event viewer? It is not very probable that service does not start without any errors or warnings generated.

  4. 4 Posted by Mauricio Guevar... on 26 Jul, 2018 10:51 AM

    Mauricio Guevara Souza's Avatar

    Just to clarify what i did is to remove the lines that invoke Service Fabric scripts from appveyor.yml and i RD the agent and tried to run the CleanCluster.ps1 script from a PS window inside the agent with the same result. Then I checked the services and i notices that the Service Fabric Host Service status was "starting" but it got stuck there. Then i cancel this build and fired another one and RD the agent and this time i just went to the services and manually tried to start the Service Fabric Host Service but it didn't start.

  5. Support Staff 5 Posted by Ilya Finkelshte... on 26 Jul, 2018 11:11 AM

    Ilya Finkelshteyn's Avatar

    OK, we need a repro. Can you create a repro in a public repository? Or if this can be reproduced by simple coping your appveyor.yml, please share it.

    Also please send us a links to the last good and first bad builds.

  6. 6 Posted by Mauricio Guevar... on 26 Jul, 2018 11:17 AM

    Mauricio Guevara Souza's Avatar

    It can be reproduced by just adding this to the appveyor.yml

     image: Visual Studio 2017
    init:
    - ps: Write-Host "Setting up local cluster"
    - ps: ."$ENV:ProgramFiles\Microsoft SDKs\Service
                Fabric\ClusterSetup\CleanCluster.ps1"
    - ps: ."$ENV:ProgramFiles\Microsoft SDKs\Service
               Fabric\ClusterSetup\DevClusterSetup.ps1"
    - ps: Write-Host "Validating local cluster"
    - ps: Connect-ServiceFabricCluster localhost:19000

    It will get stuck in the CleanCluster.ps1 script. This was working like a month ago and i tested and it works in the "Visual Studio 2017 Preview" image.

  7. 7 Posted by Mauricio Guevar... on 26 Jul, 2018 11:32 AM

    Mauricio Guevara Souza's Avatar
  8. Support Staff 8 Posted by Ilya Finkelshte... on 26 Jul, 2018 05:13 PM

    Ilya Finkelshteyn's Avatar

    This indeed appears to be the problem. But you are not using Visual Studio 2017, you are using Previous Visual Studio 2017. Can you please clarify why?

  9. 9 Posted by Mauricio Guevar... on 27 Jul, 2018 07:27 AM

    Mauricio Guevara's Avatar

    I started to use Previous Visual Studio 2017 because i start having this problem with Visual Studio 2017 but now i have the same problem with Previous Visual Studio 2017 and with Visual Studio 2017. Now i noticed that only Visual Studio 2017 Preview will start the local service fabric cluster correctly. Here is a link of a build using Visual Studio 2017 image that has the same problem.

    https://ci.appveyor.com/project/Configit/resource-repository/build/1.0.1626

  10. Support Staff 10 Posted by Ilya Finkelshte... on 27 Jul, 2018 09:37 AM

    Ilya Finkelshteyn's Avatar

    Oh, please do not do that silently next time. If you found this kind of issue, please report and we will fix it before or as part of next major image update (when current image becomes previous and old previous is gone).

    I filed this issue to track it. Do you need some temporary workaround or you are unblocked with Visual Studio 2017 Preview?

  11. 11 Posted by Mauricio Guevar... on 27 Jul, 2018 10:59 AM

    Mauricio Guevara's Avatar

    Hello:

    I would need a temporary workaround if possible.

  12. Support Staff 12 Posted by Ilya Finkelshte... on 28 Jul, 2018 01:06 PM

    Ilya Finkelshteyn's Avatar

    For workaround, add the following in the beginning of init before your cluster setup commands:

    - '"C:\Program Files\Docker\Docker\Docker for Windows Installer.exe" uninstall'
    - ps: $fabricSdk = Get-WmiObject Win32_Product | ? {$_.Name -match "Microsoft Azure Service Fabric SDK"}; $fabricSdk.Reinstall(10)
    

    Let us know if this works for you.

  13. 13 Posted by Mauricio Guevar... on 30 Jul, 2018 08:17 AM

    Mauricio Guevara's Avatar

    Thanks for the workaround. Is there an approximate date to have this fixed in the Visual Studio 2017 image so i can remove the workaround?

  14. Support Staff 14 Posted by Ilya Finkelshte... on 30 Jul, 2018 08:41 AM

    Ilya Finkelshteyn's Avatar

    It should not take longer than a month from now, probably less. Please subscribe to technical updates at https://ci.appveyor.com/profile to be notified when image update happens. Or watch this issue.

  15. Support Staff 15 Posted by Feodor Fitsner on 09 Aug, 2018 10:34 PM

    Feodor Fitsner's Avatar

    Hi guys,

    I've figured out a better, and faster workaround! Add the following line into appveyor.yml:

    install:
    - ps: iex ((new-object net.webclient).DownloadString('https://raw.githubusercontent.com/appveyor/ci/master/scripts/fix-service-fabric-docker.ps1'))
    

    The script downloads Docker CE and puts its directory to PATH so Service Fabric could see "correct" dockerd.exe while starting.

    Successful build run: https://ci.appveyor.com/project/appveyor-tests/service-fabric-docker

Reply to this discussion

Internal reply

Formatting help / Preview (switch to plain text) No formatting (switch to Markdown)

Attaching KB article:

»

Attached Files

You can attach files up to 10MB

If you don't have an account yet, we need to confirm you're human and not a machine trying to post spam.

Keyboard shortcuts

Generic

? Show this help
ESC Blurs the current field

Comment Form

r Focus the comment reply box
^ + ↩ Submit the comment

You can use Command ⌘ instead of Control ^ on Mac