Detection of `cl.exe` just started failing in default build image

acrichton's Avatar


12 Dec, 2018 12:29 AM

Over at the rust-lang/rust repository we've seen a wave of new failures over the last hour or so and it looks like a platform update may have happened? One of our automated tasks needed to handle the new "starting" state for builds (a bug on our end) but our builds are now failing with couldn't find required command: "cl.exe". We detect cl.exe during various Rust builds and that relies on a whole mess of weird heuristics to find it on the system, but we were wondering if a recent change to the image switched up how or where things were installed? If possible, could we revert back temporarily to the previous image?

  1. Support Staff 1 Posted by Feodor Fitsner on 12 Dec, 2018 12:36 AM

    Feodor Fitsner's Avatar

    We are rolling out the update to Visual Studio 2017 image with the latest VS 2017 15.9.4 which was already applied to GCE environment. You may get back to previous Previous Visual Studio 2017 image.

  2. 2 Posted by acrichton on 12 Dec, 2018 01:12 AM

    acrichton's Avatar

    Ok thanks!

    Is there an expected timescale for when the image rotation happens again? (when the current "previous" is overwritten with the current non-previous presumably).

    Additionally, do y'all have a list of changes between the last image and this one? (to help us dig into what's wrong with our code)

  3. Support Staff 3 Posted by Feodor Fitsner on 12 Dec, 2018 01:18 AM

    Feodor Fitsner's Avatar

    This is the list of changes:

    My best guess it is VS 2017 15.9.4 that broke your build thought that’s weird the location of cl.exe has changed.

    We try to update images in 2-3 weeks but keeping previous image, especially for VS 2017, depends on the number of issues VS 2017 upgrade brings to people. Despite it’s supposed to be “minor” VS update it’s usually quite disruptive.

  4. 4 Posted by acrichton on 12 Dec, 2018 01:25 AM

    acrichton's Avatar

    Hm we tried out Previous but it's still getting the same error, did we configure that incorrectly though?

  5. Support Staff 5 Posted by Feodor Fitsner on 12 Dec, 2018 01:35 AM

    Feodor Fitsner's Avatar

    Well, “Preview” image hasn’t been changed. I forgot you are on Preview. Will take a look - maybe it’s new jobs scheduling thought could it be some change on your side?

  6. 6 Posted by acrichton on 12 Dec, 2018 01:42 AM

    acrichton's Avatar

    Ah right I forgot about that too! Our last successful build was a few hours ago with our next failing build just recently having no substantive changes between the two. Doesn't mean that it's not our bug by any means, but I think at least it's unlikely that it was a change on our end.

    If the preview image didn't change though then... that's a bit of an oddity!

  7. Support Staff 7 Posted by Feodor Fitsner on 12 Dec, 2018 01:49 AM

    Feodor Fitsner's Avatar

    OK, let me double check and get back to you.

  8. Support Staff 8 Posted by Feodor Fitsner on 12 Dec, 2018 04:06 AM

    Feodor Fitsner's Avatar

    Alex, it must be a new job scheduling algorithm that routed that build to a wrong image (Visual Studio 2017 instead of Preview), sorry for that.

    I've put your account back to previous algo while investigating. And, yes, please ensure your infrastructure code supports starting state.

  9. 9 Posted by acrichton on 12 Dec, 2018 04:23 AM

    acrichton's Avatar

    Thanks for taking a look! A newly scheduled build looks like it's making progress, past the point of failure before, so I believe that fix has worked!

    Definitely a bug on our end for not handling starting, and should be fixed now!

    Our builds should work as-is in the Visual Studio 2017 images anyway, so I'll start investigating why they aren't working and fix that bug independently too.

    Thanks again so much for the fast response here!

  10. Support Staff 10 Posted by Feodor Fitsner on 12 Dec, 2018 04:24 AM

    Feodor Fitsner's Avatar

    Great, thanks for the update!

  11. Support Staff 11 Posted by Feodor Fitsner on 16 Dec, 2018 11:28 PM

    Feodor Fitsner's Avatar

    Hi Alex,

    Are you guys ready to be moved to a new job scheduling algorithm (remember that new Starting status)? Other than that new status the scheduling works as expected now.

  12. 12 Posted by acrichton on 17 Dec, 2018 02:54 AM

    acrichton's Avatar

    Oh sorry that wasn't clear earlier, but yes we fixed the starting status showing up as soon as we saw it, it just happened to happen at the same time as the other part of the issue! It sounds like we're still scheduled with the same image we're using today, so if that's the case we can be switched over at any time!

  13. acrichton closed this discussion on 15 Jan, 2019 07:49 PM.

Comments are currently closed for this discussion. You can start a new one.

Keyboard shortcuts


? Show this help
ESC Blurs the current field

Comment Form

r Focus the comment reply box
^ + ↩ Submit the comment

You can use Command ⌘ instead of Control ^ on Mac

Recent Discussions

21 Aug, 2019 11:06 PM
21 Aug, 2019 02:08 PM
21 Aug, 2019 01:37 PM
21 Aug, 2019 10:21 AM
21 Aug, 2019 09:10 AM


21 Aug, 2019 05:56 AM
21 Aug, 2019 03:08 AM
20 Aug, 2019 11:33 PM
20 Aug, 2019 08:07 PM
20 Aug, 2019 05:50 PM
20 Aug, 2019 12:35 PM