To the best of my knowledge that should do the trick! On my machine where I've done some dabbling before I only see "Visual C++ compilers and libraries for ARM64" (and ARM, but I doubt that's required for aarch64 dev). I specifically don't have the spectre-related things install (I'm not even sure what those are), so I don't think they should be necessary for a bare minimum toolchain!
Actually, that screenshot is what we currently have enabled on VS 2017 workers. So, apparently, ARM is already enabled - I've been wondering what you have locally that it's building locally but not with us?
Oh dear, that sounds like the issue may be on our end then! I'm unfortunately away from my windows computer for a few weeks, so we'll try to keep following up on this from our end. Thanks so much for looking into this though!