I could use some more clarification. I don't believe anything in that documentation explains what yml settings "build mode" corresponds to. Is the error message incorrectly worded? The build-configuration documentation enumerates the various flavors of Script blocks, but I don't see anything in there about a "Script Mode".
If your YAML file contains build: section, AppVeyor will work in "MSBuild" mode. Of it does not contain build: section, but contains build_script: section, it will work in a "Script Mode". Also you can explicitly disable "MSBuild" mode with setting build: off.
Playing with Build tab in UI, saving and then pressing Export YAML should give you a good idea of those settings.