You can write custom clone script (which does basically the same as AppVeyor does), wrap it into .cmd or .bat file and run with appveyor-retry as custom clone script (clone_script in YAML). Command might look like this:
- appveyor-retry myClone.cmd
If you prefer to restart new build, then it is easier for you to use $lastexitcode environment variable. However I suggest to do some measures to prevent infinite look. With variable APPVEYOR_FORCED_BUILD you can detect that build started from API, so do not start new build on error if APPVEYOR_FORCED_BUILD is true even if $lastexitcode is 255. If you don't like it because you, need more attempts than 2, you can pass custom environment variable to new build with API, and use it as count.