Slow File IO?

Weirong Zhu's Avatar

Weirong Zhu

14 Sep, 2015 11:32 PM

Since this morning, most of our builds failed. For builds that happen to be successful, they also seem to be much slower than before.

By digging into it, it seems the IO on the VM is very slow. From our log, we even see cases where creating and writing a 3KB file takes more than 3 seconds. The slow IO caused some unexpected timeout in our code, thus cause the test run (part of the build) to fail. Any idea what has been changed lately?

  1. Support Staff 1 Posted by Feodor Fitsner on 14 Sep, 2015 11:43 PM

    Feodor Fitsner's Avatar

    We are seeing like a usage spike right now (good), slowing down OSS environment (bad). We are adding additional hardware in the meantime - thank you for your patience.

  2. 2 Posted by Weirong Zhu on 14 Sep, 2015 11:47 PM

    Weirong Zhu's Avatar

    More information:

    For one successful run:, it's a lot slower than the same build for the same commit 3 days ago:

    By comparing the two, we can see that the
    * build time increases from 1:17 minutes to 2:26 minutes
    * copy binary step increase from 0:20 minutes to 0:56 minutes. This is a pure IO step, and it's almost three times slower now.
    * test time increases from 3:39 minutes to 5:07 minutes
    * total time increase from 5:18 minutes to 8:41 minutes
    We have more runs failed due to even slower perf that triggered timeout in the code.

  3. Ilya Finkelshteyn closed this discussion on 25 Aug, 2018 02:00 AM.

Comments are currently closed for this discussion. You can start a new one.

Keyboard shortcuts


? Show this help
ESC Blurs the current field

Comment Form

r Focus the comment reply box
^ + ↩ Submit the comment

You can use Command ⌘ instead of Control ^ on Mac