Flaky

Flaky tests are:

  • Tests passed and failed with the same code.
  • Tests passed before but sometimes failed
  • Very hard to fix. The person who fix is usually not the author. Even the author has no idea. The tests started to fail at some point.
  • Very expensive to your team. They get in the way when you need to deploy something.
  • People lost confidence of test suite.
  • Mitigations of flaky tests are costly: qurantine, rerun flaky, retry few times.

Flaky tests usually caused by:

  • Time, Timezone
  • Concurrency
  • Wait
  • Order
  • 3rd party library

Good posts on Flaky