Flaky

Flaky tests are:

  • Tests passed and failed with the same code.
  • Tests passed before but sometimes failed

Flaky tests caused problems like:

  • Engineering time. It is very hard to fix. The person who fix is usually not the author.
  • Very expensive to your team. They get in the way when you need to hotfix deploy something.
  • People lost confidence of your test suite. They stopped/less willing to write tests.
  • Mitigations of flaky tests are costly: qurantine, rerun flaky, retry few times, rerun your tests hourly, daily.

Flaky tests usually caused by:

  • Testing for fixed Time and Timezone
  • Concurrency — Things run together
  • Wait — Assert state fo elements not showing up quick enough on UI.
  • Untidy cleanup between tests. Order of Tests.
  • Order of result from database is not the same as you think: not behaves the same locally and on CI
  • 3rd-party library (Capybara)
  • Checking datetime, time, elapsed time fields but precision is not enough
  • Checking for ambiguous HTML elements (two .btn on the page?)

Flaky tests usually tackled by:

  • Automatic Retry
  • Rerun til passed
  • Having a team who fixed them
  • Skip them and record somewhere
  • Rerun regularly
  • Bisect against test database state (seed) (minitest-bisect)