De-Flake Your Tests: Automatically Locating Root Causes of Flaky Tests in Code At Google

Regression testing is a critical part of software development and maintenance. It ensures that modifications to existing software do not break existing behavior and functionality. One of the key assumptions about regression tests is that their results are deterministic: when executed without any modifications with the same configuration, either they always fail or they always pass. In practice, however, there exist tests that are non-deterministic, called flaky tests. Flaky tests cause the results of test runs to be unreliable, and they disrupt the software development workflow. In this paper, we present a novel technique to automatically identify the locations of the root causes of flaky tests on the code level to help developers debug and fix them. We study the technique on flaky tests across 428 projects at Google. Based on our case studies, the technique helps identify the location of the root causes of flakiness with 82% accuracy. Furthermore, our studies show that integration into the appropriate developer workflows, simplicity of debugging aides and fully automated fixes are crucial and preferred components for adoption and usability of flakiness debugging and fixing tools.

• Software Engineering