Productive Coverage: Improving the Actionability of Code Coverage

Code coverage is an intuitive and established test adequacy measure. However, not all parts of the code base are equally important, and hence additional testing may be critical for some uncovered code, whereas it may not be worthwhile for other uncovered code. As a result, simply visualizing uncovered code is not reliably actionable. To make code coverage actionable and further improve code coverage in our codebase, we developed Productive Coverage — a novel approach to code coverage that guides developers to uncovered code that that should be tested by (unit) tests. Specifically, Productive Coverage identifies uncovered code that is similar to existing code, which in turn is tested and/or frequently executed in production. We implemented and evaluated Productive Coverage for four programming languages (C++, Java, Go, and Python). The evaluation shows: (1) The developer sentiment, measured at the point of use, is strongly positive; (2) Productive Coverage meaningfully increases code coverage above a strong baseline; (3) Productive Coverage has no negative effect on code authoring efficiency; (4) Productive Coverage modestly improves code-review effiency; (5) Productive Coverage directly improves code quality and prevents bugs from being introduced, in addition to improving test quality