MuRS: Suppressing and Ranking Mutants with IdentifierTemplates

Diff-based mutation testing is a mutation testing approach thatonly generates mutants in changed lines. At Google we executemore than 150 000 000 tests and submit more than40 000 commits per day. We have successfully integrated mutation testing into ourcode review process. Over the years, we have continuously gath-ered developer feedback on the surfaced mutants and measuredthe negative feedback rate. To enhance the developer experience,we manually implemented a large number of static rules, whichare used to suppress certain mutants. In this paper, we proposeMuRS, an automatic tool that finds patterns in the source codeunder test, and uses these patterns to rank and suppress futuremutants based on historical performance of similar mutants. Because MuRS learns mutant suppression rules fully automatically,it significantly reduces the build and maintenance cost of the mu-tation testing system. To evaluate the effectiveness of MuRS, we conducted an A/B experiment, where mutants in the experimentgroup were ranked and suppressed byMuRS, and mutants in thecontrol group were randomly shuffled. The experiment showeda statistically significant negative feedback rate of 11.45% in the experiment group versus 12.41% in the control group. Furthermore,we found that statement removal mutants received both most positive and negative developer feedback, suggesting a need for furtherinvestigation to identify valuable statement removal mutants.

• Software Engineering