Google Research

Taming Google-Scale Continuous Testing

  • Atif Memon
  • Bao Nguyen
  • Eric Nickell
  • John Micco
  • Sanjeev Dhanda
  • Rob Siemborski
  • Zebao Gao
ICSE '17:Proceedings of the 39th International Conference on Software Engineering (2017) (to appear)

Abstract

Growth in Google’s code size and feature churn rate has seen increased reliance on continuous integration (CI) and testing to maintain quality. Even with enormous resources dedicated to testing, we are unable to regression test each code change individually, resulting in increased lag time between code check-ins and test result feedback to developers. We report results of a project that aims to reduce this time by: (1) controlling test workload without compromising quality, and (2) distilling test results data to inform developers, while they write code, of the impact of their latest changes on quality. We model, empirically understand, and leverage the correlations that exist between our code, test cases, developers, programming languages, and code-change and test-execution frequencies, to improve our CI and development processes. Our findings show: very few of our tests ever fail, but those that do are generally “closer” to the code they test; certain frequently modified code and certain users/tools cause more breakages; and code recently modified by multiple developers (more than 3) breaks more often.

NOTE: You can find the anonymized dataset for our paper on Google drive: https://drive.google.com/open?id=0B5_QHWCtac81VGNKYnhrQkJBZGM

Research Areas

Learn more about how we do research

We maintain a portfolio of research projects, providing individuals and teams the freedom to emphasize specific types of work