Avi Kondareddy
Avi works on Google's main CI system TAP on regression test selection and other CI-related software engineering problems.
Research Areas
Authored Publications
Sort By
Taming the Variants Multi-Architecture Continuous Testing at Google
Chandrakanth Chittappa
Ali Esmaeeli
Laura Macaddino
Sam Manfreda
David Margolin
Dharma Naidu
Sabuj Pattanayek
Sachin Sable
Ruslan Sakevych
Dushyant Acharya
Adrian Berding
Kevin Crossan
Wolff Dobson
Abhay Singh
19th IEEE International Conference on Software Testing, Verification and Validation (ICST) 2026, Daejeon, Republic of Korea, IEEE
Preview abstract
Enterprises are increasingly adopting multiple general-purpose computer architectures in the data center. This leads to new testing challenges as it creates demand to qualify the software for the additional architectures. Naively double-testing all software for both architectures is costly and unnecessary. Further, reconfiguring CI/CD to take advantage of the new architecture can be non-trivial at scale. This paper introduces CI/CD variants and an optimized testing cycle to solve these twin challenges. We empirically evaluate our solution's impact on human and machine expenses using 44k projects at Google on real production data. First, we estimate saving ~25% of machine expenses at the negligible cost of a few delayed breakage detections per day. Second, we estimate a 90+% reduction in human cost for migrating the configuration. All features described in this paper are now Generally Available at Google and we report this as an empirical case study in scaling CI/CD to new architectures.
View details
Preview abstract
With the end of Moore's law, the need for higher performance per watt, and the rise of domain specific chips Google is adopting new general purpose compute architectures. Google now routinely runs large scale software (Big Query, Spanner, PubSub, Blobstore, etc…) on both x86 and Arm CPUs. Previously most server software at Google was written for the x86-64 architecture. When adopting a new architecture the software needs to be verified for the new platform. In this talk we will discuss how we are changing our central continuous integration platforms to support multi-architecture testing.
Our primary thesis for our changes is that most bugs are not architecture specific. Most software runs well regardless of the platform. However, occasionally there are platform specific issues that need to be detected and fixed. If we naively run tests on every available platform our testing cost would scale linearly with the number of available platforms. This cost increase (double compute cost to introduce Arm) is too high and running every test twice is wasteful given that most bugs are not architecture dependent. Therefore, we must take a pragmatic approach and use historical data to guide our testing efforts.
In addition to the machine cost of running the tests, introducing a new architecture also introduces a human cost. Every team that uses CI at Google (virtually all software teams) has one or more "projects" that they have configured with a bespoke configuration format. As part of the format they specify which flags (options) they would like to pass to the build system (Bazel). Bazel then uses these high level flags to configure the individual build tools like compilers and linkers that convert the high level programs to machine code. With no changes our largest CI system, TAP, trivially supports Arm or any other architecture — just supply the relevant feature flags in each project's configuration file. However, TAP has hundreds of thousands of projects. Requiring manual reconfiguration and duplication of those projects to migrate to Arm would be incredibly costly in human time. To make sure introducing a new architecture scales "sublinearly" in human time as well we introduced an automated configuration method: "variants."
Variants automate CI to centralize build option configuration. This is accomplished using a structured, machine-understandable format, enabling CI systems to intelligently and cost-effectively schedule per-architecture testing. This presentation will focus on how this configuration automation facilitates intelligent, machine-learning-driven scheduling, creating a user experience where architectural diversity is transparent unless significant failures occur.
View details
Preview abstract
Google's approach to testing includes both testing prior to code submission (for fast validation) and after code submission (for comprehensive validation). However, Google's ever growing testing demand has lead to increased continuous integration cycle latency and machine costs. When the post code submission continuous integration cycles get longer it delays detecting breakages in the main repository which increases developer friction and lowers productivity. To mitigate this without increasing resource demand, Google is implementing Postsubmit Speculative Cycles in their Test Automation Platform (TAP). Speculative Cycles prioritize finding novel breakages faster. In this paper we present our new test scheduling architecture and the machine learning system (Transition Prediction) driving it. Both the ML system and the end-to-end test scheduling system are empirically evaluated on 3-months of our production data (120 billion test X cycle pairs, 7.7 million breaking targets, with ~20 thousand unique breakages). Using Speculative Cycles we observed a median (p50) reduction of approximately 70 minutes in the time taken to detect novel breaking targets.
View details
Preview abstract
Google's CI/CD system is undergoing a major structural change. Previous systems of stage-by-stage and thorough testing are inadequate to scale to the demands of increasing numbers of tests and rates of code submission. This presentation covers how Google CI&R will evolve to handle these issues by shifting to a smarter probabilistic paradigm that will enable decreasing load and latency while maintaining testing signal.
View details
SafeRevert: When Can Breaking Changes be Automatically Reverted?
Eric Nickell
2024 IEEE Conference on Software Testing, Verification and Validation (ICST), IEEE, Toronto, ON, Canada
Preview abstract
When bugs or defects are introduced into a large scale software repository, they reduce productivity. Programmers working on related areas of the code will encounter test failures, compile breakages, or other anomalous behavior. On encounter- ing these issues, they will need to troubleshoot and determine that their changes were not the cause of the error and that another change is a fault. They must then find that change and revert it to return the repository to a healthy state. In the past, our group has identified ways to identify the root cause (or culprit) change that introduced a test failure even when the test is flaky. This paper focuses on a related issue: At what point does the Continuous Integration system have enough evidence to support automatically reverting a change? We will motivate the problem, provide several methods to address it, and empirically evaluate our solution on a large set (34,000) real world breaking changes that occurred at Google.
View details
Flake Aware Culprit Finding
Eric Nickell
Collin Johnston
Proceedings of the 16th IEEE International Conference on Software Testing, Verification and Validation (ICST 2023), IEEE (to appear)
Preview abstract
When a change introduces a bug into a large software repository, there is
often a delay between when the change is committed and when bug is detected.
This is true even when the bug causes an existing test to fail! These delays
are caused by resource constraints which prevent the organization from running
all of the tests on every change. Due to the delay, a Continuous Integration
system needs to locate buggy commits. Locating them is complicated by flaky
tests that pass and fail non-deterministically. The flaky tests introduce
noise into the CI system requiring costly reruns to determine if a failure was
caused by a bad code change or caused by non-deterministic test behavior. This
paper presents an algorithm, Flake Aware Culprit Finding, that locates
buggy commits more accurately than a traditional bisection search. The
algorithm is based on Bayesian inference and noisy binary search, utilizing
prior information about which changes are most likely to contain the bug. A
large scale empirical study was conducted at Google on 13,000+ test breakages.
The study evaluates the accuracy and cost of the new algorithm versus a
traditional deflaked bisection search.
View details
Preview abstract
In an ideal Continuous Integration (CI) workflow, all potentially impacted builds/tests would be run before submission for every proposed change. In large-scale environments like Google’s mono-repository, this is not feasible to do in terms of both latency and compute cost, given the frequency of change requests and overall size of the codebase. Instead, the compromise is to run more comprehensive testing at later stages of the development lifecycle – batched after submission. These runs detect newly broken tests. Automated culprit finders determine which change broke the tests, the culprit change.
In order to make testing at Google more efficient and lower latency, TAP Postsubmit is developing a new scheduling algorithm that utilizes Bug Prediction metrics, features from the change, and historical information about the tests and targets to predict and rank targets by their likelihood to fail. This presentation examines the association between some of our selected features with culprits in Google3.
View details
