Sushmita Azad
Sushmita Azad is a software engineer at Google DeepMind working on training infrastructure reliability for Gemini. She previously led Google’s Test Automation Platform (TAP), working on predictive modeling for continuous integration pipelines and multi-architecture testing. Sushmita received her MS in Computer Science from the University of Illinois, Urbana-Champaign in 2020, where she applied NLP and ML techniques to computer science education.
Research Areas
Authored Publications
Sort By
Taming the Variants Multi-Architecture Continuous Testing at Google
Chandrakanth Chittappa
Ali Esmaeeli
Laura Macaddino
Sam Manfreda
David Margolin
Dharma Naidu
Sabuj Pattanayek
Sachin Sable
Ruslan Sakevych
Dushyant Acharya
Adrian Berding
Kevin Crossan
Wolff Dobson
Abhay Singh
19th IEEE International Conference on Software Testing, Verification and Validation (ICST) 2026, Daejeon, Republic of Korea, IEEE
Preview abstract
Enterprises are increasingly adopting multiple general-purpose computer architectures in the data center. This leads to new testing challenges as it creates demand to qualify the software for the additional architectures. Naively double-testing all software for both architectures is costly and unnecessary. Further, reconfiguring CI/CD to take advantage of the new architecture can be non-trivial at scale. This paper introduces CI/CD variants and an optimized testing cycle to solve these twin challenges. We empirically evaluate our solution's impact on human and machine expenses using 44k projects at Google on real production data. First, we estimate saving ~25% of machine expenses at the negligible cost of a few delayed breakage detections per day. Second, we estimate a 90+% reduction in human cost for migrating the configuration. All features described in this paper are now Generally Available at Google and we report this as an empirical case study in scaling CI/CD to new architectures.
View details
Preview abstract
With the end of Moore's law, the need for higher performance per watt, and the rise of domain specific chips Google is adopting new general purpose compute architectures. Google now routinely runs large scale software (Big Query, Spanner, PubSub, Blobstore, etc…) on both x86 and Arm CPUs. Previously most server software at Google was written for the x86-64 architecture. When adopting a new architecture the software needs to be verified for the new platform. In this talk we will discuss how we are changing our central continuous integration platforms to support multi-architecture testing.
Our primary thesis for our changes is that most bugs are not architecture specific. Most software runs well regardless of the platform. However, occasionally there are platform specific issues that need to be detected and fixed. If we naively run tests on every available platform our testing cost would scale linearly with the number of available platforms. This cost increase (double compute cost to introduce Arm) is too high and running every test twice is wasteful given that most bugs are not architecture dependent. Therefore, we must take a pragmatic approach and use historical data to guide our testing efforts.
In addition to the machine cost of running the tests, introducing a new architecture also introduces a human cost. Every team that uses CI at Google (virtually all software teams) has one or more "projects" that they have configured with a bespoke configuration format. As part of the format they specify which flags (options) they would like to pass to the build system (Bazel). Bazel then uses these high level flags to configure the individual build tools like compilers and linkers that convert the high level programs to machine code. With no changes our largest CI system, TAP, trivially supports Arm or any other architecture — just supply the relevant feature flags in each project's configuration file. However, TAP has hundreds of thousands of projects. Requiring manual reconfiguration and duplication of those projects to migrate to Arm would be incredibly costly in human time. To make sure introducing a new architecture scales "sublinearly" in human time as well we introduced an automated configuration method: "variants."
Variants automate CI to centralize build option configuration. This is accomplished using a structured, machine-understandable format, enabling CI systems to intelligently and cost-effectively schedule per-architecture testing. This presentation will focus on how this configuration automation facilitates intelligent, machine-learning-driven scheduling, creating a user experience where architectural diversity is transparent unless significant failures occur.
View details
Preview abstract
Google's approach to testing includes both testing prior to code submission (for fast validation) and after code submission (for comprehensive validation). However, Google's ever growing testing demand has lead to increased continuous integration cycle latency and machine costs. When the post code submission continuous integration cycles get longer it delays detecting breakages in the main repository which increases developer friction and lowers productivity. To mitigate this without increasing resource demand, Google is implementing Postsubmit Speculative Cycles in their Test Automation Platform (TAP). Speculative Cycles prioritize finding novel breakages faster. In this paper we present our new test scheduling architecture and the machine learning system (Transition Prediction) driving it. Both the ML system and the end-to-end test scheduling system are empirically evaluated on 3-months of our production data (120 billion test X cycle pairs, 7.7 million breaking targets, with ~20 thousand unique breakages). Using Speculative Cycles we observed a median (p50) reduction of approximately 70 minutes in the time taken to detect novel breaking targets.
View details
Preview abstract
Google's CI/CD system is undergoing a major structural change. Previous systems of stage-by-stage and thorough testing are inadequate to scale to the demands of increasing numbers of tests and rates of code submission. This presentation covers how Google CI&R will evolve to handle these issues by shifting to a smarter probabilistic paradigm that will enable decreasing load and latency while maintaining testing signal.
View details
SafeRevert: When Can Breaking Changes be Automatically Reverted?
Eric Nickell
2024 IEEE Conference on Software Testing, Verification and Validation (ICST), IEEE, Toronto, ON, Canada
Preview abstract
When bugs or defects are introduced into a large scale software repository, they reduce productivity. Programmers working on related areas of the code will encounter test failures, compile breakages, or other anomalous behavior. On encounter- ing these issues, they will need to troubleshoot and determine that their changes were not the cause of the error and that another change is a fault. They must then find that change and revert it to return the repository to a healthy state. In the past, our group has identified ways to identify the root cause (or culprit) change that introduced a test failure even when the test is flaky. This paper focuses on a related issue: At what point does the Continuous Integration system have enough evidence to support automatically reverting a change? We will motivate the problem, provide several methods to address it, and empirically evaluate our solution on a large set (34,000) real world breaking changes that occurred at Google.
View details
