Multiarch Testing at Google
Abstract
With the end of Moore's law, the need for higher performance per watt, and the rise of domain specific chips Google is adopting new general purpose compute architectures. Google now routinely runs large scale software (Big Query, Spanner, PubSub, Blobstore, etc…) on both x86 and Arm CPUs. Previously most server software at Google was written for the x86-64 architecture. When adopting a new architecture the software needs to be verified for the new platform. In this talk we will discuss how we are changing our central continuous integration platforms to support multi-architecture testing.
Our primary thesis for our changes is that most bugs are not architecture specific. Most software runs well regardless of the platform. However, occasionally there are platform specific issues that need to be detected and fixed. If we naively run tests on every available platform our testing cost would scale linearly with the number of available platforms. This cost increase (double compute cost to introduce Arm) is too high and running every test twice is wasteful given that most bugs are not architecture dependent. Therefore, we must take a pragmatic approach and use historical data to guide our testing efforts.
In addition to the machine cost of running the tests, introducing a new architecture also introduces a human cost. Every team that uses CI at Google (virtually all software teams) has one or more "projects" that they have configured with a bespoke configuration format. As part of the format they specify which flags (options) they would like to pass to the build system (Bazel). Bazel then uses these high level flags to configure the individual build tools like compilers and linkers that convert the high level programs to machine code. With no changes our largest CI system, TAP, trivially supports Arm or any other architecture — just supply the relevant feature flags in each project's configuration file. However, TAP has hundreds of thousands of projects. Requiring manual reconfiguration and duplication of those projects to migrate to Arm would be incredibly costly in human time. To make sure introducing a new architecture scales "sublinearly" in human time as well we introduced an automated configuration method: "variants."
Variants automate CI to centralize build option configuration. This is accomplished using a structured, machine-understandable format, enabling CI systems to intelligently and cost-effectively schedule per-architecture testing. This presentation will focus on how this configuration automation facilitates intelligent, machine-learning-driven scheduling, creating a user experience where architectural diversity is transparent unless significant failures occur.
Our primary thesis for our changes is that most bugs are not architecture specific. Most software runs well regardless of the platform. However, occasionally there are platform specific issues that need to be detected and fixed. If we naively run tests on every available platform our testing cost would scale linearly with the number of available platforms. This cost increase (double compute cost to introduce Arm) is too high and running every test twice is wasteful given that most bugs are not architecture dependent. Therefore, we must take a pragmatic approach and use historical data to guide our testing efforts.
In addition to the machine cost of running the tests, introducing a new architecture also introduces a human cost. Every team that uses CI at Google (virtually all software teams) has one or more "projects" that they have configured with a bespoke configuration format. As part of the format they specify which flags (options) they would like to pass to the build system (Bazel). Bazel then uses these high level flags to configure the individual build tools like compilers and linkers that convert the high level programs to machine code. With no changes our largest CI system, TAP, trivially supports Arm or any other architecture — just supply the relevant feature flags in each project's configuration file. However, TAP has hundreds of thousands of projects. Requiring manual reconfiguration and duplication of those projects to migrate to Arm would be incredibly costly in human time. To make sure introducing a new architecture scales "sublinearly" in human time as well we introduced an automated configuration method: "variants."
Variants automate CI to centralize build option configuration. This is accomplished using a structured, machine-understandable format, enabling CI systems to intelligently and cost-effectively schedule per-architecture testing. This presentation will focus on how this configuration automation facilitates intelligent, machine-learning-driven scheduling, creating a user experience where architectural diversity is transparent unless significant failures occur.
