AI in software engineering at Google: Progress and the path ahead

In 2019, a software engineer — at Google or indeed anywhere else — would have heard of advances in machine learning, and how deep learning has become remarkably effective in fields such as computer vision or language translation. However, most of them would not have imagined, let alone experienced, the ways in which machine learning might benefit what they do.

Just five years later, in 2024, there is widespread enthusiasm among software engineers about how AI is helping write code. And a significant number of those have used ML-based autocomplete, whether it is using company internal tools at large companies, e.g., Google’s internal code completion, or via commercially available products.

In this blog, we present our newest AI-powered improvements within the context of the continuing transformation of Google’s internal software development tools, and discuss further changes that we expect to see in the coming 5 years. We also present our methodology on how to build AI products that deliver value for professional software development. Our team is responsible for the software development environments where Google engineers spend the majority of their time, including inner loop (e.g., IDE, code review, code search), as well as outer loop surfaces (e.g., bug management, planning). We illustrate that improvements to these surfaces can directly impact developer productivity and satisfaction, both metrics that we monitor carefully.

We use our extensive and high quality logs of internal software engineering activities across multiple tools, which we have curated over many years. This data, for example, enables us to represent fine-grained code edits, build outcomes, edits to resolve build issues, code copy-paste actions, fixes of pasted code, code reviews, edits to fix reviewer issues, and change submissions to a repository. The training data is an aligned corpus of code with task-specific annotations in input as well as in output. The design of the data collection process, the shape of the training data, and the model that is trained on this data was described in our DIDACT blog. We continue to explore these powerful datasets with newer generations of foundation models available to us (discussed more below).

Our next significant deployments were resolving code review comments (>8% of which are now addressed with AI-based assistance) and automatically adapting pasted code to the surrounding context (now responsible for ~2% of code in the IDE2). Further deployments include instructing the IDE to perform code edits with natural language and predicting fixes to build failures. Other applications, e.g., predicting tips for code readability following a similar pattern are also possible.

Together, these deployed applications have been successful, highly-used applications at Google, with measurable impact on productivity in a real, industrial context.

What we’ve learned

Our work so far has taught us several things:

1. We achieved the highest impact with UX that naturally blends into users’ workflows. In all the above examples, a suggestion is presented to the user, taking them to the next step in their workflow with one tab or click. Experiments requiring the user to remember to trigger the feature have failed to scale.
2. We observe that with AI-based suggestions, the code author increasingly becomes a reviewer, and it is important to find a balance between the cost of review and added value. We typically address the tradeoff with acceptance rate targets.
3. Quick iterations with online A/B experiments are key, as offline metrics are often only rough proxies of user value. By surfacing our AI-based features on internal tooling, we benefit greatly from being able to easily launch and iterate, measure usage data, and ask users directly about their experience through UX research.
4. High quality data from activities of Google engineers across software tools, including interactions with our features, is essential for our model quality.

We observe across features that it’s important to optimize for conversion from the opportunity (mostly a user activity, shown at the top of the funnel below) to impact (applied AI assistance, at the bottom of the funnel), while removing bottlenecks from intermediate steps of the funnel by leveraging UX and model improvements.

What’s next

Encouraged by our successes so far, we are doubling down on bringing the latest foundation models (Gemini series) infused with the developer data (as part of DIDACT, mentioned above) to power existing and new applications of ML to software engineering in Google.

Across the industry, ML-based code completion has provided a major boost for software developers. While there are still opportunities to improve code generation, we expect the next wave of benefits to come from ML assistance in a broader range of software engineering activities, such as testing, code understanding and code maintenance; the latter being of particular interest in enterprise settings. These opportunities inform our own ongoing work. We also highlight two trends that we see in the industry:

1. Human-computer interaction has moved towards natural language as a common modality, and we are seeing a shift towards using language as the interface to software engineering tasks as well as the gateway to informational needs for software developers, all integrated in IDEs.
2. ML-based automation of larger-scale tasks — from diagnosis of an issue to landing a fix — has begun to show initial evidence of feasibility. These possibilities are driven by innovations in agents and tool use, which permit the building of systems that use one or more LLMs as a component to accomplish a larger task.

To expand on the above successes toward these next generation capabilities, the community of practitioners and researchers working in this topic would benefit from common benchmarks to help move the field towards practical engineering tasks. So far, benchmarks have been focused mostly around code generation (e.g., HumanEval). In an enterprise setting, however, benchmarks for a wider range of tasks could be particularly valuable, e.g., code migrations and production debugging. Some benchmarks, such as one for bug resolution (e.g., SWEBench), and prototypes targeting those benchmarks (e.g., from Cognition AI) have been published. We encourage the community to come together to suggest more benchmarks to span a wider range of software engineering tasks.