Adam Brown
Authored Publications
Sort By
Understanding and Designing for Trust in AI Powered Developer Tooling
Ugam Kumar
Quinn Madison
IEEE Software (2024)
Preview abstract
Trust is central to how developers engage with AI. In this article, we discuss what we learned from developers about their level of trust in AI enhanced developer tooling, and how we translated those findings into product design recommendations to support customization, and the challenges we encountered along the way.
View details
Preview abstract
It’s no secret that generative artificial intelligence (GenAI) is rapidly changing the landscape of software development, with discussions about best practices for applying this transformative technology dominating the popular press [cite cite cite]. Perhaps nowhere on Earth have these discussions been more frequent and passionate than inside the organizations dedicated to making GenAI accessible and useful to developers, including at Google. During one such discussion between researchers on our DevOps Research and Assessment (DORA) and Engineering Productivity Research (EPR) teams, we were struck by a recurring finding common to development professionals both inside and outside of Google:
Using GenAI makes developers feel more productive, and developers who trust GenAI use it more.
On the surface, this finding may seem somewhat… obvious. But, for us, it highlighted the deep need to better understand the factors that impact developers’ trust in GenAI systems and ways to foster that trust, so that developers and development firms can yield the most benefit from their investment in GenAI development tools.
Here, we share findings from seven studies conducted at Google, regarding the productivity gains of GenAI use in development, the impacts of developers’ trust on GenAI use, and the factors we’ve observed which positively impact developers’ trust in GenAI. We conclude with five suggested strategies that organizations engaged in software development might employ to foster their developers’ trust in GenAI, thereby increasing their GenAI use and maximizing GenAI-related productivity gains.
View details
Preview abstract
AI-powered software development tooling is changing the way that developers interact with tools and write code. However, the ability for AI to truly transform software development depends on developers' level of trust in the tools. In this work, we take a mixed methods approach to measuring the factors that influence developers' trust in AI-powered code completion. We identified that familiarity with AI suggestions, quality of the suggestion, and level of expertise with the language all increased acceptance rate of AI-powered suggestions. While suggestion length and presence in a test file decreased acceptance rates. Based on these findings we propose recommendations for the design of AI-powered development tools to improve trust.
View details
Preview abstract
In this installment of Developer Productivity for Humans, we present two lines of research emphasizing the human experience in measuring developer productivity: the experience of flow or focus and the experience of friction during development.
View details
Using Logs Data to Identify When Software Engineers Experience Flow or Focused Work
Ben Holtz
ACM CHI Conference on Human Factors in Computing Systems (2023) (to appear)
Preview abstract
Beyond self-report data, we lack reliable and non-intrusive methods for identifying flow. However, taking a step back and acknowledging that flow occurs during periods of focus gives us the opportunity to make progress towards measuring flow by isolating focused work. Here, we take a mixed-methods approach to design a logs based metric that leverages machine learning and a comprehensive collection of logs data to identify periods of related actions (indicating focus), and validate this metric against self-reported time in focus or flow using diary data and quarterly survey data. Our results indicate that we can determine when software engineers at a large technology company experience focused work which includes instances of flow. This metric speaks to engineering work, but can be leveraged in other domains to non-disruptively measure when people experience focus. Future research can build upon this work to identify signals associated with other facets of flow.
View details
