Andrew Macvean
Andrew has been at Google since July 2014, conducting research into developer experience. He is interested in the usability of web APIs, Client Libraries, and developer documentation.
Research Areas
Authored Publications
Google Publications
Other Publications
Sort By
Preview abstract
The evolution of AI is a pivotal moment in history, but it’s not the first time we have experienced technological advances that have changed how humans work. By looking at the advances in automobiles, we are reminded of the importance of focusing on our developers' needs and goals.
View details
API Governance at Scale
Mak Ahmad
JJ Geewax
David R Karger
Kwan-Liu Ma
ICSE 2024 Software Engineering in Practice(2024)
Preview abstract
API Governance, the process of applying standardized sets of policies and guardrails to the design and development of APIs, has only grown in importance and prominence given the continued growth in APIs being produced. In this paper, we present an Action Research style approach to investigate and understand the utility of a multi-faceted API Governance process being adopted inside Google. We first reflect on past research around API Governance, and then introduce three new components, 1. API Improvement Proposals (AIPs) the documented source of truth for API design rules, 2. API Linter, an automated analysis tool which checks for adherence to / violations of AIPs, and 3. API Readability, a program to educate and certify API design experts. These three components are designed to build upon pre-existing processes to scale and improve API design. Through a mixed-methods research strategy, containing both a survey and a series of interviews, we evaluate the utility of these approaches in supporting API Producers. Our research shows that API Producers have positive sentiment towards API Governance, validating the general direction of the program. Specifically, our study participants highlighted the positive impact of API Governance on the quality of the APIs they produced, via consistency in both the outcome and approach. This paper also discusses future research opportunities to enhance API Governance, specifically with regards to newer API Producers, who reported worse sentiment towards the program than their more experienced peers.
View details
Meta-Manager: A Tool for Collecting and Exploring Meta Information about Code
Amber Horvath
Brad A. Myers
CHI '24: Proceedings of the CHI Conference on Human Factors in Computing Systems(2024)
Preview abstract
Modern software engineering is in a state of flux. With more development utilizing AI code generation tools and the continued reliance on online programming resources, understanding code and the original intent behind it is becoming more important than it ever has been. To this end, we have developed the “Meta-Manager”, a Visual Studio Code extension, with a supplementary browser extension, that automatically collects and organizes changes made to code while keeping track of the provenance of each part of the code, including code that has been copy-pasted from popular programming resources online. These sources and subsequent changes are represented in the editor and may be explored using searching and filtering mechanisms to help developers answer historically hard-to-answer questions about code, its provenance, and its design rationale. In our evaluation of Meta-Manager, we found developers
were successfully able to use it to answer otherwise unanswerable questions about an unfamiliar code base.
View details
Preview abstract
Almost no modern software system is written from scratch, and developers are required to effectively learn to use third-party libraries and software services. Thus, many practitioners and researchers have looked for ways to create effective documentation that supports developers’ learning. However, few efforts have focused on how people actually use the documentation. In this paper, we report on an exploratory, multi-phase, mixed methods empirical study of documentation page-view logs from four cloud-based industrial services. By analyzing page-view logs for over 100,000 users, we find diverse patterns of documentation page visits. Moreover, we show statistically that which documentation pages people visit often correlates with user characteristics such as past experience with the specific product, on the one hand, and with future adoption of the API on the other hand. We discuss the implications of these results on documentation design and propose documentation page-view log analysis as a feasible technique for design audits of documentation, from ones written for software developers to ones designed to support end users (e.g., Adobe Photoshop).
View details
Using an LLM to Help With Code Understanding
Daye Nam
Vincent Hellendoorn
Bogdan Vasilescu
Brad A. Myers
ICSE '24: Proceedings of the IEEE/ACM 46th International Conference on Software Engineering(2024)
Preview abstract
Understanding code is challenging, especially when working in new and complex development environments. Code comments and documentation can help, but are typically scarce or hard to navigate. Large language models (LLMs) are revolutionizing the process of writing code. Can they do the same for helping understand it? In this study, we provide a first investigation of an LLM-based conversational UI built directly in the IDE that is geared towards code understanding. Our IDE plugin queries OpenAI's GPT-3.5-turbo model with four high-level requests without the user having to write explicit prompts: to explain a highlighted section of code, provide details of API calls used in the code, explain key domain-specific terms, and provide usage examples for an API. The plugin also allows for open-ended prompts, which are automatically contextualized to the LLM with the program being edited. We evaluate this system in a user study with 32 participants, which confirms that using our plugin can aid task completion more than web search. We additionally provide a thorough analysis of the ways developers use, and perceive the usefulness of, our system, among others finding that the usage and benefits differ between students and professionals. We conclude that in-IDE prompt-less interaction with LLMs is a promising future direction for tool builders.
View details
AI-Enhanced API Design: A New Paradigm in Usability and Efficiency
Mak Ahmad
David R Karger
Kwan-Liu Ma
CHI EA '24: Extended Abstracts of the 2024 CHI Conference on Human Factors in Computing Systems(2024)
Preview abstract
This study uses mixed methods to evaluate API design methods, focusing on design and consumption phases. Our goal was to understand the impact of API governance approaches on productivity and usability. A controlled developer experiment (n=34) demonstrated
a 10% increased requirement fulfillment using API Improvement Proposals (AIPs) and linter versus no protocols. Meanwhile, 73% of 33 surveyed API consumers preferred AIP-aligned designs for enhanced usability and comprehensibility. Complementing this, a
custom large language model called the API Architect received average expert ratings of just 5/10 for specification quality, revealing gaps versus manual design. The quantitative performance metrics combined with qualitative user feedback provide evidence from
multiple angles that strategically integrating industry best practices with maturing AI capabilities can meaningfully improve API design outcomes. This research offers empirical insights from developer and consumer perspectives to advance scholarly discourse
and industry practice regarding optimal API design workflows.
View details
Preview abstract
ML enhanced software development tooling is changing the way software engineers develop code. While the development of these tools continues to rise, studies have primarily focused on the accuracy and performance of underlying models, rather than the user experience. Understanding how engineers interact with ML enhanced tooling can help us define what successful interactions with ML based assistance look like. We therefore build upon prior research, by comparing software engineers' perceptions of two types of ML enhanced tools, (1) code completion and (2) code example suggestions. We then use our findings to inform design guidance for ML enhanced software development tooling. This research is intended to spark a growing conversation about the future of ML in software development and guide the design of developer tooling.
View details
Understanding How Programmers Can Use Annotations on Documentation
Amber Horvath
Michael Xieyang Liu
River Hendriksen
Connor Shannon
Emma Paterson
Kazi Jawad
Brad A. Myers
In CHI Conference on Human Factors in Computing Systems (CHI '22), April 29-May 5, 2022, New Orleans, LA, USA. ACM, New York, NY, USA 16 Pages(2022)
Preview abstract
Modern software development requires developers to find and effectively utilize new APIs and their documentation, but documentation has many well-known issues. Despite this, developers eventually overcome these issues but have no way of sharing what they learned. We investigate sharing this documentation-specific information through annotations, which have advantages over developer forums as the information is contextualized, not disruptive, and is short, thus easy to author. Developers can also author annotations to support their own comprehension. In order to support the documentation usage behaviors we found, we built the Adamite annotation tool, which provides features such as multiple anchors, annotation types, and pinning. In our user study, we found that developers are able to create annotations that are useful to themselves and are able to utilize annotations created by other developers when learning a new API, with readers of the annotations completing 67% more of the task, on average, than the baseline.
View details
Accessibility of Command Line Interfaces
CHI Conference on Human Factors in Computing Systems(2021)
Preview abstract
Command line interfaces (CLIs) remain a popular tool among developers and system administrators. Since CLIs are text based interfaces, they are sometimes considered accessible alternatives to predominantly visual developer tools like IDEs. However, there is no systematic evaluation of accessibility of CLIs in the literature. In this paper, we describe two studies with 12 developers on their experience of using CLIs with screen readers. Our findings show that CLIs have their own set of accessibility issues - the most important being CLIs are unstructured text interfaces. Based on our findings, we provide a set of recommendations for improving accessibility of command line interfaces.
View details
MARBLE: Mining for Boilerplate Code to Identify API Usability Problems
Daye Nam
Amber Horvath
Bogdan Vasilescu
Brad A. Myers
Proceedings 34th IEEE/ACM International Conference on Automated Software Engineering(2019)
Preview abstract
Designing usable APIs is critical to developers’ productivity and software quality, but is quite difficult. One of the challenges is that anticipating API usability barriers and real world usage is difficult, due to a lack of automated approaches to mine usability data at scale. In this paper, we focus on one particular grievance that developers repeatedly express in online discussions about APIs: “boilerplate code.” We investigate what properties make code count as boilerplate, the reasons for boilerplate, and how programmers can reduce the need for it. We then present MARBLE, a novel approach to automatically mine boilerplate code candidates from API client code repositories. MARBLE adapts existing techniques, including an API usage mining algorithm, an AST comparison algorithm, and a graph partitioning algorithm. We evaluate MARBLE with 13 Java APIs, and show that our approach successfully identifies both already known and new API-related boilerplate code instances.
View details
API Designers in the Field: Design Practices and Challenges for Creating Usable APIs
Lauren Murphy
Mary Beth Kery
Tosin Alliyu
Brad A. Myers
2018 IEEE Symposium on Visual Languages and Human-Centric Computing (VL/HCC'18)
Preview abstract
Application Programming Interfaces (APIs) are a rapidly growing industry and the usability of the APIs is crucial to programmer productivity. Although prior research has shown that APIs commonly suffer from significant usability problems, little attention has been given to studying how APIs are designed and created in the first place. We interviewed 24 professionals involved with API design from 7 major companies to identify their training and design processes. Interviewees had insights into many different aspects of designing for API usability and areas of significant struggle. Even though use cases are central to the design of APIs, during the design phase it is challenging to discern which potential use cases of the API users will value most. After an API is released, designers lack tools to gather any feedback from this data in aggregate even as developers openly discuss the API online.
View details
Discovering API Usability Problems at Scale
Emerson Murphy-Hill
Andrew Head
International Workshop on API Usage and Evolution (WAPI)(2018)
Preview abstract
Software developers’ productivity can be negatively impacted by using APIs incorrectly. In this paper, we describe an analysis technique we designed to find API usability problems by comparing successive file-level changes made by individual software developers. We applied our tool, StopMotion, to the file histories of real developers doing real tasks at Google. The results reveal several API usability challenges including simple typos, conceptual API misalignments, and conflation of similar APIs.
View details
Preliminary Analysis of REST API Style Guidelines
Lauren Murphy
Tosin Alliyu
Mary Beth Kery
Brad A. Myers
8th Workshop on Evaluation and Usability of Programming Languages and Tools (PLATEAU'2017) at SPLASH 2017(2017)
Preview abstract
We studied a collection of 32 publicly published guidelines for designing RESTful Application Programming Interfaces (APIs), each from a different company, to identify similarities and differences to see if there are overall best practices across ten different topics. Our contribution includes providing a list of topics that API authors can reference when creating or evaluating their own guidelines. Additionally, we found that while some guidelines attempt to enforce consistency, simplicity, and intuitiveness in the APIs that use these guidelines, cross-guideline comparisons show a lack of consistency in some of the topics examined, and different interpretations of what is thought to be “simple” and “intuitive.”
View details
The Uses of Interactive Explorers for Web APIs
Luke Church
8th Workshop on Evaluation and Usability of Programming Languages and Tools (PLATEAU'2017) at SPLASH 2017, ACM
Preview abstract
Interactive method invocation has become a common interaction pattern in the documentation of web application programming interfaces (APIs). One of the earliest examples of this pattern being applied at scale is the Google APIs Explorer. In this paper, we describe eight ways developers use such tools in software development, grounded on empirical analyses of the Google APIs Explorer. We then explain the utility of these tools by tying the use cases back to extant literature on programming.
View details
API Usability at Scale
Luke Church
Craig Citro
Proceedings of the 26th annual workshop of the Psychology of Programming Interest Group(2016)
Preview abstract
Designing and maintaining useful and usable APIs remains challenging. At Google we manage hundreds of APIs. In this article we report on the experience of doing so and describe six on-going challenges: resource allocation, empirically-grounded guidelines, communicating issues, supporting API evolution over time, usable auth, and usable client libraries at scale.
View details
API Design Reviews at Scale
Martin Maly
CHI EA '16 Proceedings of the 2016 CHI Conference Extended Abstracts on Human Factors in Computing Systems, ACM New York, NY, USA, pp. 849-858
Preview abstract
The number of APIs produced by Google's various business units grew at an astounding rate over the last decade, the result of which was a user experience containing wild inconsistencies and usability problems. There was no single issue that dominated the usability problems; rather, users suffered a death from a thousand papercuts. A lightweight, scalable, distributed design review process was put into place that has improved our APIs and the efficacy of our many API designers. Challenges remain, but the API design reviews at scale program has started successfully.
View details
No Results Found
