Chris Welty
Dr. Chris Welty is a Research Scientist at Google in New York. He currently works on Gemini, and has made contributions to the human evaluation of Gemini models, as well as improvements to multi-turn interactions (e.g. dialogs) for Gemini's core models, AI mode, and native speech models.
He is best known for work that focuses on understanding the continuous nature of truth in the presence of a diversity of perspectives, and how that creates bias in AI models, and impacts our ability to measure confidence and risk. He contributes to the CrowdTruth github as well as the p-value simulator for exploring models of variance.
Before Gemini, he spent several years working with the google maps team to better understand user contributions that often disagree, and his first project at Google was launched as Explore in Google Docs (which has long since been replaced by Gemini).
He is active in several research communities, notably Crowdsourcing and Human Computation community and Neurips.
Before Google, Dr. Welty was a member of the technical leadership team for IBM's Watson - the question answering computer that defeated the all-time best Jeopardy! champions in a widely televised contest. He appeared on the broadcast, discussing the technology behind Watson, as well as many articles in the popular and scientific press. His proudest moment was being interviewed for StarTrek.com about the project. He is a recipient of the AAAI Feigenbaum Prize for his work.
In the distant past, Welty played a seminal role in the development of the Semantic Web and Ontologies, and co-developed OntoClean, the first formal methodology for evaluating ontologies. He published many papers before those shown below, see his Google Scholar entry.
Research Areas
Authored Publications
Sort By
ReviewCoin: Paying for Real Work
arXiv, Google Research (2025)
Preview abstract
The peer-review process is broken and the problem is getting worse, especially in AI: large conferences like NeurIPS increasingly struggle to adequately review huge numbers of paper submissions. I propose a scalable solution that, foremost, recognizes reviewing as important, necessary, \emph{work} and rewards it with crypto-coins owned and managed by the conferences themselves. The idea is at its core quite simple: paper submissions require work (reviews, meta-reviews, etc.) to be done, and therefore the submitter must pay for that work. Each reviewer submits their review to be approved by some designated conference officer (e.g. PC chair, Area Chair, etc.), and upon approval is paid a single coin for a single review. If three reviews are required, the cost of submission should be three coins + a tax that covers payments to all the volunteers who organize the conference. After some one-time startup costs to fairly distribute coins, the process should be relatively stable with new coins minted only when a conference grows.
View details
Preview abstract
This paper introduces v0.5 of the AI Safety Benchmark, which has been created by the MLCommons AI Safety Working Group. The AI Safety Benchmark has been designed to assess the safety risks of AI systems that use chat-tuned language models. We introduce a principled approach to specifying and constructing the benchmark, which for v0.5 covers only a single use case (an adult chatting to a general-purpose assistant in English), and a limited set of personas (i.e., typical users, malicious users, and vulnerable users). We created a new taxonomy of 13 hazard categories, of which 7 have tests in the v0.5 benchmark. We plan to release version 1.0 of the AI Safety Benchmark by the end of 2024. The v1.0 benchmark will provide meaningful insights into the safety of AI systems. However, the v0.5 benchmark should not be used to assess the safety of AI systems. We have sought to fully document the limitations, flaws, and challenges of v0.5. This release of v0.5 of the AI Safety Benchmark includes (1) a principled approach to specifying and constructing the benchmark, which comprises use cases, types of systems under test (SUTs), language and context, personas, tests, and test items; (2) a taxonomy of 13 hazard categories with definitions and subcategories; (3) tests for seven of the hazard categories, each comprising a unique set of test items, i.e., prompts. There are 43,090 test items in total, which we created with templates; (4) a grading system for AI systems against the benchmark; (5) an openly available platform, and downloadable tool, called ModelBench that can be used to evaluate the safety of AI systems on the benchmark; (6) an example evaluation report which benchmarks the performance of over a dozen openly available chat-tuned language models; (7) a test specification for the benchmark.
View details
Preview abstract
We tackle the problem of providing accurate, rigorous p-values
for comparisons between the results of two evaluated systems
whose evaluations are based on a crowdsourced “gold” reference
standard. While this problem has been studied before, we argue
that the null hypotheses used in previous work have been based
on a common fallacy of equality of probabilities, as opposed to the
standard null hypothesis that two sets are drawn from the same
distribution. We propose using the standard null hypothesis, that
two systems’ responses are drawn from the same distribution, and
introduce a simulation-based framework for determining the true
p-value for this null hypothesis. We explore how to estimate the
true p-value from a single test set under different metrics, tests,
and sampling methods, and call particular attention to the role of
response variance, which exists in crowdsourced annotations as a
product of genuine disagreement, and in system predictions as a
product of stochastic training regimes, or in generative models as
an expected property of the outputs. We find that response variance
is a powerful tool for estimating p-values, and present results for
the metrics, tests, and sampling methods that make the best p-value
estimates in a simple machine learning model comparison
View details
Addressing Label Sparsity with Class-Level Common Sense for Google Maps
Sara Mc Carthy
Frontiers in Artificial Intelligence, 5 (2022)
Preview abstract
Successful knowledge graphs (KGs) solved the historical knowledge acquisition bottleneck by supplanting an expert focus with a simple, crowd-friendly one: KG nodes represent popular people, places, organizations, etc., and the graph arcs represent common sense relations like affiliations, locations, etc. Techniques for more general, categorical, KG curation do not seem to have made the same transition: the KG research community is still largely focused on logic-based methods that belie the common-sense characteristics of successful KGs.
In this paper, we propose a simple yet novel approach to acquiring \emph{class-level attributes} from the crowd that represent broad common sense associations between categories, and can be used with the classic knowledge-base default \& override technique (e.g. \cite{reiter1978}) to address the early \textit{label sparsity problem} faced by machine learning systems for problems that lack data for training. We demonstrate the effectiveness of our acquisition and reasoning approach on a pair of very real industrial-scale problems: how to augment an existing KG of places and offerings (e.g. stores and products, restaurants and dishes) with associations between them indicating the availability of the offerings at those places, which would enable the KG to provide answers to questions like, ``Where can I buy milk nearby?'' This problem has several practical challenges but for this paper we focus mostly on the label sparsity. Less than 30\% of physical places worldwide (i.e. brick \& mortar stores and restaurants) have a website, and less than half of those list their product catalog or menus, leaving a large acquisition gap to be filled by methods other than information extraction (IE). Label sparsity is a general problem, and not specific to these use cases, that prevents modern AI and machine learning techniques from applying to many applications for which labeled data is not readily available. As a result, the study of how to acquire the knowledge and data needed for AI to work is as much a problem today as it was in the 1970s and 80s during the advent of expert systems \cite{mycin1975}.
The class-level attributes approach presented here is based on a KG-inspired intuition that a lot of the knowledge people need to understand where to go to buy a product they need, or where to find the dishes they want to eat, is categorical and part of their general common sense: everyone knows grocery stores sell milk and don't sell asphalt, chinese restaurants serve fried rice and not hamburgers, etc. We acquired a mixture of instance- and class- level pairs (e.g. $\langle$\textit{Ajay Mittal Dairy}, milk$\rangle$, $\langle$GroceryStore, milk$\rangle$, resp.) from a novel 3-tier crowdsourcing method, and demonstrate the scalability advantages of the class-level approach. Our results show that crowdsourced class-level knowledge can provide rapid scaling of knowledge acquisition in shopping and dining domains. The acquired common sense knowledge also has long-term value in the KG. The approach was a critical part of enabling a worldwide \textit{local search} capability on Google Maps, with which users can find products and dishes that are available in most places on earth.
View details
Preview abstract
Annotator disagreement is often dismissed as noise or the result of poor annotation process quality. Others have argued that it can be meaningful. But lacking a rigorous statistical foundation, the analysis of disagreement patterns can resemble a high-tech form of tea-leaf-reading.
We contribute a framework for analyzing the variation of per-item annotator response distributions to data for humans-in-the-loop machine learning. We provide visualizations for, and use the framework to analyze the variance in, a crowdsourced dataset of hard-to-classify examples of the OpenImages archive.
View details
Preview abstract
Search engines including Google are beginning to support local-dining queries such as ``At which nearby restaurants can I order the Indonesian salad \textit{gado-gado}?''.
Given the low coverage of online menus worldwide, and only 30\% even having a website, this remains a challenge.
Here we leverage the power of the crowd: online users who are willing to answer questions about dish availability at restaurants visited.
While motivated users are happy to contribute knowledge for free, they are much less likely to respond to ``silly'' or embarrassing questions (e.g., ``Does \textit{Pizza Hut} serve pizza?'' or ``Does \textit{Mike's Vegan Restaurant} serve hamburgers?'')
In this paper, we study the problem of \textit{Vexation-Aware Active Learning}, where judiciously selected questions are targeted towards improving restaurant-dish model prediction, subject to a limit on the percentage of ``unsure'' answers or ``dismissals'' (e.g., swiping the app closed) used to measure vexation.
We formalize the problem as an integer linear program and solve it efficiently using a distributed solution that scales linearly with the number of candidate questions.
Since our algorithm relies on precise estimation of the unsure-dismiss rate (UDR), we give a regression model that provides accurate results compared to baselines including collaborative filtering.
Finally, we demonstrate in a live system that our proposed vexation-aware strategy performs competitively against classical (margin-based) active learning approaches while not exceeding UDR bounds.
View details
Empirical methodology for crowdsourcing ground truth
Anca Dumitrache
Benjamin Timmermans
Oana Inel
Semantic Web Journal, 12:3; 2021 (2021)
Preview abstract
The process of gathering ground truth data through human annotation is a major bottleneck in the use of information
extraction methods for populating the Semantic Web. Crowdsourcing-based approaches are gaining popularity in the attempt to
solve the issues related to volume of data and lack of annotators. Typically these practices use inter-annotator agreement as a
measure of quality. However, in many domains, such as event detection, there is ambiguity in the data, as well as a multitude
of perspectives of the information examples. We present an empirically derived methodology for efficiently gathering of ground
truth data in a diverse set of use cases covering a variety of domains and annotation tasks. Central to our approach is the use of
CrowdTruth metrics that capture inter-annotator disagreement. We show that measuring disagreement is essential for acquiring
a high quality ground truth. We achieve this by comparing the quality of the data aggregated with CrowdTruth metrics with
majority vote, over a set of diverse crowdsourcing tasks: Medical Relation Extraction, Twitter Event Identification, News Event
Extraction and Sound Interpretation. We also show that an increased number of crowd workers leads to growth and stabilization
in the quality of annotations, going against the usual practice of employing a small number of annotators.
View details
Shopping Sense: Rapid Instance-Level Knowledge Acquisition for Google Maps from Class-Level Common Sense
Sara Mc Carthy
Ninth AAAI Conference on Human Computation and Crowdsourcing (HCOMP-2021), pp. 143-154
Preview abstract
Successful knowledge graphs (KGs) solved the historical knowledge acquisition bottleneck by supplanting an expert focus with a simple, crowd-friendly one: KG nodes represent popular people, places, organizations, etc., and the graph arcs represent common sense relations like affiliations, locations, etc. Techniques for more general, categorical, KG curation do not seem to have made the same transition: the KG research community is still largely focused on methods that belie the common-sense characteristics of successful KGs. In this paper, we propose a simple approach to acquiring and reasoning with class-level attributes from the crowd that represent broad common sense associations between categories. We pick a very real industrial-scale data set and problem: how to augment an existing knowledge graph of places and products with associations between them indicating the availability of the products at those places, which would enable a KG to provide answers to questions like, ``Where can I buy milk nearby?'' This problem has several practical challenges, not least of which is that only 30\% of physical stores (i.e. brick \& mortar stores) have a website, and fewer list their product inventory, leaving a large acquisition gap to be filled by methods other than information extraction (IE). Based on a KG-inspired intuition that a lot of the class-level pairs are part of people's general common sense, e.g. everyone knows grocery stores sell milk and don't sell asphalt, we acquired a mixture of instance- and class- level pairs (e.g. {Ajay Mittal Dairy, milk}, {GroceryStore, milk}, resp.) from a novel 3-tier crowdsourcing method, and demonstrate the scalability advantages of the class-level approach. Our results show that crowdsourced class-level knowledge can provide rapid scaling of knowledge acquisition in this and similar domains, as well as long-term value in the KG.
View details
Preview abstract
The AI bookies have spent a lot of time and energy collecting bets from AI researchers, and have met with universal approval of the idea of scientific betting, and nearly universal silence in the acquisition of bets. We have collected a few in this column over the past two years, in the first column we published the “will voice interfaces become the standard” bet, as well as a set of 10 predictions from Eric Horvitz that we proposed as bets awaiting challengers. No challengers have emerged. In this article we review the methods we've used to collect bets and conclude that people need ideas for bets to make. We propose five new bets and solicit participants in them.
View details
Preview abstract
A common step in developing an understanding of a vertical domain, e.g. shopping, dining, movies, medicine, etc., is curating a taxonomy of categories specific to the domain. These human created artifacts have been the subject of research in embeddings that attempt to encode aspects of the partial ordering property of taxonomies. We compare Box Embeddings, a natural containment representation of category taxonomies, to partial-order embeddings and a baseline Bayes Net, in the context of representing the Medical Subject Headings (MeSH) taxonomy given a set of 300K PubMed articles with subject labels from MeSH. We deeply explore the experimental properties of training box embeddings, including preparation of the training data, sampling ratios and class balance, initialization strategies, and propose a fix to the original box objective. We then present first results in using these techniques for representing a bipartite learning problem (i.e. collaborative filtering) in the presence of taxonomic relations within each partition, inferring disease (anatomical) locations from their use as subject labels in journal articles. Our box model substantially outperforms all baselines for taxonomic reconstruction and bipartite relationship experiments. This performance improvement is observed both in overall accuracy and the weighted spread by true taxonomic depth.
View details
