Tom Kwiatkowski
I am a research scientist at Google where I work on computational Natural Language Understanding, with applications in search and question answering. Prior to this, I got my PhD at the University of Edinburgh and I did a Post-Doc at the University of Washington.
Research Areas
Authored Publications
Google Publications
Other Publications
Sort By
1-Pager: One Pass Answer Generation and Evidence Retrieval
Palak Jain
The 2023 Conference on Empirical Methods in Natural Language Processing(2023) (to appear)
Preview abstract
We present 1-PAGER the first system that answers a question and retrieves evidence using
a single Transformer-based model and decoding process. 1-PAGER incrementally partitions
the retrieval corpus using constrained decoding to select a document and answer string,
and we show that this is competitive with comparable retrieve-and-read alternatives according to both retrieval and answer accuracy metrics. 1-PAGER also outperforms the equivalent ‘closed-book’ question answering model,
by grounding predictions in an evidence corpus. While 1-PAGER is not yet on-par with
more expensive systems that read many more
documents before generating an answer, we argue that it provides an important step toward
attributed generation by folding retrieval into
the sequence-to-sequence paradigm that is currently dominant in NLP. We also show that the
search paths used to partition the corpus are
easy to read and understand, paving a way forward for interpretable neural retrieval.
View details
NAIL: Lexical Retrieval Indices with Efficient Non-Autoregressive Decoders
Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, EMNLP 2023 (to appear)
Preview abstract
Neural document rerankers are extremely effective in terms of accuracy. However, the best models require dedicated hardware for serving, which is costly and often not feasible. To avoid this serving-time requirement, we present a method of capturing up to 86% of the gains of a Transformer cross-attention model with a lexicalized scoring function that only requires 10-6% of the Transformer's FLOPs per document and can be served using commodity CPUs. When combined with a BM25 retriever, this approach matches the quality of a state-of-the art dual encoder retriever, that still requires an accelerator for query encoding. We introduce NAIL (Non-Autoregressive Indexing with Language models) as a model architecture that is compatible with recent encoder-decoder and decoder-only large language models, such as T5, GPT-3 and PaLM. This model architecture can leverage existing pre-trained checkpoints and can be fine-tuned for efficiently constructing document representations that do not require neural processing of queries.
View details
Attributed Question Answering: Evaluation and Modeling for Attributed Large Language Models
Pat Verga
Jacob Eisenstein
Jianmo Ni
William Weston Cohen
Slav Petrov
arXiv(2022)
Preview abstract
Large language models (LLMs) have shown impressive results across a variety of tasks while requiring little or no direct supervision. Further, there is mounting evidence that LLMs may have potential in information-seeking scenarios. We believe the ability of an LLM to attribute the text that it generates is likely to be crucial for both system developers and users in this setting. We propose and study Attributed QA as a key first step in the development of attributed LLMs. We develop a reproducable evaluation framework for the task, using human annotations as a gold standard and a correlated automatic metric that we show is suitable for development settings. We describe and benchmark a broad set of architectures for the task. Our contributions give some concrete answers to two key questions (How to measure attribution?, and How well do current state-of-the-art methods perform on attribution?), and give some hints as to how to address a third key question (How to build LLMs with attribution?).
View details
MOLEMAN: Mention-Only Linking of Entities with a Mention Annotation Network
Jan A. Botha
Dan Bikel
Andrew McCallum
Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 2: Short Papers), Association for Computational Linguistics, Online(2021), pp. 278-285
Preview abstract
We present an instance-based nearest neighbor approach to entity linking. In contrast to most prior entity retrieval systems which represent each entity with a single vector, we build a contextualized mention-encoder that learns to place similar mentions of the same entity closer in vector space than mentions of different entities. This approach allows all mentions of an entity to serve as "class prototypes" as inference involves retrieving from the full set of labeled entity mentions in the training set and applying the nearest mention neighbor’s entity label. Our model is trained on a large multilingual corpus of mention pairs derived from Wikipedia hyperlinks, and performs nearest neighbor inference on an index of 700 million mentions. It is simpler to train, gives more interpretable predictions, and outperforms all other systems on two multilingual entity linking benchmarks.
View details
TyDi QA: A Benchmark for Information-Seeking Question Answering in Typologically Diverse Languages
Eunsol Choi
Transactions of the Association for Computational Linguistics(2020)
Preview abstract
Confidently making progress on multilingual modeling requires challenging, trustworthy evaluations. We present TyDi QA, a question answering dataset covering 11 typologically diverse languages. Until recently, most multilingual research in natural language processing has been limited to machine translation or to technical tasks such as tagging and parsing. Question answering offers a scenario that is natural in that non-technical users intuitively understand the task, allowing high quality data collection in the absence of abundant annotators with expertise in both linguistics and the language of interest. This allows us select languages that are diverse with regard to their typology -- the set of linguistic features that each language expresses. We expect that models that can perform well on this set will generalize across a large number of the languages in the world. To encourage a more realistic distribution, the data is collected entirely in each native language without the use of translation (human or otherwise) and question creation is performed without seeing the answers. We present a quantitative analysis of the data quality, we provide example-level linguistic analyses and glosses of language phenomena that would not be found in English-only corpora, and we measure the performance of baseline systems.
View details
Learning Cross-Context Entity Representations from Text
Jeffrey Ling
Zifei Shan
Thibault Févry
arXiv(2020)
Preview abstract
Language modeling tasks, in which words, or word-pieces, are predicted on the basis of a local context, have been very effective for learning word embeddings and context dependent representations of phrases. Motivated by the observation that efforts to code world knowledge into machine readable knowledge bases or human readable encyclopedias tend to be entity-centric, we investigate the use of a fill-in-the-blank task to learn context independent representations of entities from the text contexts in which those entities were mentioned. We show that large scale training of neural models allows us to learn high quality entity representations, and we demonstrate successful results on four domains: (1) existing entity-level typing benchmarks, including a 64% error reduction over previous work on TypeNet (Murty et al., 2018); (2) a novel few-shot category reconstruction task; (3) existing entity linking benchmarks, where we match the state-of-the-art on CoNLL-Aida without linking-specific features and obtain a score of 89.8% on TAC-KBP 2010 without using any alias table, external knowledge base or in domain training data and (4) answering trivia questions, which uniquely identify entities. Our global entity representations encode fine-grained type categories, such as Scottish footballers, and can answer trivia questions such as: Who was the last inmate of Spandau jail in Berlin?
View details
Empirical Evaluation of Pretraining Strategies for Supervised Entity Linking
Thibault Févry
AKBC 2020 - Automated Knowledge Base Construction
Preview abstract
In this work, we present an entity linking model which combines a Transformer architecture with large scale pretraining from Wikipedia links. Our model achieves the state-of-the-art on two commonly used entity linking datasets: 96.7% on CoNLL and 94.9% on TAC-KBP. We present detailed analyses to understand what design choices are important for entity linking, including choices of negative entity candidates, Transformer architecture, and input perturbations. Lastly, we present promising results on more challenging settings such as end-to-end entity linking and entity linking without in-domain training data.
View details
Entities as Experts: Sparse Memory Access with Entity Supervision
Thibault Févry
Eunsol Choi
EMNLP 2020 - Conference on Empirical Methods in Natural Language Processing (to appear)
Preview abstract
We focus on the problem of capturing declarative knowledge about entities in the learned parameters of a language model. We introduce a new model—Entities as Experts (EAE)— that can access distinct memories of the entities mentioned in a piece of text. Unlike previous efforts to integrate entity knowledge into sequence models, EAE’s entity representations are learned directly from text. We show that EAE’s learned representations capture sufficient knowledge to answer TriviaQA questions such as “Which Dr. Who villain has been played by Roger Delgado, Anthony Ainley, Eric Roberts?”, outperforming an encoder-generator Transformer model with 10× the parameters. According to the LAMA knowledge probes, EAE contains more factual knowledge than a similarly sized BERT, as well as previous approaches that integrate external sources of entity knowledge. Because EAE associates parameters with specific entities, it only needs to access a fraction of its parameters at inference time, and we show that the correct identification and representation of entities is essential to EAE’s performance.
View details
Natural Questions: a Benchmark for Question Answering Research
Olivia Redfield
Danielle Epstein
Illia Polosukhin
Matthew Kelcey
Jacob Devlin
Llion Jones
Ming-Wei Chang
Jakob Uszkoreit
Slav Petrov
Transactions of the Association of Computational Linguistics(2019) (to appear)
Preview abstract
We present the Natural Questions corpus, a question answering dataset. Questions consist of real anonymized, aggregated queries issued to the Google search engine. An annotator is presented with a question along with a Wikipedia page from the top 5 search results, and annotates a long answer (typically a paragraph) and a short answer (one or more entities) if present on the page, or marks null if no long/short answer is present. The public release consists of 307,373 training examples with single annotations, 7,830 examples with 5-way annotations for development data, and a further 7,842 examples 5-way annotated sequestered as test data. We present experiments validating quality of the data. We also describe analysis of 25-way annotations on 302 examples, giving insights into human variability on the annotation task. We introduce robust metrics for the purposes of evaluating question answering systems; demonstrate high human upper bounds on these metrics; and establish baseline results using competitive methods drawn from related literature.
View details
BoolQ: Exploring the Surprising Difficulty of Natural Yes/No Questions
Christopher Clark
Ming-Wei Chang
NAACL 2019
Preview abstract
In this paper we study yes/no questions that are naturally occurring---meaning that they are generated in unprompted and unconstrained settings. We build a reading comprehension dataset, BoolQ, of such questions, and show that they are unexpectedly challenging. They often query for complex, non-factoid information, and require difficult entailment-like inference to solve.
We also explore the effectiveness of a range of transfer learning baselines. We find that transferring from entailment data is more effective than transferring from paraphrase or extractive QA data, and that it, surprisingly, continues to be very beneficial even when starting from massive pre-trained language models such as BERT. Our best method trains BERT on MultiNLI and then re-trains it on our train set. It achieves 80.4% accuracy compared to 90% accuracy of human annotators (and 62% majority-baseline), leaving a significant gap for future work.
View details
Preview abstract
Language modeling tasks, in which words are predicted on the basis of a local context, have been very effective for learning word embeddings and context dependent representations of phrases. Motivated by the observation that efforts to code world knowledge into machine readable knowledge bases tend to be entity-centric, we investigate the use of a fill-in-the-blank task to learn context independent representations of entities from the contexts in which those entities were mentioned. We show that large scale training of neural models allows us to learn extremely high fidelity entity typing information, which we demonstrate with few-shot reconstruction of Wikipedia categories. Our learning approach is powerful enough to encode specialized topics such as Giro d'Italia cyclists.
View details
Matching the Blanks: Distributional Similarity for Relation Learning
Jeffrey Ling
ACL 2019 - The 57th Annual Meeting of the Association for Computational Linguistics(2019) (to appear)
Preview abstract
General purpose relation extractors, which can model arbitrary relations, are a core aspiration in information extraction. Efforts have been made to build general purpose extractors that represent relations with their surface forms, or which jointly embed surface forms with relations from an existing knowledge graph. How ever, both of these approaches are limited in their ability to generalize. In this paper, we build on extensions of Harris’ distributional hypothesis to relations, as well as recent advances in learning text representations (specifically, BERT), to build task agnostic relation representations solely from entity-linked text.
We show that these representations significantly outperform previous work on exemplar based relation extraction (FewRel) even without using any of that task’s training data. We also show that models initialized with our task agnostic representations, and then tuned on supervised relation extraction datasets, significantly outperform the previous methods on SemEval 2010 Task 8, KBP37, and TACRED.
View details
Incremental Learning from Text for Question Answering
Samira Abnar
William Weston Cohen
Continual Learning Workshop, Neural Information Processing Systems (NIPS) 2018
Preview abstract
Any system which performs goal-directed continual learning must not only learn incrementally but process and absorb information incrementally. Such a system also has to understand when its goals have been achieved. In this paper, we consider these issues in the context of question answering. Current state-of-the-art question answering models reason over an entire passage, not incrementally. As we will show, naive approaches to incremental reading, such as restriction to unidirectional language models in the model, perform poorly. We present extensions to the DocQA [2] model to allow incremental reading without loss of accuracy. The model also jointly learns to provide the best answer given the text that is seen so far and predict whether this best-so-far answer is sufficient.
View details
Multi-Mention Learning for Reading Comprehension with Neural Cascades
Swabha Swayamdipta
International Conference on Learning Representations (ICLR)(2018)
Preview abstract
Reading comprehension is a challenging task, especially when executed across
longer or across multiple evidence documents, where the answer is likely to reoccur.
Existing neural architectures typically do not scale to the entire evidence,
and hence, resort to selecting a single passage in the document (either via truncation
or other means), and carefully searching for the answer within that passage.
However, in some cases, this strategy can be suboptimal, since by focusing on a
specific passage, it becomes difficult to leverage multiple mentions of the same
answer throughout the document. In this work, we take a different approach by
constructing lightweight models that are combined in a cascade to find the answer.
Each submodel consists only of feed-forward networks equipped with an attention
mechanism, making it trivially parallelizable. We show that our approach can
scale to approximately an order of magnitude larger evidence documents and can
aggregate information from multiple mentions of each answer candidate across
the document. Empirically, our approach achieves state-of-the-art performance
on both the Wikipedia and web domains of the TriviaQA dataset, outperforming
more complex, recurrent architectures.
View details
Learning Recurrent Span Representations for Extractive Question Answering
Shimi Salant
Jonathan Berant
arXiv 1611.01436(2017)
Preview abstract
The reading comprehension task, that asks questions about a given evidence document,
is a central problem in natural language understanding. Recent formulations
of this task have typically focused on answer selection from a set of candidates
pre-defined manually or through the use of an external NLP pipeline. However,
Rajpurkar et al. (2016) recently released the SQUAD dataset in which the answers
can be arbitrary strings from the supplied text. In this paper, we focus on
this answer extraction task, presenting a novel model architecture that efficiently
builds fixed length representations of all spans in the evidence document with a recurrent
network. We show that scoring explicit span representations significantly
improves performance over other approaches that factor the prediction into separate
predictions about words or start and end markers. Our approach improves
upon the best published results of Wang & Jiang (2016) by 5% and decreases the
error of Rajpurkar et al.’s baseline by > 50%.
View details
Transforming Dependency Structures to Logical Forms for Semantic Parsing
Siva Reddy
Oscar Täckström
Mark Steedman
Mirella Lapata
Transactions of the Association for Computational Linguistics, 4(2016)
Preview abstract
The strongly typed syntax of grammar formalisms such as CCG, TAG, LFG and HPSG offers a synchronous framework for deriving syntactic
structures and semantic logical forms. In contrast - partly due to the lack of a strong type system - dependency structures are easy to annotate and have become a widely used form of syntactic analysis for many languages. However, the lack of a type system makes a formal
mechanism for deriving logical forms from dependency structures challenging. We address this by introducing a robust system based on the lambda calculus for deriving neo-Davidsonian logical forms from dependency trees. These logical forms are then used for semantic parsing of natural language to Freebase. Experiments on the Free917 and WebQuestions datasets show that our representation is superior to the original dependency trees and that it outperforms a CCG-based representation on this task. Compared to prior work, we obtain the strongest result to date on Free917 and competitive results on WebQuestions.
View details
Scalable semantic parsing with partial ontologies
Preview abstract
We consider the problem of building scalable semantic parsers for Freebase, and present a new approach for learning to do partial analyses that ground as much of the input text as possible without requiring that all content words be mapped to Freebase concepts. We study this problem on two newly introduced large-scale noun phrase datasets, and present a new semantic parsing model and semi-supervised learning approach for reasoning with partial ontological support. Experiments demonstrate strong performance on two tasks: referring expression resolution and entity attribute extraction. In both cases, the partial analyses allow us to improve precision over strong baselines, while parsing many phrases that would be ignored by existing techniques.
View details
Morpho-syntactic Lexical Generalization for CCG Semantic Parsing.
Adrienne Wang
Luke S Zettlemoyer
Proceedings of the Conference on Empirical Methods in Natural Language Processing(2014)
Preview abstract
In this paper, we demonstrate that significant performance gains can be
achieved in CCG semantic parsing by introducing a linguistically motivated
grammar induction scheme. We present a new morpho-syntactic factored
lexicon that models systematic variations in morphology, syntax, and
semantics across word classes. The grammar uses domain-independent
facts about the English language to restrict the number of incorrect parses
that must be considered, thereby enabling effective learning from less
data. Experiments in benchmark domains match previous models with
one quarter of the data and provide new state-of-the-art results with all
available data, including up to 45% relative test-error reduction.
View details
Scaling Semantic Parsers with On-the-fly Ontology Matching
Eunsol Choi
Yoav Artzi
Luke Zettlemoyer
Proceedings of the 2013 conference on empirical methods in natural language processing
Preview abstract
We consider the challenge of learning semantic parsers that scale to large, open-domain problems, such as question answering with Freebase. In such settings, the sentences cover a wide variety of topics and include many phrases whose meaning is difficult to represent in a fixed target ontology. For example, even simple phrases such as 'daughter' and 'number of people living in' cannot be directly represented in Freebase, whose ontology instead encodes facts about gender, parenthood, and population. In this paper, we introduce a new semantic parsing approach that learns to resolve such ontologi-cal mismatches. The parser is learned from question-answer pairs, uses a probabilistic CCG to build linguistically motivated logical-form meaning representations, and includes an ontology matching model that adapts the output logical forms for each target ontology. Experiments demonstrate state-of-the-art performance on two benchmark semantic parsing datasets, including a nine point accuracy improvement on a recent Freebase QA corpus.
View details
A probabilistic model of syntactic and semantic acquisition from child-directed utterances and their meanings
Sharon Goldwater
Luke Zettlemoyer
Mark Steedman
Proceedings of the 13th Conference of the European Chapter of the Association for Computational Linguistics(2012)
Preview abstract
This paper presents an incremental probabilistic learner that models the acquistion of syntax and semantics from a corpus of child-directed utterances paired with possible representations of their meanings. These meaning representations approximate the contextual input available to the child; they do not specify the meanings of individual words or syntactic derivations. The learner then has to infer the meanings and syntactic properties of the words in the input along with a parsing model. We use the CCG grammatical framework and train a non-parametric Bayesian model of parse structure with online variational Bayesian expectation maximization. When tested on utterances from the CHILDES corpus, our learner outperforms a state-of-the-art semantic parser. In addition, it models such aspects of child acquisition as "fast mapping," while also countering previous criticisms of statistical syntactic learners.
View details
Using syntactic and confusion network structure for out-of-vocabulary word detection
Alex Marin
Mari Ostendorf
Luke Zettlemoyer
Spoken Language Technology Workshop (SLT), 2012 IEEE
Preview abstract
This paper addresses the problem of detecting words that are out-of-vocabulary (OOV) for a speech recognition system to improve automatic speech translation. The detection system leverages confidence prediction techniques given a confusion network representation and parsing with OOV word tokens to
identify spans associated with true OOV words. Working in a resource-constrained domain, we achieve OOV detection Fscores of 60-66 and reduce word error rate by 12% re
View details
Lexical generalization in CCG grammar induction for semantic parsing
Luke Zettlemoyer
Sharon Goldwater
Mark Steedman
Proceedings of the 2011 conference on empirical methods in natural language processing
Preview abstract
We consider the problem of learning factored probabilistic CCG grammars for semantic
parsing from data containing sentences paired with logical-form meaning representations.
Traditional CCG lexicons list lexical items that pair words and phrases with syntactic
and semantic content. Such lexicons can be inefficient when words appear repeatedly
with closely related lexical content. In this paper, we introduce factored lexicons, which
include both lexemes to model word meaning and templates to model systematic variation in
word usage. We also present an algorithm for learning factored CCG lexicons, along with a
probabilistic parse-selection model. Evaluations on benchmark datasets demonstrate that
the approach learns highly accurate parsers, whose generalization performance benefits
greatly from the lexical factoring.
View details
Inducing probabilistic CCG grammars from logical form with higher-order unification
Luke Zettlemoyer
Sharon Goldwater
Mark Steedman
Proceedings of the 2010 conference on Empirical Methods in Natural Language Processing
Preview abstract
This paper addresses the problem of learning to map sentences to logical form, given
training data consisting of natural language sentences paired with logical representations
of their meaning. Previous approaches have been designed for particular natural languages
or specific meaning representations; here we present a more general method. The approach
induces a probabilistic CCG grammar that represents the meaning of individual words
and defines how these meanings can be combined to analyze complete sentences. We
use higher-order unification to define a hypothesis space containing all grammars consistent
with the training data, and develop an online learning algorithm that efficiently
searches this space while simultaneously estimating the parameters of a log-linear parsing
model. Experiments demonstrate high accuracy on benchmark data sets in four languages
with two different meaning representations.
View details
