Christopher Hidey
Research Areas
Authored Publications
Sort By
Preview abstract
Model churn occurs when re-training a model yields different predictions despite using the same data and hyper-parameters. Churn reduction is crucial for industry conversational systems where users expect consistent results for the same queries. In this setting, compute resources are often limited due to latency requirements during serving and overall time constraints during re-training. To address this issue, we propose a compute-efficient method that mitigates churn without requiring extra resources for training or inference. Our approach involves a lightweight data pre-processing step that pairs semantic parses based on their “function call signature” and encourages similarity through an additional loss based on Jensen-Shannon Divergence. We validate the effectiveness of our method in three scenarios: academic (+3.93 percent improvement on average in a churn reduction metric), simulated noisy data (+8.09), and industry (+5.28) settings.
View details
DAMP: Doubly Aligned Multilingual Parser for Task-Oriented Dialogue
William Held
Rahul Goel
Diyi Yang
Rushin Shah
Association for Computational Linguistics (2023)
Preview abstract
Modern virtual assistants use internal semantic parsing engines to convert user utterances to actionable commands. However, prior work has demonstrated that semantic parsing is a difficult multilingual transfer task with low transfer efficiency compared to other tasks. In global markets such as India and Latin America, this is a critical issue as switching between languages is prevalent for bilingual users. In this work we dramatically improve the zero-shot performance of a multilingual and codeswitched semantic parsing system using two stages of multilingual alignment. First, we show that constrastive alignment pretraining improves both English performance and transfer efficiency. We then introduce a constrained optimization approach for hyperparameter-free adversarial alignment during finetuning. Our Doubly Aligned Multilingual Parser (DAMP) improves mBERT transfer performance by 3x, 6x, and 81x on the Spanglish, Hinglish and Multilingual Task Oriented Parsing benchmarks respectively and outperforms XLM-R and mT5-Large using 3.2x fewer parameters.
View details
Preview abstract
Semantic parser is a core component of modern vir-tual assistants like Google Assistant and Amazon Alexa.While sequence-to-sequence based auto-regressive (AR) ap-proaches are common for conversational semantic parsing,recent studies (Babu et al. 2021; Shrivastava et al. 2021) em-ploy non-autoregressive (NAR) decoders to reduce inferencelatency while maintaining competitive parsing quality. How-ever, a major drawback of NAR decoders is the difficulty of generating top-koutputs with approaches such as beam search. Due to inherent ambiguity in natural language, gener-ating diverse top-koutputs is essential for conversational se-mantic parsers. To address this challenge, we propose a novelNAR semantic parser which introduces intent conditioning on the decoder. Inspired by the traditional intent and slot tagging parsers, we decouple the first intent prediction from the rest of the parse. The intent conditioning allows the model to better control beam-search and improves the quality and diversity oftop-koutputs. Since we do not have top-klabels during train-ing, to avoid training and inference mismatch, we introduce a hybrid teacher-forcing approach. We evaluate our proposedapproach on conversational semantic parsing datasets, TOP and TOPv2. Similar to the existing NAR models we maintain theO(1)decoding time complexity while generating more diverse outputs and improving top-3 exact match (EM) by2.4points. In comparison with AR models, our approach speeds up beam-search inference by6.7times on CPU with compet-itive top-kEM
View details
Preview abstract
Retraining modern deep learning systems can lead to variations in model performance even when trained using the same data and hyperparameters by simply using different random seeds. This phenomenon is known as model churn or model jitter. This issue is often exacerbated in real world settings, where noise may be introduced in the data collection process. In this work we tackle the problem of stable retraining with a novel focus on structured prediction for conversational semantic parsing. We first quantify the model churn by introducing metrics for agreement between predictions across multiple re-trainings. Next, we devise realistic scenarios for noise injection and demonstrate the effectiveness of various churn reduction techniques such as ensembling and distillation. Lastly, we discuss practical tradeoffs between such techniques and show that codistillation provides a sweet spot in terms of churn reduction with only a modest increase in resource usage.
View details
