Google Research

Schema-Guided Dialogue


The Schema-Guided Dialogue (SGD) dataset consists of over 20k annotated multi-domain, task-oriented conversations between a human and a virtual assistant. These conversations involve interactions with services and APIs spanning 20 domains, such as banks, events, media, calendar, travel, and weather. For most of these domains, the dataset contains multiple different APIs, many of which have overlapping functionalities but different interfaces, which reflects common real-world scenarios. The wide range of available annotations can be used for intent prediction, slot filling, dialogue state tracking, policy imitation learning, language generation, and user simulation learning, among other tasks for developing large-scale virtual assistants. Additionally, the dataset contains unseen domains and services in the evaluation set to quantify the performance in zero-shot or few-shot settings.

The dataset consists of schemas outlining the interface of different APIs alongside annotated dialogues. The dialogues have been generated with the help of a dialogue simulator and paid crowd-workers. The data collection approach is summarized in this paper.

Schema-Guided Dialogue - eXtended (SGD-X) is a benchmark for measuring the robustness of dialogue systems to linguistic variations in schemas. SGD-X extends the SGD dataset with 5 crowdsourced variants for every schema, where variants are semantically similar yet stylistically diverse. Models trained on SGD are evaluated on SGD-X to measure how well they can generalize in a real-world setting, where a large variety of linguistic styles exist.