Google Research

Scones: Towards Conversational Authoring of Sketches

  • David Ha
  • Eldon Schoop
  • Forrest Huang
  • John Canny
IUI 2020 - 25th annual conference on intelligent user interfaces


Refining sketches through iteration and critique is a crucial step to developing effective designs. We introduce Scones, a mixed-initiative, machine learning driven system which enables users to iteratively create sketches from text instructions. Scones is a novel deep learning based system that generates scenes of sketched objects composed with semantic relationships from natural language. Scones achieves state-of-the-art performance on a text-based scene modification task, and introduces a mask-conditioned sketching model that can generate sketches with poses specified by high-level scene information. In an exploratory user evaluation of Scones, participants reported enjoying an iterative drawing task with Scones, and suggested additional features for further applications. We believe Scones is an early step towards automated, intelligent systems which support human-in-the-loop applications for communicating ideas through drawing in art and design.

Learn more about how we do research

We maintain a portfolio of research projects, providing individuals and teams the freedom to emphasize specific types of work