Scones: Towards Conversational Authoring of Sketches

Refining sketches through iteration and critique is a crucial step to developing effective designs. We introduce Scones, a mixed-initiative, machine learning driven system which enables users to iteratively create sketches from text instructions. Scones is a novel deep learning based system that generates scenes of sketched objects composed with semantic relationships from natural language. Scones achieves state-of-the-art performance on a text-based scene modification task, and introduces a mask-conditioned sketching model that can generate sketches with poses specified by high-level scene information. In an exploratory user evaluation of Scones, participants reported enjoying an iterative drawing task with Scones, and suggested additional features for further applications. We believe Scones is an early step towards automated, intelligent systems which support human-in-the-loop applications for communicating ideas through drawing in art and design.

• Human-Computer Interaction and Visualization

• Machine Intelligence