# Scones: Towards Conversational Authoring of Sketches

arXiv: https://arxiv.org/abs/2005.07781 · HTML: https://mlsanity.com/paper/2005.07781/

## Key facts

- Cited by 7 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 12 May 2020.

## Built on by (most cited first, 7 of 7)

- [Beyond Text Generation: Supporting Writers with Continuous Automatic Text Summaries](https://mlsanity.com/paper/2208.09323/) (2022)
- [Modern Evolution Strategies for Creativity: Fitting Concrete Images and Abstract Concepts](https://mlsanity.com/paper/2109.08857/) (2021)
- [Cloud2Curve: Generation and Vectorization of Parametric Sketches](https://mlsanity.com/paper/2103.15536/) (2021)
- [Creating User Interface Mock-ups from High-Level Text Descriptions with Deep-Learning Models](https://mlsanity.com/paper/2110.07775/) (2021)
- [DrawTalking: Building Interactive Worlds by Sketching and Speaking](https://mlsanity.com/paper/2401.05631/) (2024)
- [DrawMon: A Distributed System for Detection of Atypical Sketch Content in Concurrent Pictionary Games](https://mlsanity.com/paper/2211.05429/) (2022)
- [The Image Reconstruction Game: Drawing Common Ground Through Iterative Multimodal Dialogue](https://mlsanity.com/paper/2606.01901/) (2026)

## Abstract (arXiv)

Iteratively refining and critiquing sketches are crucial steps to developing effective designs. We introduce Scones, a mixed-initiative, machine-learning-driven system that enables users to iteratively author sketches from text instructions. Scones is a novel deep-learning-based system that iteratively generates scenes of sketched objects composed with semantic specifications from natural language. Scones exceeds state-of-the-art performance on a text-based scene modification task, and introduces a mask-conditioned sketching model that can generate sketches with poses specified by high-level scene information. In an exploratory user evaluation of Scones, participants reported enjoying an iterative drawing task with Scones, and suggested additional features for further applications. We believe Scones is an early step towards automated, intelligent systems that support human-in-the-loop applications for communicating ideas through sketching in art and design.