# Sketchforme: Composing Sketched Scenes from Text Descriptions for Interactive Applications

arXiv: https://arxiv.org/abs/1904.04399 · HTML: https://mlsanity.com/paper/1904.04399/

## Key facts

- Cited by 7 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 8 Apr 2019.

## Built on by (most cited first, 7 of 7)

- [Modern Evolution Strategies for Creativity: Fitting Concrete Images and Abstract Concepts](https://mlsanity.com/paper/2109.08857/) (2021)
- [A Parse-Then-Place Approach for Generating Graphic Layouts from Textual Descriptions](https://mlsanity.com/paper/2308.12700/) (2023)
- [Sketch-based Creativity Support Tools using Deep Learning](https://mlsanity.com/paper/2111.09991/) (2021)
- [Painter: Teaching Auto-regressive Language Models to Draw Sketches](https://mlsanity.com/paper/2308.08520/) (2023)
- [Inferring spatial relations from textual descriptions of images](https://mlsanity.com/paper/2102.00997/) (2021)
- [Reimagining Application User Interface (UI) Design using Deep Learning Methods: Challenges and Opportunities](https://mlsanity.com/paper/2303.13055/) (2023)
- [Visual Intelligence through Human Interaction](https://mlsanity.com/paper/2111.06913/) (2021)

## Abstract (arXiv)

Sketching and natural languages are effective communication media for interactive applications. We introduce Sketchforme, the first neural-network-based system that can generate sketches based on text descriptions specified by users. Sketchforme is capable of gaining high-level and low-level understanding of multi-object sketched scenes without being trained on sketched scene datasets annotated with text descriptions. The sketches composed by Sketchforme are expressive and realistic: we show in our user study that these sketches convey descriptions better than human-generated sketches in multiple cases, and 36.5% of those sketches are considered to be human-generated. We develop multiple interactive applications using these generated sketches, and show that Sketchforme can significantly improve language learning applications and support intelligent language-based sketching assistants.