# DrawTalking: Building Interactive Worlds by Sketching and Speaking

arXiv: https://arxiv.org/abs/2401.05631 · HTML: https://mlsanity.com/paper/2401.05631/

## Key facts

- Cited by 5 papers in the map (data through 14 Oct 2026).
- Builds on 6 of its 76 references that are in the map.
- First posted on arXiv 11 Jan 2024.

## Builds on (most cited first, 6 of 6)

- [Interactive Task and Concept Learning from Natural Language Instructions and GUI Demonstrations](https://mlsanity.com/paper/1909.00031/) (2019)
- [RealitySketch: Embedding Responsive Graphics and Visualizations in AR through Dynamic Sketching](https://mlsanity.com/paper/2008.08688/) (2020)
- [RealityTalk: Real-Time Speech-Driven Augmented Presentation for AR Live Storytelling](https://mlsanity.com/paper/2208.06350/) (2022)
- [Scones: Towards Conversational Authoring of Sketches](https://mlsanity.com/paper/2005.07781/) (2020)
- [CrossTalk: Intelligent Substrates for Language-Oriented Interaction in Video-Based Communication and Collaboration](https://mlsanity.com/paper/2308.03311/) (2023)
- [Chalktalk : A Visualization and Communication Language -- As a Tool in the Domain of Computer Science Education](https://mlsanity.com/paper/1809.07166/) (2018)

## Built on by (most cited first, 5 of 5)

- [ProactiveVA: Proactive Visual Analytics with LLM-Based UI Agent](https://mlsanity.com/paper/2507.18165/) (2025)
- [Interaction as Intelligence: Deep Research With Human-AI Partnership](https://mlsanity.com/paper/2507.15759/) (2025)
- [MapStory: Prototyping Editable Map Animations with LLM Agents](https://mlsanity.com/paper/2505.21966/) (2025)
- [Steerable Chatbots: Exploring Personalization Control Interfaces via LLM Activation Steering](https://mlsanity.com/paper/2505.04260/) (2025)
- [VoCa: Designing Speech-Canvas Interaction for Voice-Based Conversational Agents](https://mlsanity.com/paper/2610.04706/) (2026)

## Abstract (arXiv)

We introduce DrawTalking, an approach to building and controlling interactive worlds by sketching and speaking while telling stories. It emphasizes user control and flexibility, and gives programming-like capability without requiring code. An early open-ended study with our prototype shows that the mechanics resonate and are applicable to many creative-exploratory use cases, with the potential to inspire and inform research in future natural interfaces for creative exploration and authoring.