# TalkSketch: Multimodal Generative AI for Real-time Sketch Ideation with Speech

arXiv: https://arxiv.org/abs/2511.05817 · HTML: https://mlsanity.com/paper/2511.05817/

## Key facts

- Cited by 2 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 8 Nov 2025.

## Built on by (most cited first, 2 of 2)

- [When Drawing Is Not Enough: Exploring Spontaneous Speech with Sketch for Intent Alignment in Multimodal LLMs](https://mlsanity.com/paper/2604.11964/) (2026)
- [Creativity from Friction: Human-AI Interaction for Exploratory Structural Design](https://mlsanity.com/paper/2607.07521/) (2026)

## Abstract (arXiv)

Sketching is a widely used medium for generating and exploring early-stage design concepts. While generative AI (GenAI) chatbots are increasingly used for idea generation, designers often struggle to craft effective prompts and find it difficult to express evolving visual concepts through text alone. In the formative study (N=6), we examined how designers use GenAI during ideation, revealing that text-based prompting disrupts creative flow. To address these issues, we developed TalkSketch, an embedded multimodal AI sketching system that integrates freehand drawing with real-time speech input. TalkSketch aims to support a more fluid ideation process through capturing verbal descriptions during sketching and generating context-aware AI responses. Our work highlights the potential of GenAI tools to engage the design process itself rather than focusing on output.