# VoCa: Designing Speech-Canvas Interaction for Voice-Based Conversational Agents

arXiv: https://arxiv.org/abs/2610.04706 · HTML: https://mlsanity.com/paper/2610.04706/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 23 of its 82 references that are in the map.
- First posted on arXiv 3 Oct 2026.

## Builds on (most cited first, 23 of 23)

- [Beyond Turn-Based Interfaces: Synchronous LLMs as Full-Duplex Dialogue Agents](https://mlsanity.com/paper/2409.15594/) (2024)
- [Graphologue: Exploring Large Language Model Responses with Interactive Diagrams](https://mlsanity.com/paper/2305.11473/) (2023)
- [Luminate: Structured Generation and Exploration of Design Space with Large Language Models for Human-AI Co-Creation](https://mlsanity.com/paper/2310.12953/) (2023)
- [GazePointAR: A Context-Aware Multimodal Voice Assistant for Pronoun Disambiguation in Wearable Augmented Reality](https://mlsanity.com/paper/2404.08213/) (2024)
- [DynaVis: Dynamically Synthesized UI Widgets for Visualization Editing](https://mlsanity.com/paper/2401.10880/) (2024)
- [Generative and Malleable User Interfaces with Generative and Evolving Task-Driven Data Model](https://mlsanity.com/paper/2503.04084/) (2025)
- [User Interaction Patterns and Breakdowns in Conversing with LLM-Powered Voice Assistants](https://mlsanity.com/paper/2309.13879/) (2023)
- [Sensible Agent: A Framework for Unobtrusive Interaction with Proactive AR Agents](https://mlsanity.com/paper/2509.09255/) (2025)
- [GPTVoiceTasker: Advancing Multi-step Mobile Task Efficiency Through Dynamic Interface Exploration and Learning](https://mlsanity.com/paper/2401.14268/) (2024)
- [Data Player: Automatic Generation of Data Videos with Narration-Animation Interplay](https://mlsanity.com/paper/2308.04703/) (2023)
- [Interactive Sketchpad: A Multimodal Tutoring System for Collaborative, Visual Problem-Solving](https://mlsanity.com/paper/2503.16434/) (2025)
- [VOICE: Visual Oracle for Interaction, Conversation, and Explanation](https://mlsanity.com/paper/2304.04083/) (2023)
- [Cocobo: Exploring Large Language Models as the Engine for End-User Robot Programming](https://mlsanity.com/paper/2407.20712/) (2024)
- [DrawTalking: Building Interactive Worlds by Sketching and Speaking](https://mlsanity.com/paper/2401.05631/) (2024)
- [Screen or No Screen? Lessons Learnt from a Real-World Deployment Study of Using Voice Assistants With and Without Touchscreen for Older Adults](https://mlsanity.com/paper/2307.07723/) (2023)
- [A Multimodal GUI Architecture for Interfacing with LLM-Based Conversational Assistants](https://mlsanity.com/paper/2510.06223/) (2025)
- [Proactive Agentic Whiteboards: Enhancing Diagrammatic Learning](https://mlsanity.com/paper/2512.01234/) (2025)
- [PointAloud: An Interaction Suite for AI-Supported Pointer-Centric Think-Aloud Computing](https://mlsanity.com/paper/2602.09296/) (2026)
- [What Are You Doing? Effects of Intermediate Feedback from Agentic LLM In-Car Assistants During Multi-Step Processing](https://mlsanity.com/paper/2602.15569/) (2026)
- [Orality: A Semantic Canvas for Externalizing and Clarifying Thoughts with Speech](https://mlsanity.com/paper/2603.02544/) (2026)
- [Speech-Synchronized Whiteboard Generation via VLM-Driven Structured Drawing Representations](https://mlsanity.com/paper/2603.25870/) (2026)
- [When Drawing Is Not Enough: Exploring Spontaneous Speech with Sketch for Intent Alignment in Multimodal LLMs](https://mlsanity.com/paper/2604.11964/) (2026)
- [Who's That Player?: Externalizing Query Interpretation in Spoken XR Sports Interaction](https://mlsanity.com/paper/2608.00876/) (2026)

## Abstract (arXiv)

People write and sketch while speaking to explain, organize, and develop content together. Inspired by these practices, we investigate how voice agents can use a canvas alongside speech in multi-turn conversations with users. We conducted a two-part formative study: an observational study of how pairs coordinated speech and boardwork, followed by a design workshop that informed a design space for speech-canvas interaction with voice agents. Building on these insights, we developed VoCa, a voice agent that coordinates speech with visual object creation, annotation, and attention guidance. A five-day deployment with 18 participants examined usability, experiences of speech-canvas interaction, patterns of use, and desired improvements. Participants' experiences highlighted opportunities for speech-canvas interaction in learning, work, and daily life, alongside challenges in coordinating what agents say and show in ways users can follow and influence. These findings inform how voice agents can use a canvas alongside speech in conversation.