# RealityTalk: Real-Time Speech-Driven Augmented Presentation for AR Live Storytelling

arXiv: https://arxiv.org/abs/2208.06350 · HTML: https://mlsanity.com/paper/2208.06350/

## Key facts

- Cited by 10 papers in the map (data through 14 Oct 2026).
- Builds on 4 of its 40 references that are in the map.
- First posted on arXiv 12 Aug 2022.

## Builds on (most cited first, 4 of 4)

- [Realtime Multi-Person 2D Pose Estimation using Part Affinity Fields](https://mlsanity.com/paper/1611.08050/) (2016)
- [RealitySketch: Embedding Responsive Graphics and Visualizations in AR through Dynamic Sketching](https://mlsanity.com/paper/2008.08688/) (2020)
- [Chalktalk : A Visualization and Communication Language -- As a Tool in the Domain of Computer Science Education](https://mlsanity.com/paper/1809.07166/) (2018)
- [HoloBoard: a Large-format Immersive Teaching Board based on pseudo HoloGraphics](https://mlsanity.com/paper/2110.01824/) (2021)

## Built on by (most cited first, 10 of 10)

- [ExpressEdit: Video Editing with Natural Language and Sketching](https://mlsanity.com/paper/2403.17693/) (2024)
- [Teachable Reality: Prototyping Tangible Augmented Reality with Everyday Objects by Leveraging Interactive Machine Teaching](https://mlsanity.com/paper/2302.11046/) (2023)
- [Augmented Math: Authoring AR-Based Explorable Explanations by Augmenting Static Math Textbooks](https://mlsanity.com/paper/2307.16112/) (2023)
- [DrawTalking: Building Interactive Worlds by Sketching and Speaking](https://mlsanity.com/paper/2401.05631/) (2024)
- [MapStory: Prototyping Editable Map Animations with LLM Agents](https://mlsanity.com/paper/2505.21966/) (2025)
- [SUMART: SUMmARizing Translation from Wordy to Concise Expression](https://mlsanity.com/paper/2504.09860/) (2025)
- [Empowering Children to Create AI-Enabled Augmented Reality Experiences](https://mlsanity.com/paper/2508.08467/) (2025)
- [Breaking the Curse of Knowledge: Designing Personalized Jargon Support for Real-Time Online Meetings](https://mlsanity.com/paper/2508.10239/) (2025)
- [Reality Copilot: Voice-First Human-AI Collaboration in Mixed Reality Using Large Multimodal Models](https://mlsanity.com/paper/2602.11025/) (2026)
- [Toward User-Mediated Self-Repair in Ubiquitous Robots Through Goal-Oriented Agentic AI](https://mlsanity.com/paper/2609.26157/) (2026)

## Abstract (arXiv)

We present RealityTalk, a system that augments real-time live presentations with speech-driven interactive virtual elements. Augmented presentations leverage embedded visuals and animation for engaging and expressive storytelling. However, existing tools for live presentations often lack interactivity and improvisation, while creating such effects in video editing tools require significant time and expertise. RealityTalk enables users to create live augmented presentations with real-time speech-driven interactions. The user can interactively prompt, move, and manipulate graphical elements through real-time speech and supporting modalities. Based on our analysis of 177 existing video-edited augmented presentations, we propose a novel set of interaction techniques and then incorporated them into RealityTalk. We evaluate our tool from a presenter's perspective to demonstrate the effectiveness of our system.