# From Speech-to-Spatial: Grounding Utterances on A Live Shared View with Augmented Reality

arXiv: https://arxiv.org/abs/2602.03059 · HTML: https://mlsanity.com/paper/2602.03059/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Builds on 28 of its 73 references that are in the map.
- First posted on arXiv 3 Feb 2026.

## Builds on (most cited first, 28 of 28)

- [From Local to Global: A Graph RAG Approach to Query-Focused Summarization](https://mlsanity.com/paper/2404.16130/) (2024)
- [ConceptGraphs: Open-Vocabulary 3D Scene Graphs for Perception and Planning](https://mlsanity.com/paper/2309.16650/) (2023)
- [MASt3R-SLAM: Real-Time Dense SLAM with 3D Reconstruction Priors](https://mlsanity.com/paper/2412.12392/) (2024)
- [3D Dynamic Scene Graphs: Actionable Spatial Perception with Places, Objects, and Humans](https://mlsanity.com/paper/2002.06289/) (2020)
- [Language Conditioned Spatial Relation Reasoning for 3D Object Grounding](https://mlsanity.com/paper/2211.09646/) (2022)
- [GazePointAR: A Context-Aware Multimodal Voice Assistant for Pronoun Disambiguation in Wearable Augmented Reality](https://mlsanity.com/paper/2404.08213/) (2024)
- [Satori: Towards Proactive AR Assistant with Belief-Desire-Intention User Modeling](https://mlsanity.com/paper/2410.16668/) (2024)
- [Memoro: Using Large Language Models to Realize a Concise Interface for Real-Time Memory Augmentation](https://mlsanity.com/paper/2403.02135/) (2024)
- [Sensible Agent: A Framework for Unobtrusive Interaction with Proactive AR Agents](https://mlsanity.com/paper/2509.09255/) (2025)
- [Augmented Object Intelligence with XR-Objects](https://mlsanity.com/paper/2404.13274/) (2024)
- [OmniQuery: Contextually Augmenting Captured Multimodal Memory to Enable Personal Question Answering](https://mlsanity.com/paper/2409.08250/) (2024)
- [Encode-Store-Retrieve: Augmenting Human Memory through Language-Encoded Egocentric Perception](https://mlsanity.com/paper/2308.05822/) (2023)
- [ImaginateAR: AI-Assisted In-Situ Authoring in Augmented Reality](https://mlsanity.com/paper/2504.21360/) (2025)
- [Guided Reality: Generating Visually-Enriched AR Task Guidance with LLMs and Vision Models](https://mlsanity.com/paper/2508.03547/) (2025)
- [Design Patterns for Situated Visualization in Augmented Reality](https://mlsanity.com/paper/2307.09157/) (2023)
- [Explainable XR: Understanding User Behaviors of XR Environments using LLM-assisted Analytics Framework](https://mlsanity.com/paper/2501.13778/) (2025)
- [Interactive and Incremental Learning of Spatial Object Relations from Human Demonstrations](https://mlsanity.com/paper/2305.09551/) (2023)
- [A Deixis-Centered Approach for Documenting Remote Synchronous Communication around Data Visualizations](https://mlsanity.com/paper/2408.04041/) (2024)
- [Meta-Objects: Interactive and Multisensory Virtual Objects Learned from the Real World for Use in Augmented Reality](https://mlsanity.com/paper/2404.17179/) (2024)
- [Augmented Conversation with Embedded Speech-Driven On-the-Fly Referencing in AR](https://mlsanity.com/paper/2405.18537/) (2024)
- [GesPrompt: Leveraging Co-Speech Gestures to Augment LLM-Based Interaction in Virtual Reality](https://mlsanity.com/paper/2505.05441/) (2025)
- [Remote Assistance with Mixed Reality for Procedural Tasks](https://mlsanity.com/paper/2208.03261/) (2022)
- [Context-Dependent Memory in Situated Visualization](https://mlsanity.com/paper/2311.12288/) (2023)
- [Learning to Generate Unambiguous Spatial Referring Expressions for Real-World Environments](https://mlsanity.com/paper/1904.07165/) (2019)
- [Video-Conferencing Beyond Screen-Sharing and Thumbnail Webcam Videos: Gesture-Aware Augmented Reality Video for Data-Rich Remote Presentations](https://mlsanity.com/paper/2501.05345/) (2025)
- [Reality Proxy: Fluid Interactions with Real-World Objects in MR via Abstract Representations](https://mlsanity.com/paper/2507.17248/) (2025)
- [Revisiting put-that-there, context aware window interactions via LLMs](https://mlsanity.com/paper/2511.02378/) (2025)
- [CORA: Consistency-Guided Semi-Supervised Framework for Reasoning Segmentation](https://mlsanity.com/paper/2511.17755/) (2025)

## Built on by (most cited first, 1 of 1)

- [VisionClaw: Always-On AI Agents through Smart Glasses](https://mlsanity.com/paper/2604.03486/) (2026)

## Abstract (arXiv)

We introduce Speech-to-Spatial, a referent disambiguation framework that converts verbal remote-assistance instructions into spatially grounded AR guidance. Unlike prior systems that rely on additional cues (e.g., gesture, gaze) or manual expert annotations, Speech-to-Spatial infers the intended target solely from spoken references (speech input). Motivated by our formative study of speech referencing patterns, we characterize recurring ways people specify targets (Direct Attribute, Relational, Remembrance, and Chained) and ground them to our object-centric relational graph. Given an utterance, referent cues are parsed and rendered as persistent in-situ AR visual guidance, reducing iterative micro-guidance ("a bit more to the right", "now, stop.") during remote guidance. We demonstrate the use cases of our system with remote guided assistance and intent disambiguation scenarios. Our evaluation shows that Speechto-Spatial improves task efficiency, reduces cognitive load, and enhances usability compared to a conventional voice-only baseline, transforming disembodied verbal instruction into visually explainable, actionable guidance on a live shared view.