# Reality Copilot: Voice-First Human-AI Collaboration in Mixed Reality Using Large Multimodal Models

arXiv: https://arxiv.org/abs/2602.11025 · HTML: https://mlsanity.com/paper/2602.11025/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Placed in MLLM training and visual instruction tuning, one of 16 directions in Multimodal LLMs / vision-language models.
- Builds on 11 of its 30 references that are in the map.
- First posted on arXiv 11 Feb 2026.

## Placed in

- Multimodal LLMs / vision-language models / [MLLM training and visual instruction tuning](https://mlsanity.com/mllm/mllm-training-and-visual-instruction-tuning/)

## Builds on (most cited first, 11 of 11)

- [SAM 3: Segment Anything with Concepts](https://mlsanity.com/paper/2511.16719/) (2025)
- [A Survey on Multimodal Large Language Models for Autonomous Driving](https://mlsanity.com/paper/2311.12320/) (2023)
- [PixelLM: Pixel Reasoning with Large Multimodal Model](https://mlsanity.com/paper/2312.02228/) (2023)
- [TextMonkey: An OCR-Free Large Multimodal Model for Understanding Document](https://mlsanity.com/paper/2403.04473/) (2024)
- [3DALL-E: Integrating Text-to-Image AI in 3D Design Workflows](https://mlsanity.com/paper/2210.11603/) (2022)
- [Vidi: Large Multimodal Models for Video Understanding and Editing](https://mlsanity.com/paper/2504.15681/) (2025)
- [AIdeation: Designing a Human-AI Collaborative Ideation System for Concept Designers](https://mlsanity.com/paper/2502.14747/) (2025)
- [RealityTalk: Real-Time Speech-Driven Augmented Presentation for AR Live Storytelling](https://mlsanity.com/paper/2208.06350/) (2022)
- [LMM-Det: Make Large Multimodal Models Excel in Object Detection](https://mlsanity.com/paper/2507.18300/) (2025)
- [XRoboToolkit: A Cross-Platform Framework for Robot Teleoperation](https://mlsanity.com/paper/2508.00097/) (2025)
- [EmBARDiment: an Embodied AI Agent for Productivity in XR](https://mlsanity.com/paper/2408.08158/) (2024)

## Abstract (arXiv)

Large Multimodal Models (LMMs) have shown strong potential for assisting users in tasks, such as programming, content creation, and information access, yet their interaction remains largely limited to traditional interfaces such as desktops and smartphones. Meanwhile, advances in mixed reality (MR) hardware have enabled applications that extend beyond entertainment and into everyday use. However, most existing MR systems rely primarily on manual input (e.g., hand gestures or controllers) and provide limited intelligent assistance due to the lack of integration with large-scale AI models. We present Reality Copilot, a voice-first human-AI assistant for mixed reality that leverages LMMs to enable natural speech-based interaction. The system supports contextual understanding of physical environments, realistic 3D content generation, and real-time information retrieval. In addition to in-headset interaction, Reality Copilot facilitates cross-platform workflows by generating context-aware textual content and exporting generated assets. This work explores the design space of LMM-powered human-AI collaboration in mixed reality.