# A 3D Mixed Reality Interface for Human-Robot Teaming

arXiv: https://arxiv.org/abs/2310.02392 · HTML: https://mlsanity.com/paper/2310.02392/

## Key facts

- Cited by 6 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 3 Oct 2023.

## Built on by (most cited first, 6 of 6)

- [VidBot: Learning Generalizable 3D Actions from In-the-Wild 2D Human Videos for Zero-Shot Robotic Manipulation](https://mlsanity.com/paper/2503.07135/) (2025)
- [FrontierNet: Learning Visual Cues to Explore](https://mlsanity.com/paper/2501.04597/) (2025)
- [Active Visual Localization for Multi-Agent Collaboration: A Data-Driven Approach](https://mlsanity.com/paper/2310.02650/) (2023)
- [ActLoc: Learning to Localize on the Move via Active Viewpoint Selection](https://mlsanity.com/paper/2508.20981/) (2025)
- [Sight Over Site: Perception-Aware Reinforcement Learning for Efficient Robotic Inspection](https://mlsanity.com/paper/2509.17877/) (2025)
- [fARfetch: Enabling Collocated AR-HRC in Large Visually Diverse Environments with VLM-Driven AR Content Adaptation](https://mlsanity.com/paper/2606.25162/) (2026)

## Abstract (arXiv)

This paper presents a mixed-reality human-robot teaming system. It allows human operators to see in real-time where robots are located, even if they are not in line of sight. The operator can also visualize the map that the robots create of their environment and can easily send robots to new goal positions. The system mainly consists of a mapping and a control module. The mapping module is a real-time multi-agent visual SLAM system that co-localizes all robots and mixed-reality devices to a common reference frame. Visualizations in the mixed-reality device then allow operators to see a virtual life-sized representation of the cumulative 3D map overlaid onto the real environment. As such, the operator can effectively "see through" walls into other rooms. To control robots and send them to new locations, we propose a drag-and-drop interface. An operator can grab any robot hologram in a 3D mini map and drag it to a new desired goal pose. We validate the proposed system through a user study and real-world deployments. We make the mixed-reality application publicly available at https://github.com/cvg/HoloLens_ros.