# Human-Robot Interaction in a Shared Augmented Reality Workspace

arXiv: https://arxiv.org/abs/2007.12656 · HTML: https://mlsanity.com/paper/2007.12656/

## Key facts

- Cited by 10 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 24 Jul 2020.

## Built on by (most cited first, 10 of 10)

- [YouRefIt: Embodied Reference Understanding with Language and Gesture](https://mlsanity.com/paper/2109.03413/) (2021)
- [Augmented Reality and Robotics: A Survey and Taxonomy for AR-enhanced Human-Robot Interaction and Robotic Interfaces](https://mlsanity.com/paper/2203.03254/) (2022)
- [Spatial and Visual Perspective-Taking via View Rotation and Relation Reasoning for Embodied Reference Understanding](https://mlsanity.com/paper/2309.01073/) (2023)
- [STRAP: Structured Object Affordance Segmentation with Point Supervision](https://mlsanity.com/paper/2304.08492/) (2023)
- [On the Emergence of Symmetrical Reality](https://mlsanity.com/paper/2401.15132/) (2024)
- [Beyond Object Categories: Multi-Attribute Reference Understanding for Visual Grounding](https://mlsanity.com/paper/2503.19240/) (2025)
- [AD-DINO: Attention-Dynamic DINO for Distance-Aware Embodied Reference Understanding](https://mlsanity.com/paper/2411.08451/) (2024)
- [Learning Visualization Policies of Augmented Reality for Human-Robot Collaboration](https://mlsanity.com/paper/2211.07028/) (2022)
- [Self-supervised 6-DoF Robot Grasping by Demonstration via Augmented Reality Teleoperation System](https://mlsanity.com/paper/2404.03067/) (2024)
- [What Do You Mean? Exploring How Humans and AI Interact with Symbols and Meanings in Their Interactions](https://mlsanity.com/paper/2510.05378/) (2025)

## Abstract (arXiv)

We design and develop a new shared Augmented Reality (AR) workspace for Human-Robot Interaction (HRI), which establishes a bi-directional communication between human agents and robots. In a prototype system, the shared AR workspace enables a shared perception, so that a physical robot not only perceives the virtual elements in its own view but also infers the utility of the human agent--the cost needed to perceive and interact in AR--by sensing the human agent's gaze and pose. Such a new HRI design also affords a shared manipulation, wherein the physical robot can control and alter virtual objects in AR as an active agent; crucially, a robot can proactively interact with human agents, instead of purely passively executing received commands. In experiments, we design a resource collection game that qualitatively demonstrates how a robot perceives, processes, and manipulates in AR and quantitatively evaluates the efficacy of HRI using the shared AR workspace. We further discuss how the system can potentially benefit future HRI studies that are otherwise challenging.