# Virtual, Augmented, and Mixed Reality for Human-Robot Interaction: A Survey and Virtual Design Element Taxonomy

arXiv: https://arxiv.org/abs/2202.11249 · HTML: https://mlsanity.com/paper/2202.11249/

## Key facts

- Cited by 5 papers in the map (data through 14 Oct 2026).
- Builds on 13 of its 112 references that are in the map.
- First posted on arXiv 23 Feb 2022.

## Builds on (most cited first, 13 of 13)

- [ORB-SLAM: a Versatile and Accurate Monocular SLAM System](https://mlsanity.com/paper/1502.00956/) (2015)
- [PoseCNN: A Convolutional Neural Network for 6D Object Pose Estimation in Cluttered Scenes](https://mlsanity.com/paper/1711.00199/) (2017)
- [SPECTER: Document-level Representation Learning using Citation-informed Transformers](https://mlsanity.com/paper/2004.07180/) (2020)
- [Baxter's Homunculus: Virtual Reality Spaces for Teleoperation in Manufacturing](https://mlsanity.com/paper/1703.01270/) (2017)
- [A VR System for Immersive Teleoperation and Live Exploration with a Mobile Robot](https://mlsanity.com/paper/1908.02949/) (2019)
- [Communicating Robot Arm Motion Intent Through Mixed Reality Head-mounted Displays](https://mlsanity.com/paper/1708.03655/) (2017)
- [Virtual-to-Real-World Transfer Learning for Robots on Wilderness Trails](https://mlsanity.com/paper/1901.05599/) (2019)
- [Using Socially Expressive Mixed Reality Arms for Enhancing Low-Expressivity Robots](https://mlsanity.com/paper/1911.09713/) (2019)
- [A Comparison of Visualisation Methods for Disambiguating Verbal Requests in Human-Robot Interaction](https://mlsanity.com/paper/1801.08760/) (2018)
- [DronePick: Object Picking and Delivery Teleoperation with the Drone Controlled by a Wearable Tactile Display](https://mlsanity.com/paper/1908.02432/) (2019)
- [Enabling Intuitive Human-Robot Teaming Using Augmented Reality and Gesture Control](https://mlsanity.com/paper/1909.06415/) (2019)
- [Augmented-Reality-Based Visualization of Navigation Data of Mobile Robots on the Microsoft Hololens -- Possibilities and Limitations](https://mlsanity.com/paper/1912.12109/) (2019)
- [Negotiation-based Human-Robot Collaboration via Augmented Reality](https://mlsanity.com/paper/1909.11227/) (2019)

## Built on by (most cited first, 5 of 5)

- [The Social Context of Human-Robot Interactions](https://mlsanity.com/paper/2508.13982/) (2025)
- [Enabling Robots to Identify Missing Steps in Robot Tasks for Guided Learning from Demonstration](https://mlsanity.com/paper/2311.18355/) (2023)
- [RAMPA: Robotic Augmented Reality for Machine Programming by DemonstrAtion](https://mlsanity.com/paper/2410.13412/) (2024)
- [MR-SLAM: Immersive Spatial Supervision for Multi-Robot Mapping via Mixed Reality](https://mlsanity.com/paper/2605.16432/) (2026)
- [Extended Reality as a Mediation Layer for Situated Human Control in Human-Robot Teaming](https://mlsanity.com/paper/2607.25047/) (2026)

## Abstract (arXiv)

Virtual, Augmented, and Mixed Reality for Human-Robot Interaction (VAM-HRI) has been gaining considerable attention in research in recent years. However, the HRI community lacks a set of shared terminology and framework for characterizing aspects of mixed reality interfaces, presenting serious problems for future research. Therefore, it is important to have a common set of terms and concepts that can be used to precisely describe and organize the diverse array of work being done within the field. In this paper, we present a novel taxonomic framework for different types of VAM-HRI interfaces, composed of four main categories of virtual design elements (VDEs). We present and justify our taxonomy and explain how its elements have been developed over the last 30 years as well as the current directions VAM-HRI is headed in the coming decade.