# Extended Reality System for Robotic Learning from Human Demonstration

arXiv: https://arxiv.org/abs/2409.12862 · HTML: https://mlsanity.com/paper/2409.12862/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Placed in Robot manipulation imitation learning, one of 13 directions in Offline RL and imitation learning from data.
- Builds on 11 of its 29 references that are in the map.
- First posted on arXiv 19 Sep 2024.

## Placed in

- Offline RL and imitation learning from data / [Robot manipulation imitation learning](https://mlsanity.com/offlinerl/robot-manipulation-imitation-learning/)

## Builds on (most cited first, 11 of 11)

- [Behavioral Cloning from Observation](https://mlsanity.com/paper/1805.01954/) (2018)
- [Implicit Behavioral Cloning](https://mlsanity.com/paper/2109.00137/) (2021)
- [A Survey of Inverse Reinforcement Learning: Challenges, Methods and Progress](https://mlsanity.com/paper/1806.06877/) (2018)
- [Robot Operating System 2: Design, Architecture, and Uses In The Wild](https://mlsanity.com/paper/2211.07752/) (2022)
- [OPEN TEACH: A Versatile Teleoperation System for Robotic Manipulation](https://mlsanity.com/paper/2403.07870/) (2024)
- [The Effect of Modeling Human Rationality Level on Learning Rewards from Multiple Feedback Types](https://mlsanity.com/paper/2208.10687/) (2022)
- [Feature Expansive Reward Learning: Rethinking Human Input](https://mlsanity.com/paper/2006.13208/) (2020)
- [Augmented Reality and Robotics: A Survey and Taxonomy for AR-enhanced Human-Robot Interaction and Robotic Interfaces](https://mlsanity.com/paper/2203.03254/) (2022)
- [Ergodic imitation: Learning from what to do and what not to do](https://mlsanity.com/paper/2103.17098/) (2021)
- [Communicating Robot Arm Motion Intent Through Mixed Reality Head-mounted Displays](https://mlsanity.com/paper/1708.03655/) (2017)
- [SliceIt! -- A Dual Simulator Framework for Learning Robot Food Slicing](https://mlsanity.com/paper/2404.02569/) (2024)

## Abstract (arXiv)

Many real-world tasks are intuitive for a human to perform, but difficult to encode algorithmically when utilizing a robot to perform the tasks. In these scenarios, robotic systems can benefit from expert demonstrations to learn how to perform each task. In many settings, it may be difficult or unsafe to use a physical robot to provide these demonstrations, for example, considering cooking tasks such as slicing with a knife. Extended reality provides a natural setting for demonstrating robotic trajectories while bypassing safety concerns and providing a broader range of interaction modalities. We propose the Robot Action Demonstration in Extended Reality (RADER) system, a generic extended reality interface for learning from demonstration. We additionally present its application to an existing state-of-the-art learning from demonstration approach and show comparable results between demonstrations given on a physical robot and those given using our extended reality system.