# Learning robot motor skills with mixed reality

arXiv: https://arxiv.org/abs/2203.11324 · HTML: https://mlsanity.com/paper/2203.11324/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 3 of its 13 references that are in the map.
- First posted on arXiv 21 Mar 2022.

## Builds on (most cited first, 3 of 3)

- [Deep Imitation Learning for Complex Manipulation Tasks from Virtual Reality Teleoperation](https://mlsanity.com/paper/1710.04615/) (2017)
- [PYROBOCOP : Python-based Robotic Control & Optimization Package for Manipulation and Collision Avoidance](https://mlsanity.com/paper/2106.03220/) (2021)
- [HAIR: Head-mounted AR Intention Recognition](https://mlsanity.com/paper/2102.11162/) (2021)

## Abstract (arXiv)

Mixed Reality (MR) has recently shown great success as an intuitive interface for enabling end-users to teach robots. Related works have used MR interfaces to communicate robot intents and beliefs to a co-located human, as well as developed algorithms for taking multi-modal human input and learning complex motor behaviors. Even with these successes, enabling end-users to teach robots complex motor tasks still poses a challenge because end-user communication is highly task dependent and world knowledge is highly varied. We propose a learning framework where end-users teach robots a) motion demonstrations, b) task constraints, c) planning representations, and d) object information, all of which are integrated into a single motor skill learning framework based on Dynamic Movement Primitives (DMPs). We hypothesize that conveying this world knowledge will be intuitive with an MR interface, and that a sample-efficient motor skill learning framework which incorporates varied modalities of world knowledge will enable robots to effectively solve complex tasks.