# Deep Imitation Learning for Complex Manipulation Tasks from Virtual Reality Teleoperation

arXiv: https://arxiv.org/abs/1710.04615 · HTML: https://mlsanity.com/paper/1710.04615/

## Key facts

- Cited by 289 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 12 Oct 2017.

## Built on by (most cited first, 30 of 289)

- [RT-1: Robotics Transformer for Real-World Control at Scale](https://mlsanity.com/paper/2212.06817/) (2022)
- [Learning Fine-Grained Bimanual Manipulation with Low-Cost Hardware](https://mlsanity.com/paper/2304.13705/) (2023)
- [GR00T N1: An Open Foundation Model for Generalist Humanoid Robots](https://mlsanity.com/paper/2503.14734/) (2025)
- [What Matters in Learning from Offline Human Demonstrations for Robot Manipulation](https://mlsanity.com/paper/2108.03298/) (2021)
- [BridgeData V2: A Dataset for Robot Learning at Scale](https://mlsanity.com/paper/2308.12952/) (2023)
- [BC-Z: Zero-Shot Task Generalization with Robotic Imitation Learning](https://mlsanity.com/paper/2202.02005/) (2022)
- [Implicit Behavioral Cloning](https://mlsanity.com/paper/2109.00137/) (2021)
- [RoboCasa: Large-Scale Simulation of Everyday Tasks for Generalist Robots](https://mlsanity.com/paper/2406.02523/) (2024)
- [Vision-Language Foundation Models as Effective Robot Imitators](https://mlsanity.com/paper/2311.01378/) (2023)
- [MimicGen: A Data Generation System for Scalable Robot Learning using Human Demonstrations](https://mlsanity.com/paper/2310.17596/) (2023)
- [Behavior Transformers: Cloning k modes with one stone](https://mlsanity.com/paper/2206.11251/) (2022)
- [PEBBLE: Feedback-Efficient Interactive Reinforcement Learning via Relabeling Experience and Unsupervised Pre-training](https://mlsanity.com/paper/2106.05091/) (2021)
- [Bridge Data: Boosting Generalization of Robotic Skills with Cross-Domain Datasets](https://mlsanity.com/paper/2109.13396/) (2021)
- [Affordances from Human Videos as a Versatile Representation for Robotics](https://mlsanity.com/paper/2304.08488/) (2023)
- [MimicPlay: Long-Horizon Imitation Learning by Watching Human Play](https://mlsanity.com/paper/2302.12422/) (2023)
- [Interactive Language: Talking to Robots in Real Time](https://mlsanity.com/paper/2210.06407/) (2022)
- [Language Conditioned Imitation Learning over Unstructured Data](https://mlsanity.com/paper/2005.07648/) (2020)
- [RH20T: A Comprehensive Robotic Dataset for Learning Diverse Skills in One-Shot](https://mlsanity.com/paper/2307.00595/) (2023)
- [RoboMIND: Benchmark on Multi-embodiment Intelligence Normative Data for Robot Manipulation](https://mlsanity.com/paper/2412.13877/) (2024)
- [Steering Your Diffusion Policy with Latent Space Reinforcement Learning](https://mlsanity.com/paper/2506.15799/) (2025)
- [Hand-Object Contact Consistency Reasoning for Human Grasps Generation](https://mlsanity.com/paper/2104.03304/) (2021)
- [DexCap: Scalable and Portable Mocap Data Collection System for Dexterous Manipulation](https://mlsanity.com/paper/2403.07788/) (2024)
- [The Surprising Effectiveness of Representation Learning for Visual Imitation](https://mlsanity.com/paper/2112.01511/) (2021)
- [AnyTeleop: A General Vision-Based Dexterous Robot Arm-Hand Teleoperation System](https://mlsanity.com/paper/2307.04577/) (2023)
- [Data Scaling Laws in Imitation Learning for Robotic Manipulation](https://mlsanity.com/paper/2410.18647/) (2024)
- [Getting aligned on representational alignment](https://mlsanity.com/paper/2310.13018/) (2023)
- [VIOLA: Imitation Learning for Vision-Based Manipulation with Object Proposal Priors](https://mlsanity.com/paper/2210.11339/) (2022)
- [Large Language Models for Robotics: A Survey](https://mlsanity.com/paper/2311.07226/) (2023)
- [THE COLOSSEUM: A Benchmark for Evaluating Generalization for Robotic Manipulation](https://mlsanity.com/paper/2402.08191/) (2024)
- [RoboTwin: Dual-Arm Robot Benchmark with Generative Digital Twins](https://mlsanity.com/paper/2504.13059/) (2025)

## Abstract (arXiv)

Imitation learning is a powerful paradigm for robot skill acquisition. However, obtaining demonstrations suitable for learning a policy that maps from raw pixels to actions can be challenging. In this paper we describe how consumer-grade Virtual Reality headsets and hand tracking hardware can be used to naturally teleoperate robots to perform complex tasks. We also describe how imitation learning can learn deep neural network policies (mapping from pixels to actions) that can acquire the demonstrated skills. Our experiments showcase the effectiveness of our approach for learning visuomotor skills.