# Error-Aware Imitation Learning from Teleoperation Data for Mobile Manipulation

arXiv: https://arxiv.org/abs/2112.05251 · HTML: https://mlsanity.com/paper/2112.05251/

## Key facts

- Cited by 59 papers in the map (data through 14 Oct 2026).
- Placed in Robot manipulation imitation learning, one of 13 directions in Offline RL and imitation learning from data.
- Builds on 25 of its 70 references that are in the map.
- First posted on arXiv 9 Dec 2021.

## Placed in

- Offline RL and imitation learning from data / [Robot manipulation imitation learning](https://mlsanity.com/offlinerl/robot-manipulation-imitation-learning/)

## Builds on (most cited first, 25 of 25)

- [Auto-Encoding Variational Bayes](https://mlsanity.com/paper/1312.6114/) (2013)
- [Deep Learning for Anomaly Detection: A Survey](https://mlsanity.com/paper/1901.03407/) (2019)
- [Deep Imitation Learning for Complex Manipulation Tasks from Virtual Reality Teleoperation](https://mlsanity.com/paper/1710.04615/) (2017)
- [ThreeDWorld: A Platform for Interactive Multi-Modal Physical Simulation](https://mlsanity.com/paper/2007.04954/) (2020)
- [RoboTurk: A Crowdsourcing Platform for Robotic Skill Learning through Imitation](https://mlsanity.com/paper/1811.02790/) (2018)
- [Can Autonomous Vehicles Identify, Recover From, and Adapt to Distribution Shifts?](https://mlsanity.com/paper/2006.14911/) (2020)
- [Learning to Generalize Across Long-Horizon Tasks from Human Demonstrations](https://mlsanity.com/paper/2003.06085/) (2020)
- [EnsembleDAgger: A Bayesian Approach to Safe Imitation Learning](https://mlsanity.com/paper/1807.08364/) (2018)
- [Interactive Gibson Benchmark (iGibson 0.5): A Benchmark for Interactive Navigation in Cluttered Environments](https://mlsanity.com/paper/1910.14442/) (2019)
- [Uncertainty-Aware Reinforcement Learning for Collision Avoidance](https://mlsanity.com/paper/1702.01182/) (2017)
- [HRL4IN: Hierarchical Reinforcement Learning for Interactive Navigation with Mobile Manipulators](https://mlsanity.com/paper/1910.11432/) (2019)
- [ReLMoGen: Leveraging Motion Generation in Reinforcement Learning for Mobile Manipulation](https://mlsanity.com/paper/2008.07792/) (2020)
- [Articulated Object Interaction in Unknown Scenes with Whole-Body Mobile Manipulation](https://mlsanity.com/paper/2103.10534/) (2021)
- [FFRob: Leveraging Symbolic Planning for Efficient Task and Motion Planning](https://mlsanity.com/paper/1608.01335/) (2016)
- [Learning Multi-Arm Manipulation Through Collaborative Teleoperation](https://mlsanity.com/paper/2012.06738/) (2020)
- [VOILA: Visual-Observation-Only Imitation Learning for Autonomous Navigation](https://mlsanity.com/paper/2105.09371/) (2021)
- [GONet: A Semi-Supervised Deep Learning Approach For Traversability Estimation](https://mlsanity.com/paper/1803.03254/) (2018)
- [VRKitchen: an Interactive 3D Virtual Environment for Task-oriented Learning](https://mlsanity.com/paper/1903.05757/) (2019)
- [Learning Kinematic Feasibility for Mobile Manipulation through Deep Reinforcement Learning](https://mlsanity.com/paper/2101.05325/) (2021)
- [Detect, Reject, Correct: Crossmodal Compensation of Corrupted Sensors](https://mlsanity.com/paper/2012.00201/) (2020)
- [Failure Prediction for Autonomous Driving](https://mlsanity.com/paper/1805.01811/) (2018)
- [The RobotriX: An eXtremely Photorealistic and Very-Large-Scale Indoor Dataset of Sequences with Robot Trajectories and Interactions](https://mlsanity.com/paper/1901.06514/) (2019)
- [Whole-Body MPC for a Dynamically Stable Mobile Manipulator](https://mlsanity.com/paper/1902.10415/) (2019)
- [Shared Multi-Task Imitation Learning for Indoor Self-Navigation](https://mlsanity.com/paper/1808.04503/) (2018)
- [Combined Task and Action Learning from Human Demonstrations for Mobile Manipulation Applications](https://mlsanity.com/paper/1908.10184/) (2019)

## Built on by (most cited first, 30 of 59)

- [Mobile ALOHA: Learning Bimanual Mobile Manipulation with Low-Cost Whole-Body Teleoperation](https://mlsanity.com/paper/2401.02117/) (2024)
- [MimicGen: A Data Generation System for Scalable Robot Learning using Human Demonstrations](https://mlsanity.com/paper/2310.17596/) (2023)
- [A Survey of Imitation Learning: Algorithms, Recent Developments, and Challenges](https://mlsanity.com/paper/2309.02473/) (2023)
- [Deep Whole-Body Control: Learning a Unified Policy for Manipulation and Locomotion](https://mlsanity.com/paper/2210.10044/) (2022)
- [DexMimicGen: Automated Data Generation for Bimanual Dexterous Manipulation via Imitation Learning](https://mlsanity.com/paper/2410.24185/) (2024)
- [SAFE: Multitask Failure Detection for Vision-Language-Action Models](https://mlsanity.com/paper/2506.09937/) (2025)
- [Learning and Retrieval from Prior Data for Skill-based Imitation Learning](https://mlsanity.com/paper/2210.11435/) (2022)
- [Imitating Task and Motion Planning with Visuomotor Transformers](https://mlsanity.com/paper/2305.16309/) (2023)
- [Visual Whole-Body Control for Legged Loco-Manipulation](https://mlsanity.com/paper/2403.16967/) (2024)
- [Deep Imitation Learning for Humanoid Loco-manipulation through Human Teleoperation](https://mlsanity.com/paper/2309.01952/) (2023)
- [SkillMimicGen: Automated Demonstration Generation for Efficient Skill Learning and Deployment](https://mlsanity.com/paper/2410.18907/) (2024)
- [Can We Detect Failures Without Failure Data? Uncertainty-Aware Runtime Failure Detection for Imitation Learning Policies](https://mlsanity.com/paper/2503.08558/) (2025)
- [Failure Prediction at Runtime for Generative Robot Policies](https://mlsanity.com/paper/2510.09459/) (2025)
- [Model-Based Runtime Monitoring with Interactive Imitation Learning](https://mlsanity.com/paper/2310.17552/) (2023)
- [Multi-Task Interactive Robot Fleet Learning with Visual World Models](https://mlsanity.com/paper/2410.22689/) (2024)
- [ACE: A Cross-Platform Visual-Exoskeletons System for Low-Cost Dexterous Teleoperation](https://mlsanity.com/paper/2408.11805/) (2024)
- [Diff-DAgger: Uncertainty Estimation with Diffusion Policy for Robotic Manipulation](https://mlsanity.com/paper/2410.14868/) (2024)
- [Uncertainty-aware Latent Safety Filters for Avoiding Out-of-Distribution Failures](https://mlsanity.com/paper/2505.00779/) (2025)
- [DynaMem: Online Dynamic Spatio-Semantic Memory for Open World Mobile Manipulation](https://mlsanity.com/paper/2411.04999/) (2024)
- [Adaptive Mobile Manipulation for Articulated Objects In the Open World](https://mlsanity.com/paper/2401.14403/) (2024)
- [Learning Generalizable Feature Fields for Mobile Manipulation](https://mlsanity.com/paper/2403.07563/) (2024)
- [TeleMoMa: A Modular and Versatile Teleoperation System for Mobile Manipulation](https://mlsanity.com/paper/2403.07869/) (2024)
- [Interactive Robot Learning from Verbal Correction](https://mlsanity.com/paper/2310.17555/) (2023)
- [SPIN: Simultaneous Perception, Interaction and Navigation](https://mlsanity.com/paper/2405.07991/) (2024)
- [M2Diffuser: Diffusion-based Trajectory Optimization for Mobile Manipulation in 3D Scenes](https://mlsanity.com/paper/2410.11402/) (2024)
- [IntervenGen: Interventional Data Generation for Robust and Data-Efficient Robot Imitation Learning](https://mlsanity.com/paper/2405.01472/) (2024)
- [AC-DiT: Adaptive Coordination Diffusion Transformer for Mobile Manipulation](https://mlsanity.com/paper/2507.01961/) (2025)
- [TidyBot++: An Open-Source Holonomic Mobile Manipulator for Robot Learning](https://mlsanity.com/paper/2412.10447/) (2024)
- [N2M2: Learning Navigation for Arbitrary Mobile Manipulation Motions in Unseen and Dynamic Environments](https://mlsanity.com/paper/2206.08737/) (2022)
- [Asking for Help: Failure Prediction in Behavioral Cloning through Value Approximation](https://mlsanity.com/paper/2302.04334/) (2023)

## Abstract (arXiv)

In mobile manipulation (MM), robots can both navigate within and interact with their environment and are thus able to complete many more tasks than robots only capable of navigation or manipulation. In this work, we explore how to apply imitation learning (IL) to learn continuous visuo-motor policies for MM tasks. Much prior work has shown that IL can train visuo-motor policies for either manipulation or navigation domains, but few works have applied IL to the MM domain. Doing this is challenging for two reasons: on the data side, current interfaces make collecting high-quality human demonstrations difficult, and on the learning side, policies trained on limited data can suffer from covariate shift when deployed. To address these problems, we first propose Mobile Manipulation RoboTurk (MoMaRT), a novel teleoperation framework allowing simultaneous navigation and manipulation of mobile manipulators, and collect a first-of-its-kind large scale dataset in a realistic simulated kitchen setting. We then propose a learned error detection system to address the covariate shift by detecting when an agent is in a potential failure state. We train performant IL policies and error detectors from this data, and achieve over 45% task success rate and 85% error detection success rate across multiple multi-stage tasks when trained on expert data. Codebase, datasets, visualization, and more available at https://sites.google.com/view/il-for-mm/home.