mlsanityData to 7 Oct 2026

What Matters in Learning from Offline Human Demonstrations for Robot Manipulation

arXiv 2108.03298 ↗First posted 6 Aug 2021cs.RO cs.AI cs.LGcited by 699

On the map: Offline RL and imitation learning from data → Real-world robot offline RL

Abstract (arXiv)

Imitating human demonstrations is a promising approach to endow robots with various manipulation capabilities. While recent advances have been made in imitation learning and batch (offline) reinforcement learning, a lack of open-source human datasets and reproducible learning methods make assessing the state of the field difficult. In this paper, we conduct an extensive study of six offline learning algorithms for robot manipulation on five simulated and three real-world multi-stage manipulation tasks of varying complexity, and with datasets of varying quality. Our study analyzes the most critical challenges when learning from offline human data for manipulation. Based on the study, we derive a series of lessons including the sensitivity to different algorithmic design choices, the dependence on the quality of the demonstrations, and the variability based on the stopping criteria due to the different objectives in training and evaluation. We also highlight opportunities for learning from human datasets, such as the ability to learn proficient policies on challenging, multi-stage tasks beyond the scope of current reinforcement learning methods, and the ability to easily scale to natural, real-world manipulation scenarios where only raw sensory signals are available. We have open-sourced our datasets and all algorithm implementations to facilitate future research and fair comparisons in learning from human demonstration data. Codebase, datasets, trained models, and more available at https://arise-initiative.github.io/robomimic-web/

Key facts. Cited by 699 papers in the map (data through 7 Oct 2026). Placed in Real-world robot offline RL, one of 13 directions in Offline RL and imitation learning from data. Builds on 62 of its 90 references that are in the map. First posted on arXiv 6 Aug 2021.

Builds on

62 of 90 references.

References · 62

  1. 2015Deep Residual Learning for Image Recognitioncited by 51,809
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  3. 2014Adam: A Method for Stochastic Optimizationcited by 36,960
  4. 2015U-Net: Convolutional Networks for Biomedical Image Segmentationcited by 19,604
  5. 2015You Only Look Once: Unified, Real-Time Object Detectioncited by 4,829
  6. 2018Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actorcited by 4,635
  7. 2013Auto-Encoding Variational Bayescited by 3,980
  8. 2017Deep reinforcement learning from human preferencescited by 3,846
  9. 2015Continuous control with deep reinforcement learningcited by 3,279
  10. 2020Offline Reinforcement Learning: Tutorial, Review, and Perspectives on Open Problemscited by 1,873
  11. 2020Conservative Q-Learning for Offline Reinforcement Learningcited by 1,777
  12. 2018Know What You Don't Know: Unanswerable Questions for SQuADcited by 1,538
38 more
  1. 2021Decision Transformer: Reinforcement Learning via Sequence Modelingcited by 1,528
  2. 2020D4RL: Datasets for Deep Data-Driven Reinforcement Learningcited by 1,406
  3. 2018Off-Policy Deep Reinforcement Learning without Explorationcited by 1,227
  4. 2019Meta-World: A Benchmark and Evaluation for Multi-Task and Meta Reinforcement Learningcited by 1,168
  5. 2017Deep Reinforcement Learning that Matterscited by 727
  6. 2019RLBench: The Robot Learning Benchmark & Learning Environmentcited by 669
  7. 2019Behavior Regularized Offline Reinforcement Learningcited by 654
  8. 2020MOPO: Model-based Offline Policy Optimizationcited by 630
  9. 2020MOReL : Model-Based Offline Reinforcement Learningcited by 582
  10. 2020Image Augmentation Is All You Need: Regularizing Deep Reinforcement Learning from Pixelscited by 573
  11. 2020Learning Quadrupedal Locomotion over Challenging Terraincited by 494
  12. 2020robosuite: A Modular Simulation Framework and Benchmark for Robot Learningcited by 484
  13. 2020Reinforcement Learning with Augmented Datacited by 452
  14. 2021COMBO: Conservative Offline Model-Based Policy Optimizationcited by 365
  15. 2016Benchmarking Deep Reinforcement Learning for Continuous Controlcited by 361
  16. 2020Transporter Networks: Rearranging the Visual World for Robotic Manipulationcited by 342
  17. 2017Deep Imitation Learning for Complex Manipulation Tasks from Virtual Reality Teleoperationcited by 286
  18. 2020Critic Regularized Regressioncited by 269
  19. 2017One-Shot Visual Imitation Learning via Meta-Learningcited by 225
  20. 2020Accelerating Reinforcement Learning with Learned Skill Priorscited by 220
  21. 2020Keep Doing What Worked: Behavioral Modelling Priors for Offline Reinforcement Learningcited by 213
  22. 2018RoboTurk: A Crowdsourcing Platform for Robotic Skill Learning through Imitationcited by 202
  23. 2020OPAL: Offline Primitive Discovery for Accelerating Offline Reinforcement Learningcited by 164
  24. 2019Benchmarking Model-Based Reinforcement Learningcited by 158
  25. 2020What Matters In On-Policy Reinforcement Learning? A Large-Scale Empirical Studycited by 157
  26. 2017Interactive Learning from Policy-Dependent Human Feedbackcited by 155
  27. 2015Deep Spatial Autoencoders for Visuomotor Learningcited by 149
  28. 2020Acme: A Research Framework for Distributed Reinforcement Learningcited by 146
  29. 2020SoftGym: Benchmarking Deep Reinforcement Learning for Deformable Object Manipulationcited by 118
  30. 2019End-to-End Robotic Reinforcement Learning without Reward Engineeringcited by 115
  31. 2020Learning to Generalize Across Long-Horizon Tasks from Human Demonstrationscited by 114
  32. 2020Hyperparameter Selection for Offline Reinforcement Learningcited by 113
  33. 2020EMaQ: Expected-Max Q-Learning Operator for Simple Yet Effective Offline and Online RLcited by 113
  34. 2020Visual Imitation Made Easycited by 111
  35. 2020CausalWorld: A Robotic Manipulation Benchmark for Causal Structure and Transfer Learningcited by 109
  36. 2019Self-Supervised Correspondence in Visuomotor Policy Learningcited by 106
  37. 2020Human-in-the-Loop Imitation Learning using Remote Teleoperationcited by 103
  38. 2021Representation Matters: Offline Pretraining for Sequential Decision Makingcited by 90

50 most cited of 62 All 62 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 6; 2022: 48; 2023: 84; 2024: 144; 2025: 193; 2026: 224.

699 papers.

Sort:

By area:

  1. 2023Learning Fine-Grained Bimanual Manipulation with Low-Cost Hardwarecited by 1,490
  2. 2023LIBERO: Benchmarking Knowledge Transfer for Lifelong Robot Learningcited by 1,338
  3. 2025GR00T N1: An Open Foundation Model for Generalist Humanoid Robotscited by 1,167
  4. 2023BridgeData V2: A Dataset for Robot Learning at Scalecited by 634
  5. 2022VIMA: General Robot Manipulation with Multimodal Promptscited by 403
  6. 2024RoboCasa: Large-Scale Simulation of Everyday Tasks for Generalist Robotscited by 391
  7. 2022VIP: Towards Universal Visual Reward and Representation via Value-Implicit Pre-Trainingcited by 384
  8. 2024Mobile ALOHA: Learning Bimanual Mobile Manipulation with Low-Cost Whole-Body Teleoperationcited by 377
  9. 2024One Step Diffusion via Shortcut Modelscited by 373
  10. 2023MimicGen: A Data Generation System for Scalable Robot Learning using Human Demonstrationscited by 308
  11. 2022Behavior Transformers: Cloning k modes with one stonecited by 299
  12. 2024Diffusion Policy Policy Optimizationcited by 255
38 more
  1. 2023Orbit: A Unified Simulation Framework for Interactive Robot Learning Environmentscited by 239
  2. 2023Affordances from Human Videos as a Versatile Representation for Roboticscited by 221
  3. 2024A Survey on 3D Gaussian Splattingcited by 221
  4. 2025Unified World Models: Coupling Video and Action Diffusion for Pretraining on Large Robotic Datasetscited by 218
  5. 2022Interactive Language: Talking to Robots in Real Timecited by 217
  6. 2024ManiSkill3: GPU Parallelized Robotics Simulation and Rendering for Generalizable Embodied AIcited by 216
  7. 2023MimicPlay: Long-Horizon Imitation Learning by Watching Human Playcited by 214
  8. 2023Any-point Trajectory Modeling for Policy Learningcited by 207
  9. 2024RT-H: Action Hierarchies Using Languagecited by 204
  10. 2025Unified Video Action Modelcited by 195
  11. 2021RvS: What is Essential for Offline RL via Supervised Learning?cited by 184
  12. 2023LIV: Language-Image Representations and Rewards for Robotic Controlcited by 180
  13. 2023RoboAgent: Generalization and Efficiency in Robot Manipulation via Semantic Augmentations and Action Chunkingcited by 167
  14. 2023Language-Driven Representation Learning for Roboticscited by 163
  15. 2025Flow Q-Learningcited by 162
  16. 2024EgoMimic: Scaling Imitation Learning via Egocentric Videocited by 160
  17. 2024Consistency Policy: Accelerated Visuomotor Policies via Consistency Distillationcited by 159
  18. 2024DexCap: Scalable and Portable Mocap Data Collection System for Dexterous Manipulationcited by 149
  19. 2025Steering Your Diffusion Policy with Latent Space Reinforcement Learningcited by 149
  20. 2024Scaling Proprioceptive-Visual Learning with Heterogeneous Pre-trained Transformerscited by 145
  21. 2025SimpleVLA-RL: Scaling VLA Training via Reinforcement Learningcited by 141
  22. 2024Open-TeleVision: Teleoperation with Immersive Active Visual Feedbackcited by 137
  23. 2022VIOLA: Imitation Learning for Vision-Based Manipulation with Object Proposal Priorscited by 135
  24. 2024Reconciling Reality through Simulation: A Real-to-Sim-to-Real Approach for Robust Manipulationcited by 135
  25. 2023Large Language Models for Robotics: A Surveycited by 132
  26. 2023RT-Trajectory: Robotic Task Generalization via Hindsight Trajectory Sketchescited by 131
  27. 2023Q-Transformer: Scalable Offline Reinforcement Learning via Autoregressive Q-Functionscited by 129
  28. 2022Pessimistic Bootstrapping for Uncertainty-Driven Offline Reinforcement Learningcited by 125
  29. 2025HAMSTER: Hierarchical Action Models For Open-World Robot Manipulationcited by 125
  30. 2025Spatial Forcing: Implicit Spatial Representation Alignment for Vision-language-action Modelcited by 125
  31. 2025On the Generalization of SFT: A Reinforcement Learning Perspective with Reward Rectificationcited by 121
  32. 2022Habitat-Web: Learning Embodied Object-Search Strategies from Human Demonstrations at Scalecited by 116
  33. 20243D Diffusion Policy: Generalizable Visuomotor Policy Learning via Simple 3D Representationscited by 115
  34. 2024Survey on Large Language Model-Enhanced Reinforcement Learning: Concept, Taxonomy, and Methodscited by 114
  35. 2024DexMimicGen: Automated Data Generation for Bimanual Dexterous Manipulation via Imitation Learningcited by 104
  36. 2022Robot Learning on the Job: Human-in-the-Loop Autonomy and Learning During Deploymentcited by 101
  37. 2023On Bringing Robots Homecited by 101
  38. 2024HumanoidBench: Simulated Humanoid Benchmark for Whole-Body Locomotion and Manipulationcited by 101

50 most cited of 699 All 699 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2020Offline Learning from Demonstrations and Unlabeled Experience
  2. 2025Robust Offline Imitation Learning Through State-level Trajectory Stitching
  3. 2023Offline Skill Generalization via Task and Motion Planning
  4. 2021Efficient Robotic Manipulation Through Offline-to-Online Reinforcement Learning and Goal-Aware State Information
  5. 2025RM-RL: Role-Model Reinforcement Learning for Precise Robot Manipulation
  6. 2026Human-to-Robot Interaction: Learning from Video Demonstration for Robot Imitation
  7. 2025Learning What to Do and What Not To Do: Offline Imitation from Expert and Undesirable Demonstrations
  8. 2022Implicit Offline Reinforcement Learning via Supervised Learning