mlsanityData to 7 Oct 2026

Reinforcement and Imitation Learning for Diverse Visuomotor Skills

arXiv 1802.09564 ↗First posted 26 Feb 2018cs.RO cs.AI cs.LGcited by 124

On the map: Offline RL and imitation learning from data → Robot manipulation imitation learning

Abstract (arXiv)

We propose a model-free deep reinforcement learning method that leverages a small amount of demonstration data to assist a reinforcement learning agent. We apply this approach to robotic manipulation tasks and train end-to-end visuomotor policies that map directly from RGB camera inputs to joint velocities. We demonstrate that our approach can solve a wide variety of visuomotor tasks, for which engineering a scripted controller would be laborious. In experiments, our reinforcement and imitation agent achieves significantly better performances than agents trained with reinforcement learning or imitation learning alone. We also illustrate that these policies, trained with large visual and dynamics variations, can achieve preliminary successes in zero-shot sim2real transfer. A brief visual description of this work can be viewed in https://youtu.be/EDl8SQUNjj0

Key facts. Cited by 124 papers in the map (data through 7 Oct 2026). Placed in Robot manipulation imitation learning, one of 13 directions in Offline RL and imitation learning from data. Builds on 40 of its 54 references that are in the map. First posted on arXiv 26 Feb 2018.

Builds on

40 of 54 references.

References · 40

  1. 2014Adam: A Method for Stochastic Optimizationcited by 36,960
  2. 2017Proximal Policy Optimization Algorithmscited by 14,441
  3. 2015Continuous control with deep reinforcement learningcited by 3,279
  4. 2015Trust Region Policy Optimizationcited by 2,503
  5. 2016Generative Adversarial Imitation Learningcited by 1,576
  6. 2017Domain Randomization for Transferring Deep Neural Networks from Simulation to the Real Worldcited by 1,401
  7. 2016Progressive Neural Networkscited by 1,332
  8. 2015End-to-End Training of Deep Visuomotor Policiescited by 847
  9. 2017Learning Complex Dexterous Manipulation with Deep Reinforcement Learning and Demonstrationscited by 685
  10. 2017Sim-to-Real Transfer of Robotic Control with Dynamics Randomizationcited by 616
  11. 2017Time-Contrastive Networks: Self-Supervised Learning from Videocited by 453
  12. 2016Guided Cost Learning: Deep Inverse Optimal Control via Policy Optimizationcited by 400
28 more
  1. 2016Reinforcement Learning with Unsupervised Auxiliary Taskscited by 400
  2. 2016Deep Reinforcement Learning for Robotic Manipulation with Asynchronous Off-Policy Updatescited by 342
  3. 2017Overcoming Exploration in Reinforcement Learning with Demonstrationscited by 322
  4. 2016Learning Hand-Eye Coordination for Robotic Grasping with Deep Learning and Large-Scale Data Collectioncited by 312
  5. 2017Leveraging Demonstrations for Deep Reinforcement Learning on Robotics Problems with Sparse Rewardscited by 302
  6. 2017One-Shot Imitation Learningcited by 232
  7. 2017One-Shot Visual Imitation Learning via Meta-Learningcited by 225
  8. 2015Supersizing Self-supervision: Learning to Grasp from 50K Tries and 700 Robot Hourscited by 208
  9. 2017Asymmetric Actor Critic for Image-Based Robot Learningcited by 202
  10. 2017Imitation from Observation: Learning to Imitate Behaviors from Raw Video via Context Translationcited by 167
  11. 2017Emergence of Locomotion Behaviours in Rich Environmentscited by 159
  12. 2015Learning Continuous Control Policies by Stochastic Value Gradientscited by 156
  13. 2016Continuous Deep Q-Learning with Model-based Accelerationcited by 149
  14. 2017Using Simulation and Domain Adaptation to Improve Efficiency of Deep Robotic Graspingcited by 138
  15. 2017Vision-Based Multi-Task Manipulation for Inexpensive Robots Using End-To-End Learning from Demonstrationcited by 100
  16. 2016Sim-to-Real Robot Learning from Pixels with Progressive Netscited by 89
  17. 2017Robust Imitation of Diverse Behaviorscited by 85
  18. 2017Learning human behaviors from motion capture by adversarial imitationcited by 75
  19. 2017Transferring End-to-End Visuomotor Control from Simulation to Real World for a Multi-Stage Taskcited by 66
  20. 2017Data-efficient Deep Reinforcement Learning for Dexterous Manipulationcited by 61
  21. 2016Learning and Transfer of Modulated Locomotor Controllerscited by 59
  22. 2016Learning Dexterous Manipulation for a Soft Robotic Hand from Human Demonstrationcited by 35
  23. 2016Learning Dexterous Manipulation Policies from Experience and Imitationcited by 32
  24. 2017Learning a visuomotor controller for real world robotic grasping using simulated depth imagescited by 29
  25. 2016Collective Robot Reinforcement Learning with Distributed Asynchronous Guided Policy Searchcited by 25
  26. 2016Path Integral Guided Policy Searchcited by 14
  27. 2017InfoGAIL: Interpretable Imitation Learning from Visual Demonstrationscited by 11
  28. 2017Reasoning About Liquids via Closed-Loop Simulationcited by 7

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2018: 3; 2019: 9; 2020: 10; 2021: 20; 2022: 25; 2023: 19; 2024: 18; 2025: 14; 2026: 6.

124 papers.

Sort:

By area:

  1. 2021A Minimalist Approach to Offline Reinforcement Learningcited by 882
  2. 2024RoboCasa: Large-Scale Simulation of Everyday Tasks for Generalist Robotscited by 391
  3. 2018Reward learning from human preferences and demonstrations in Ataricited by 314
  4. 2023MimicGen: A Data Generation System for Scalable Robot Learning using Human Demonstrationscited by 308
  5. 2022Behavior Transformers: Cloning k modes with one stonecited by 299
  6. 2023Cal-QL: Calibrated Offline RL Pre-Training for Efficient Online Fine-Tuningcited by 230
  7. 2018Discriminator-Actor-Critic: Addressing Sample Inefficiency and Reward Bias in Adversarial Imitation Learningcited by 203
  8. 2019A Divergence Minimization Perspective on Imitation Learning Methodscited by 188
  9. 2025VLA-RL: Towards Masterful and General Robotic Manipulation with Scalable Reinforcement Learningcited by 166
  10. 2023Foundation Models for Decision Making: Problems, Methods, and Opportunitiescited by 160
  11. 2019Goal-conditioned Imitation Learningcited by 159
  12. 2021The Surprising Effectiveness of Representation Learning for Visual Imitationcited by 149
38 more
  1. 2023Vision-Language Models as Success Detectorscited by 138
  2. 2020Human-in-the-Loop Imitation Learning using Remote Teleoperationcited by 103
  3. 2020Toward the Fundamental Limits of Imitation Learningcited by 96
  4. 2019Recent Advances in Imitation Learning from Observationcited by 92
  5. 2021Coarse-to-Fine Imitation Learning: Robot Manipulation from a Single Demonstrationcited by 92
  6. 2020State-Only Imitation Learning for Dexterous Manipulationcited by 90
  7. 2022DexVIP: Learning Dexterous Grasping with Human Hand Pose Priors from Videocited by 83
  8. 2021Demonstration-Guided Reinforcement Learning with Learned Skillscited by 73
  9. 2020Offline Learning from Demonstrations and Unlabeled Experiencecited by 72
  10. 2021Beyond Pick-and-Place: Tackling Robotic Stacking of Diverse Shapescited by 70
  11. 2023Imitating Task and Motion Planning with Visuomotor Transformerscited by 63
  12. 2021Imitation Learning: Progress, Taxonomies and Challengescited by 56
  13. 2024FLaRe: Achieving Masterful and Adaptive Robot Policies with Large-Scale Reinforcement Learning Fine-Tuningcited by 54
  14. 2024TRANSIC: Sim-to-Real Policy Transfer by Learning from Online Correctioncited by 50
  15. 2021Cross-Domain Imitation Learning via Optimal Transportcited by 49
  16. 2022Vision-Based Manipulators Need to Also See from Their Handscited by 46
  17. 2024RACER: Rich Language-Guided Failure Recovery Policies for Imitation Learningcited by 46
  18. 2019Task-Relevant Adversarial Imitation Learningcited by 45
  19. 2021Adversarial Skill Chaining for Long-Horizon Robot Manipulation via Terminal State Regularizationcited by 44
  20. 2025Sim-and-Real Co-Training: A Simple Recipe for Vision-Based Robotic Manipulationcited by 42
  21. 2025Sim-to-Real Reinforcement Learning for Vision-Based Dexterous Manipulation on Humanoidscited by 40
  22. 2024Diffusion Meets DAgger: Supercharging Eye-in-hand Imitation Learningcited by 34
  23. 2022Frame Mining: a Free Lunch for Learning Robotic Manipulation from 3D Point Cloudscited by 31
  24. 2020Imitation Learning with Sinkhorn Distancescited by 30
  25. 2022Chain of Thought Imitation with Procedure Cloningcited by 30
  26. 2023Human-in-the-Loop Task and Motion Planning for Imitation Learningcited by 28
  27. 2021Residual Reinforcement Learning from Demonstrationscited by 27
  28. 2021Robust Multi-Modal Policies for Industrial Assembly via Reinforcement Learning and Demonstrations: A Large-Scale Studycited by 24
  29. 2023Learning Realistic Traffic Agents in Closed-loopcited by 24
  30. 2021CRIL: Continual Robot Imitation Learning via Generative and Prediction Modelcited by 23
  31. 2020Semi-supervised reward learning for offline reinforcement learningcited by 22
  32. 2024Bootstrapping Reinforcement Learning with Imitation for Vision-Based Agile Flightcited by 22
  33. 2021iCurb: Imitation Learning-based Detection of Road Curbs using Aerial Images for Autonomous Drivingcited by 21
  34. 2023Demonstration-Guided Reinforcement Learning with Efficient Exploration for Task Automation of Surgical Robotcited by 21
  35. 2023One-Shot Imitation Learning: A Pose Estimation Perspectivecited by 21
  36. 2024Countering Reward Over-optimization in LLM with Demonstration-Guided Reinforcement Learningcited by 20
  37. 2025Physics-Driven Data Generation for Contact-Rich Manipulation via Trajectory Optimizationcited by 20
  38. 2019Scaling data-driven robotics with reward sketching and batch reinforcement learningcited by 18

50 most cited of 124 All 124 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2024Dreamitate: Real-World Visuomotor Policy Learning via Video Generation
  2. 2020Goal-Conditioned End-to-End Visuomotor Control for Versatile Skill Primitives
  3. 2019Self-Supervised Correspondence in Visuomotor Policy Learning
  4. 2025SViP: Sequencing Bimanual Visuomotor Policies with Object-Centric Motion Primitives
  5. 2025Latent Policy Barrier: Learning Robust Visuomotor Policies by Staying In-Distribution
  6. 2019Accept Synthetic Objects as Real: End-to-End Training of Attentive Deep Visuomotor Policies for Manipulation in Clutter
  7. 2026Spatially Conditioned Diffusion Policy: Learning Precise and Robust Manipulation with a Single RGB Camera
  8. 2025ImitDiff: Transferring Foundation-Model Priors for Distraction Robust Visuomotor Policy