mlsanityData to 14 Oct 2026

How to Train Your Robot with Deep Reinforcement Learning; Lessons We've Learned

arXiv 2102.02915 ↗First posted 4 Feb 2021cs.RO cs.LGcited by 213

Not on the map. Why?

Cited by 57 in Deep reinforcement learning methods, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

Deep reinforcement learning (RL) has emerged as a promising approach for autonomously acquiring complex behaviors from low level sensor observations. Although a large portion of deep RL research has focused on applications in video games and simulated control, which does not connect with the constraints of learning in real environments, deep RL has also demonstrated promise in enabling physical robots to learn complex skills in the real world. At the same time,real world robotics provides an appealing domain for evaluating such algorithms, as it connects directly to how humans learn; as an embodied agent in the real world. Learning to perceive and move in the real world presents numerous challenges, some of which are easier to address than others, and some of which are often not considered in RL research that focuses only on simulated domains. In this review article, we present a number of case studies involving robotic deep RL. Building off of these case studies, we discuss commonly perceived challenges in deep RL and how they have been addressed in these works. We also provide an overview of other outstanding challenges, many of which are unique to the real-world robotics setting and are not often the focus of mainstream RL research. Our goal is to provide a resource both for roboticists and machine learning researchers who are interested in furthering the progress of deep RL in the real world.

Key facts. Cited by 213 papers in the map (data through 14 Oct 2026). Builds on 118 of its 171 references that are in the map. First posted on arXiv 4 Feb 2021.

Builds on

118 of 171 references.

References · 118

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,399
  2. 2017Proximal Policy Optimization Algorithmscited by 15,215
  3. 2017Model-Agnostic Meta-Learning for Fast Adaptation of Deep Networkscited by 4,897
  4. 2018Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actorcited by 4,885
  5. 2017Deep reinforcement learning from human preferencescited by 3,912
  6. 2015Continuous control with deep reinforcement learningcited by 3,407
  7. 2013Playing Atari with Deep Reinforcement Learningcited by 3,274
  8. 2015Trust Region Policy Optimizationcited by 2,580
  9. 2018Addressing Function Approximation Error in Actor-Critic Methodscited by 2,417
  10. 2016OpenAI Gymcited by 1,736
  11. 2017Domain Randomization for Transferring Deep Neural Networks from Simulation to the Real Worldcited by 1,498
  12. 2019Self-training with Noisy Student improves ImageNet classificationcited by 1,311
38 more
  1. 2018Off-Policy Deep Reinforcement Learning without Explorationcited by 1,241
  2. 2018Soft Actor-Critic Algorithms and Applicationscited by 1,240
  3. 2017Curiosity-driven Exploration by Self-supervised Predictioncited by 1,194
  4. 2017Hindsight Experience Replaycited by 1,136
  5. 2016End to End Learning for Self-Driving Carscited by 1,009
  6. 2018Exploration by Random Network Distillationcited by 993
  7. 2017Constrained Policy Optimizationcited by 983
  8. 2015End-to-End Training of Deep Visuomotor Policiescited by 875
  9. 2019Stabilizing Off-Policy Q-Learning via Bootstrapping Error Reductioncited by 840
  10. 2017GradNorm: Gradient Normalization for Adaptive Loss Balancing in Deep Multitask Networkscited by 834
  11. 2018Multi-Task Learning as Multi-Objective Optimizationcited by 821
  12. 2019Advantage-Weighted Regression: Simple and Scalable Off-Policy Reinforcement Learningcited by 782
  13. 2018QT-Opt: Scalable Deep Reinforcement Learning for Vision-Based Robotic Manipulationcited by 743
  14. 2017Sim-to-Real Transfer of Robotic Control with Dynamics Randomizationcited by 672
  15. 2017Reinforcement Learning with Deep Energy-Based Policiescited by 669
  16. 2016Unifying Count-Based Exploration and Intrinsic Motivationcited by 639
  17. 2018Exploring the Limits of Weakly Supervised Pretrainingcited by 605
  18. 2020Learning Quadrupedal Locomotion over Challenging Terraincited by 604
  19. 2016Deep Exploration via Bootstrapped DQNcited by 527
  20. 2019Learning agile and dynamic motor skills for legged robotscited by 504
  21. 2016RL2: Fast Reinforcement Learning via Slow Reinforcement Learningcited by 503
  22. 2017Time-Contrastive Networks: Self-Supervised Learning from Videocited by 459
  23. 2017Robust Adversarial Reinforcement Learningcited by 451
  24. 2015Policy Distillationcited by 440
  25. 2016Guided Cost Learning: Deep Inverse Optimal Control via Policy Optimizationcited by 406
  26. 2019Efficient Off-Policy Meta-Reinforcement Learning via Probabilistic Context Variablescited by 389
  27. 2019Learning by Cheatingcited by 383
  28. 2016Deep Reinforcement Learning for Robotic Manipulation with Asynchronous Off-Policy Updatescited by 353
  29. 2017Overcoming Exploration in Reinforcement Learning with Demonstrationscited by 340
  30. 2017Leveraging Demonstrations for Deep Reinforcement Learning on Robotics Problems with Sparse Rewardscited by 319
  31. 2018Sim-to-Real: Learning Agile Locomotion For Quadruped Robotscited by 319
  32. 2016Learning Hand-Eye Coordination for Robotic Grasping with Deep Learning and Large-Scale Data Collectioncited by 316
  33. 2016Unsupervised Pixel-Level Domain Adaptation with Generative Adversarial Networkscited by 308
  34. 2016Learning from Simulated and Unsupervised Images through Adversarial Trainingcited by 288
  35. 2016Deep Visual Foresight for Planning Robot Motioncited by 285
  36. 2018Learning to Adapt in Dynamic, Real-World Environments Through Meta-Reinforcement Learningcited by 276
  37. 2018Residual Reinforcement Learning for Robot Controlcited by 274
  38. 2019RoboNet: Large-Scale Multi-Robot Learningcited by 270

50 most cited of 118 All 118 (JSON)

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2021: 20; 2022: 25; 2023: 51; 2024: 59; 2025: 33; 2026: 25.

213 papers.

Sort:

By area:

  1. 2024Deep Reinforcement Learning for Robotics: A Survey of Real-World Successescited by 176
  2. 2025SimpleVLA-RL: Scaling VLA Training via Reinforcement Learningcited by 151
  3. 2023Learning a Diffusion Model Policy from Rewards via Q-Score Matchingcited by 138
  4. 2023Learning Agile Soccer Skills for a Bipedal Robot with Deep Reinforcement Learningcited by 133
  5. 2024Survey on Large Language Model-Enhanced Reinforcement Learning: Concept, Taxonomy, and Methodscited by 116
  6. 2024SafeAgentBench: A Benchmark for Safe Task Planning of Embodied LLM Agentscited by 116
  7. 2021MiniHack the Planet: A Sandbox for Open-Ended Reinforcement Learning Researchcited by 97
  8. 2024Aligning Large Language Models with Representation Editing: A Control Perspectivecited by 64
  9. 2023Datasets and Benchmarks for Offline Safe Reinforcement Learningcited by 59
  10. 2024Learning Multimodal Behaviors from Scratch with Diffusion Policy Gradientcited by 51
  11. 2024BadRobot: Jailbreaking Embodied LLM Agents in the Physical Worldcited by 43
  12. 2025Towards General-Purpose Model-Free Reinforcement Learningcited by 41
38 more
  1. 2024Towards Generalist Robot Learning from Internet Video: A Surveycited by 40
  2. 2023Continual Semantic Segmentation with Automatic Memory Sample Selectioncited by 39
  3. 2023World Models and Predictive Coding for Cognitive and Developmental Robotics: Frontiers and Challengescited by 36
  4. 2025Maximum Entropy Reinforcement Learning with Diffusion Policycited by 34
  5. 2025CarPlanner: Consistent Auto-regressive Trajectory Planning for Large-scale Reinforcement Learning in Autonomous Drivingcited by 34
  6. 2021Offline Decentralized Multi-Agent Reinforcement Learningcited by 33
  7. 2021SurRoL: An Open-source Reinforcement Learning Centered and dVRK Compatible Platform for Surgical Robot Learningcited by 31
  8. 2022MineRL Diamond 2021 Competition: Overview, Results, and Lessons Learnedcited by 26
  9. 2023Decision S4: Efficient Sequence-Based RL via State Spaces Layerscited by 26
  10. 2021Avoiding Catastrophe: Active Dendrites Enable Multi-Task Learning in Dynamic Environmentscited by 25
  11. 2023Grow Your Limits: Continuous Improvement with Real-World RL for Robotic Locomotioncited by 24
  12. 2021Vision-Guided Quadrupedal Locomotion in the Wild with Multi-Modal Delay Randomizationcited by 23
  13. 2023Demonstration-Guided Reinforcement Learning with Efficient Exploration for Task Automation of Surgical Robotcited by 23
  14. 2022Robot Learning of Mobile Manipulation with Reachability Behavior Priorscited by 22
  15. 2023Accelerating Reinforcement Learning of Robotic Manipulations via Feedback from Large Language Modelscited by 22
  16. 2021Optimization-Based Algebraic Multigrid Coarsening Using Reinforcement Learningcited by 21
  17. 2021Bayesian Controller Fusion: Leveraging Control Priors in Deep Reinforcement Learning for Roboticscited by 20
  18. 2022Learning Implicit Priors for Motion Optimizationcited by 20
  19. 2023Hierarchical Programmatic Reinforcement Learning via Learning to Compose Programscited by 20
  20. 2021Tactile Sim-to-Real Policy Transfer via Real-to-Sim Image Translationcited by 19
  21. 2021Learning from Demonstrations for Autonomous Soft-tissue Retractioncited by 18
  22. 2023MAHALO: Unifying Offline Reinforcement Learning and Imitation Learning from Observationscited by 18
  23. 2024FLD: Fourier Latent Dynamics for Structured Motion Representation and Learningcited by 18
  24. 2022Safe Reinforcement Learning of Dynamic High-Dimensional Robotic Tasks: Navigation, Manipulation, Interactioncited by 17
  25. 2024Learning Robot Soccer from Egocentric Vision with Deep Reinforcement Learningcited by 17
  26. 2024So You Think You Can Scale Up Autonomous Robot Data Collection?cited by 17
  27. 2023A Human-Centered Safe Robot Reinforcement Learning Framework with Interactive Behaviorscited by 15
  28. 2024General-purpose foundation models for increased autonomy in robot-assisted surgerycited by 15
  29. 2024Safe Reinforcement Learning on the Constraint Manifold: Theory and Applicationscited by 15
  30. 2023Games for Artificial Intelligence Research: A Review and Perspectivescited by 14
  31. 2025Optimal Task Order for Continual Learning of Multiple Taskscited by 14
  32. 2023Human-in-the-loop Embodied Intelligence with Interactive Simulation Environment for Surgical Robot Learningcited by 13
  33. 2023GraspLDM: Generative 6-DoF Grasp Synthesis using Latent Diffusion Modelscited by 13
  34. 2025Integrating Reinforcement Learning with Visual Generative Models: Foundations and Advancescited by 12
  35. 2024Failures Are Fated, But Can Be Faded: Characterizing and Mitigating Unwanted Behaviors in Large-Scale Vision and Language Modelscited by 11
  36. 2023Recent Advances of Deep Robotic Affordance Learning: A Reinforcement Learning Perspectivecited by 10
  37. 2023Learning a Universal Human Prior for Dexterous Manipulation from Human Preferencecited by 10
  38. 2023ReLU to the Rescue: Improve Your On-Policy Actor-Critic with Positive Advantagescited by 10

50 most cited of 213 All 213 (JSON)