mlsanityData to 7 Oct 2026

Deep Reinforcement Learning for Robotic Manipulation with Asynchronous Off-Policy Updates

arXiv 1610.00633 ↗First posted 3 Oct 2016cs.RO cs.AI cs.LGcited by 342

Not on the map. Why?

Cited by 36 in Offline RL and imitation learning from data, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

Reinforcement learning holds the promise of enabling autonomous robots to learn large repertoires of behavioral skills with minimal human intervention. However, robotic applications of reinforcement learning often compromise the autonomy of the learning process in favor of achieving training times that are practical for real physical systems. This typically involves introducing hand-engineered policy representations and human-supplied demonstrations. Deep reinforcement learning alleviates this limitation by training general-purpose neural network policies, but applications of direct deep reinforcement learning algorithms have so far been restricted to simulated settings and relatively simple tasks, due to their apparent high sample complexity. In this paper, we demonstrate that a recent deep reinforcement learning algorithm based on off-policy training of deep Q-functions can scale to complex 3D manipulation tasks and can learn deep neural network policies efficiently enough to train on real physical robots. We demonstrate that the training times can be further reduced by parallelizing the algorithm across multiple robots which pool their policy updates asynchronously. Our experimental evaluation shows that our method can learn a variety of 3D manipulation skills in simulation and a complex door opening skill on real robots without any prior demonstrations or manually designed representations.

Key facts. Cited by 342 papers in the map (data through 7 Oct 2026). First posted on arXiv 3 Oct 2016.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2018: 2; 2019: 1; 2020: 7; 2021: 82; 2022: 69; 2023: 75; 2024: 46; 2025: 37; 2026: 23.

342 papers.

Sort:

By area:

  1. 2020Offline Reinforcement Learning: Tutorial, Review, and Perspectives on Open Problemscited by 1,873
  2. 2020D4RL: Datasets for Deep Data-Driven Reinforcement Learningcited by 1,406
  3. 2023Eureka: Human-Level Reward Design via Coding Large Language Modelscited by 542
  4. 2023Augmented Language Models: a Surveycited by 398
  5. 2021Learning to Walk in Minutes Using Massively Parallel Deep Reinforcement Learningcited by 395
  6. 2020Language Conditioned Imitation Learning over Unstructured Datacited by 208
  7. 2021How to Train Your Robot with Deep Reinforcement Learning; Lessons We've Learnedcited by 201
  8. 2018Reinforcement and Imitation Learning for Diverse Visuomotor Skillscited by 124
  9. 2021Benchmarks for Deep Off-Policy Evaluationcited by 89
  10. 2022Can Wikipedia Help Offline Reinforcement Learning?cited by 76
  11. 2022A Walk in the Park: Learning to Walk in 20 Minutes With Model-Free Reinforcement Learningcited by 76
  12. 2023Octopus: Embodied Vision-Language Programmer from Environmental Feedbackcited by 73
38 more
  1. 2021Asymmetric self-play for automatic goal discovery in robotic manipulationcited by 71
  2. 2019Multi-Agent Adversarial Inverse Reinforcement Learningcited by 60
  3. 2022Lipschitz-constrained Unsupervised Skill Discoverycited by 60
  4. 2021Learning to Synthesize Programs as Interpretable and Generalizable Policiescited by 59
  5. 2023Multi-Stage Cable Routing through Hierarchical Imitation Learningcited by 59
  6. 2020RL Unplugged: A Suite of Benchmarks for Offline Reinforcement Learningcited by 57
  7. 2023Enhancing Deep Reinforcement Learning: A Tutorial on Generative Diffusion Models in Network Optimizationcited by 56
  8. 2021A Survey of Machine Learning for Computer Architecture and Systemscited by 53
  9. 2021Memory-based Deep Reinforcement Learning for POMDPscited by 53
  10. 2021Autonomous Drone Racing with Deep Reinforcement Learningcited by 51
  11. 2022Look Closer: Bridging Egocentric and Third-Person Views with Transformers for Robotic Manipulationcited by 49
  12. 2025AutoEval: Autonomous Evaluation of Generalist Robot Manipulation Policies in the Real Worldcited by 46
  13. 2024Point Cloud Matters: Rethinking the Impact of Different Observation Spaces on Robot Learningcited by 45
  14. 2018DQN-TAMER: Human-in-the-Loop Reinforcement Learning with Intractable Feedbackcited by 39
  15. 2022Machine Learning in Aerodynamic Shape Optimizationcited by 39
  16. 2022Modular Lifelong Reinforcement Learning via Neural Compositioncited by 39
  17. 2025VLA-R1: Enhancing Reasoning in Vision-Language-Action Modelscited by 38
  18. 2021State Augmented Constrained Reinforcement Learning: Overcoming the Limitations of Learning with Rewardscited by 37
  19. 2022Skill-based Meta-Reinforcement Learningcited by 36
  20. 2023Trompt: Towards a Better Deep Neural Network for Tabular Datacited by 36
  21. 2023A Finite Sample Complexity Bound for Distributionally Robust Q-learningcited by 35
  22. 2024V-IRL: Grounding Virtual Intelligence in Real Lifecited by 35
  23. 2025Laminar: A Scalable Asynchronous RL Post-Training Frameworkcited by 35
  24. 2021Self-Supervised Online Reward Shaping in Sparse-Reward Environmentscited by 34
  25. 2021Lifelong Robotic Reinforcement Learning by Retaining Experiencescited by 34
  26. 2023Not Only Rewards But Also Constraints: Applications on Legged Robot Locomotioncited by 33
  27. 2021Autonomous Reinforcement Learning via Subgoal Curriculacited by 32
  28. 2022LEAGUE: Guided Skill Learning and Abstraction for Long-Horizon Manipulationcited by 32
  29. 2025Adversarial Attacks on Robotic Vision Language Action Modelscited by 31
  30. 2022Graph Lifelong Learning: A Surveycited by 30
  31. 2022RLx2: Training a Sparse Deep Reinforcement Learning Model from Scratchcited by 30
  32. 2024Understanding Scaling Laws with Statistical and Approximation Theory for Transformer Neural Networks on Intrinsically Low-dimensional Datacited by 30
  33. 2023Self-Improving Robots: End-to-End Autonomous Visuomotor Reinforcement Learningcited by 29
  34. 2021SurRoL: An Open-source Reinforcement Learning Centered and dVRK Compatible Platform for Surgical Robot Learningcited by 28
  35. 2021Finite-Sample Analysis of Off-Policy Natural Actor-Critic Algorithmcited by 26
  36. 2021FedAdapt: Adaptive Offloading for IoT Devices in Federated Learningcited by 26
  37. 2021Online Bootstrap Inference For Policy Evaluation in Reinforcement Learningcited by 26
  38. 2023Decision S4: Efficient Sequence-Based RL via State Spaces Layerscited by 26

50 most cited of 342 All 342 (JSON)