# Deep Reinforcement Learning for Vision-Based Robotic Grasping: A Simulated Comparative Evaluation of Off-Policy Methods

arXiv: https://arxiv.org/abs/1802.10264 · HTML: https://mlsanity.com/paper/1802.10264/

## Key facts

- Cited by 33 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 28 Feb 2018.

## Built on by (most cited first, 30 of 33)

- [DexVIP: Learning Dexterous Grasping with Human Hand Pose Priors from Video](https://mlsanity.com/paper/2202.00164/) (2022)
- [Variance-Aware Off-Policy Evaluation with Linear Function Approximation](https://mlsanity.com/paper/2106.11960/) (2021)
- [Learning High-DOF Reaching-and-Grasping via Dynamic Representation of Gripper-Object Interaction](https://mlsanity.com/paper/2204.13998/) (2022)
- [InterACT: Inter-dependency Aware Action Chunking with Hierarchical Attention Transformers for Bimanual Manipulation](https://mlsanity.com/paper/2409.07914/) (2024)
- [An Instrumental Variable Approach to Confounded Off-Policy Evaluation](https://mlsanity.com/paper/2212.14468/) (2022)
- [PFRL: Pose-Free Reinforcement Learning for 6D Pose Estimation](https://mlsanity.com/paper/2102.12096/) (2021)
- [Accelerating Interactive Human-like Manipulation Learning with GPU-based Simulation and High-quality Demonstrations](https://mlsanity.com/paper/2212.02126/) (2022)
- [Meta-Learning Regrasping Strategies for Physical-Agnostic Objects](https://mlsanity.com/paper/2205.11110/) (2022)
- [Learning from Ambiguous Demonstrations with Self-Explanation Guided Reinforcement Learning](https://mlsanity.com/paper/2110.05286/) (2021)
- [Minimizing Human Assistance: Augmenting a Single Demonstration for Deep Reinforcement Learning](https://mlsanity.com/paper/2209.11275/) (2022)
- [RMBench: Benchmarking Deep Reinforcement Learning for Robotic Manipulator Control](https://mlsanity.com/paper/2210.11262/) (2022)
- [Model-based Offline Reinforcement Learning with Local Misspecification](https://mlsanity.com/paper/2301.11426/) (2023)
- [Stochastic Q-learning for Large Discrete Action Spaces](https://mlsanity.com/paper/2405.10310/) (2024)
- [Visionary: Vision architecture discovery for robot learning](https://mlsanity.com/paper/2103.14633/) (2021)
- [Depth-aware Object Segmentation and Grasp Detection for Robotic Picking Tasks](https://mlsanity.com/paper/2111.11114/) (2021)
- [Grasp Anything: Combining Teacher-Augmented Policy Gradient Learning with Instance Segmentation to Grasp Arbitrary Objects](https://mlsanity.com/paper/2403.10187/) (2024)
- [End-to-end Reinforcement Learning of Robotic Manipulation with Robust Keypoints Representation](https://mlsanity.com/paper/2202.06027/) (2022)
- [CoGrasp: 6-DoF Grasp Generation for Human-Robot Collaboration](https://mlsanity.com/paper/2210.03173/) (2022)
- [Efficient Representations of Object Geometry for Reinforcement Learning of Interactive Grasping Policies](https://mlsanity.com/paper/2211.10957/) (2022)
- [Towards Real-World Efficiency: Domain Randomization in Reinforcement Learning for Pre-Capture of Free-Floating Moving Targets by Autonomous Robots](https://mlsanity.com/paper/2406.06460/) (2024)
- [Multi-Type Preference Learning: Empowering Preference-Based Reinforcement Learning with Equal Preferences](https://mlsanity.com/paper/2409.07268/) (2024)
- [Training and Evaluation of Deep Policies using Reinforcement Learning and Generative Models](https://mlsanity.com/paper/2204.08573/) (2022)
- [Improving Wind Resistance Performance of Cascaded PID Controlled Quadcopters using Residual Reinforcement Learning](https://mlsanity.com/paper/2308.01648/) (2023)
- [MSPR: Multi-scale Predictive Representations for Goal-conditioned Reinforcement Learning](https://mlsanity.com/paper/2605.09364/) (2026)
- [Optimality Inductive Biases and Agnostic Guidelines for Offline Reinforcement Learning](https://mlsanity.com/paper/2107.01407/) (2021)
- [Benchmarking Deep Reinforcement Learning Algorithms for Vision-based Robotics](https://mlsanity.com/paper/2201.04224/) (2022)
- [GloCAL: Glocalized Curriculum-Aided Learning of Multiple Tasks with Application to Robotic Grasping](https://mlsanity.com/paper/2204.06835/) (2022)
- [A Secure and Efficient Multi-Object Grasping Detection Approach for Robotic Arms](https://mlsanity.com/paper/2209.03511/) (2022)
- [CRC-RL: A Novel Visual Feature Representation Architecture for Unsupervised Reinforcement Learning](https://mlsanity.com/paper/2301.13473/) (2023)
- [The Role of Tactile Sensing for Learning Reach and Grasp](https://mlsanity.com/paper/2502.20367/) (2025)

## Abstract (arXiv)

In this paper, we explore deep reinforcement learning algorithms for vision-based robotic grasping. Model-free deep reinforcement learning (RL) has been successfully applied to a range of challenging environments, but the proliferation of algorithms makes it difficult to discern which particular approach would be best suited for a rich, diverse task like grasping. To answer this question, we propose a simulated benchmark for robotic grasping that emphasizes off-policy learning and generalization to unseen objects. Off-policy learning enables utilization of grasping data over a wide variety of objects, and diversity is important to enable the method to generalize to new objects that were not seen during training. We evaluate the benchmark tasks against a variety of Q-function estimation methods, a method previously proposed for robotic grasping with deep neural network models, and a novel approach based on a combination of Monte Carlo return estimation and an off-policy correction. Our results indicate that several simple methods provide a surprisingly strong competitor to popular algorithms such as double Q-learning, and our analysis of stability sheds light on the relative tradeoffs between the algorithms.