# Learning a visuomotor controller for real world robotic grasping using simulated depth images

arXiv: https://arxiv.org/abs/1706.04652 · HTML: https://mlsanity.com/paper/1706.04652/

## Key facts

- Cited by 29 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 14 Jun 2017.

## Built on by (most cited first, 29 of 29)

- [RT-1: Robotics Transformer for Real-World Control at Scale](https://mlsanity.com/paper/2212.06817/) (2022)
- [How to Train Your Robot with Deep Reinforcement Learning; Lessons We've Learned](https://mlsanity.com/paper/2102.02915/) (2021)
- [Reinforcement and Imitation Learning for Diverse Visuomotor Skills](https://mlsanity.com/paper/1802.09564/) (2018)
- [You Only Demonstrate Once: Category-Level Manipulation from Single Visual Demonstration](https://mlsanity.com/paper/2201.12716/) (2022)
- [Scalable Scene Flow from Point Clouds in the Real World](https://mlsanity.com/paper/2103.01306/) (2021)
- [Learning High-DOF Reaching-and-Grasping via Dynamic Representation of Gripper-Object Interaction](https://mlsanity.com/paper/2204.13998/) (2022)
- [Learning Depth via Leveraging Semantics: Self-supervised Monocular Depth Estimation with Both Implicit and Explicit Semantic Guidance](https://mlsanity.com/paper/2102.06685/) (2021)
- [Vision-driven Compliant Manipulation for Reliable, High-Precision Assembly Tasks](https://mlsanity.com/paper/2106.14070/) (2021)
- [Receding Horizon Inverse Reinforcement Learning](https://mlsanity.com/paper/2206.04477/) (2022)
- [GKNet: grasp keypoint network for grasp candidates detection](https://mlsanity.com/paper/2106.08497/) (2021)
- [GraspARL: Dynamic Grasping via Adversarial Reinforcement Learning](https://mlsanity.com/paper/2203.02119/) (2022)
- [Ab Initio Particle-based Object Manipulation](https://mlsanity.com/paper/2107.08865/) (2021)
- [Towards Disturbance-Free Visual Mobile Manipulation](https://mlsanity.com/paper/2112.12612/) (2021)
- [Zero-shot Depth Completion via Test-time Alignment with Affine-invariant Depth Prior](https://mlsanity.com/paper/2502.06338/) (2025)
- [Simulation-based Bayesian inference for multi-fingered robotic grasping](https://mlsanity.com/paper/2109.14275/) (2021)
- [Visionary: Vision architecture discovery for robot learning](https://mlsanity.com/paper/2103.14633/) (2021)
- [Grasp-Oriented Fine-grained Cloth Segmentation without Real Supervision](https://mlsanity.com/paper/2110.02903/) (2021)
- [Meta-Optimization for Higher Model Generalizability in Single-Image Depth Prediction](https://mlsanity.com/paper/2305.07269/) (2023)
- [Pre-training of Deep RL Agents for Improved Learning under Domain Randomization](https://mlsanity.com/paper/2104.14386/) (2021)
- [Pretrained Encoders are All You Need](https://mlsanity.com/paper/2106.05139/) (2021)
- [Sim-and-Real Reinforcement Learning for Manipulation: A Consensus-based Approach](https://mlsanity.com/paper/2302.13423/) (2023)
- [Integration of Vision-based Object Detection and Grasping for Articulated Manipulator in Lunar Conditions](https://mlsanity.com/paper/2309.01055/) (2023)
- [Data-efficient learning of object-centric grasp preferences](https://mlsanity.com/paper/2203.00384/) (2022)
- [VGQ-CNN: Moving Beyond Fixed Cameras and Top-Grasps for Grasp Quality Prediction](https://mlsanity.com/paper/2203.04874/) (2022)
- [Boosting Generalizability towards Zero-Shot Cross-Dataset Single-Image Indoor Depth by Meta-Initialization](https://mlsanity.com/paper/2409.02486/) (2024)
- [Development of a robust cascaded architecture for intelligent robot grasping using limited labelled data](https://mlsanity.com/paper/2112.03001/) (2021)
- [PEGG-Net: Pixel-Wise Efficient Grasp Generation in Complex Scenes](https://mlsanity.com/paper/2203.16301/) (2022)
- [GP-net: Flexible Viewpoint Grasp Proposal](https://mlsanity.com/paper/2209.10404/) (2022)
- [Sim-to-Real Deep Reinforcement Learning with Manipulators for Pick-and-place](https://mlsanity.com/paper/2309.09247/) (2023)

## Abstract (arXiv)

We want to build robots that are useful in unstructured real world applications, such as doing work in the household. Grasping in particular is an important skill in this domain, yet it remains a challenge. One of the key hurdles is handling unexpected changes or motion in the objects being grasped and kinematic noise or other errors in the robot. This paper proposes an approach to learning a closed-loop controller for robotic grasping that dynamically guides the gripper to the object. We use a wrist-mounted sensor to acquire depth images in front of the gripper and train a convolutional neural network to learn a distance function to true grasps for grasp configurations over an image. The training sensor data is generated in simulation, a major advantage over previous work that uses real robot experience, which is costly to obtain. Despite being trained in simulation, our approach works well on real noisy sensor images. We compare our controller in simulated and real robot experiments to a strong baseline for grasp pose detection, and find that our approach significantly outperforms the baseline in the presence of kinematic noise, perceptual errors and disturbances of the object during grasping.