# A Deep Learning Approach to Grasping the Invisible

arXiv: https://arxiv.org/abs/1909.04840 · HTML: https://mlsanity.com/paper/1909.04840/

## Key facts

- Cited by 19 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 11 Sep 2019.

## Built on by (most cited first, 19 of 19)

- [A Survey of Embodied Learning for Object-Centric Robotic Manipulation](https://mlsanity.com/paper/2408.11537/) (2024)
- [A Joint Modeling of Vision-Language-Action for Target-oriented Grasping in Clutter](https://mlsanity.com/paper/2302.12610/) (2023)
- [INVIGORATE: Interactive Visual Grounding and Grasping in Clutter](https://mlsanity.com/paper/2108.11092/) (2021)
- [ICGNet: A Unified Approach for Instance-Centric Grasping](https://mlsanity.com/paper/2401.09939/) (2024)
- [Collision-Aware Target-Driven Object Grasping in Constrained Environments](https://mlsanity.com/paper/2104.00776/) (2021)
- [Self-Supervised Interactive Object Segmentation Through a Singulation-and-Grasping Approach](https://mlsanity.com/paper/2207.09314/) (2022)
- [Attribute-Based Robotic Grasping with Data-Efficient Adaptation](https://mlsanity.com/paper/2501.02149/) (2025)
- [Attribute-Based Robotic Grasping with One-Grasp Adaptation](https://mlsanity.com/paper/2104.02271/) (2021)
- [Smart Explorer: Recognizing Objects in Dense Clutter via Interactive Exploration](https://mlsanity.com/paper/2208.03496/) (2022)
- [Mechanical Search on Shelves using a Novel "Bluction" Tool](https://mlsanity.com/paper/2201.08968/) (2022)
- [Efficient Alignment of Unconditioned Action Prior for Language-conditioned Pick and Place in Clutter](https://mlsanity.com/paper/2503.09423/) (2025)
- [Sim-and-Real Reinforcement Learning for Manipulation: A Consensus-based Approach](https://mlsanity.com/paper/2302.13423/) (2023)
- [LAC-Net: Linear-Fusion Attention-Guided Convolutional Network for Accurate Robotic Grasping Under the Occlusion](https://mlsanity.com/paper/2408.03238/) (2024)
- [Bootstrapping Concept Formation in Small Neural Networks](https://mlsanity.com/paper/2110.13665/) (2021)
- [Category-level Shape Estimation for Densely Cluttered Objects](https://mlsanity.com/paper/2302.11983/) (2023)
- [BEAC: Imitating Complex Exploration and Task-oriented Behaviors for Invisible Object Nonprehensile Manipulation](https://mlsanity.com/paper/2503.16803/) (2025)
- [ClutterNav: Gradient-Guided Search for Efficient 3D Clutter Removal with Learned Costmaps](https://mlsanity.com/paper/2511.12479/) (2025)
- [Visual Prompt Guided Unified Pushing Policy](https://mlsanity.com/paper/2602.19193/) (2026)
- [Learning Object-Centric Spatial Reasoning for Sequential Manipulation in Cluttered Environments](https://mlsanity.com/paper/2603.02511/) (2026)

## Abstract (arXiv)

We study an emerging problem named "grasping the invisible" in robotic manipulation, in which a robot is tasked to grasp an initially invisible target object via a sequence of pushing and grasping actions. In this problem, pushes are needed to search for the target and rearrange cluttered objects around it to enable effective grasps. We propose to solve the problem by formulating a deep learning approach in a critic-policy format. The target-oriented motion critic, which maps both visual observations and target information to the expected future rewards of pushing and grasping motion primitives, is learned via deep Q-learning. We divide the problem into two subtasks, and two policies are proposed to tackle each of them, by combining the critic predictions and relevant domain knowledge. A Bayesian-based policy accounting for past action experience performs pushing to search for the target; once the target is found, a classifier-based policy coordinates target-oriented pushing and grasping to grasp the target in clutter. The motion critic and the classifier are trained in a self-supervised manner through robot-environment interactions. Our system achieves a 93% and 87% task success rate on each of the two subtasks in simulation and an 85% task success rate in real robot experiments on the whole problem, which outperforms several baselines by large margins. Supplementary material is available at https://sites.google.com/umn.edu/grasping-invisible.