# Visuomotor Mechanical Search: Learning to Retrieve Target Objects in Clutter

arXiv: https://arxiv.org/abs/2008.06073 · HTML: https://mlsanity.com/paper/2008.06073/

## Key facts

- Cited by 16 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 13 Aug 2020.

## Built on by (most cited first, 16 of 16)

- [Long-Horizon Manipulation of Unknown Objects via Task and Motion Planning with Estimated Affordances](https://mlsanity.com/paper/2108.04145/) (2021)
- [A Joint Modeling of Vision-Language-Action for Target-oriented Grasping in Clutter](https://mlsanity.com/paper/2302.12610/) (2023)
- [INVIGORATE: Interactive Visual Grounding and Grasping in Clutter](https://mlsanity.com/paper/2108.11092/) (2021)
- [Learning Environment-Aware Affordance for 3D Articulated Object Manipulation under Occlusions](https://mlsanity.com/paper/2309.07510/) (2023)
- [The Role of Predictive Uncertainty and Diversity in Embodied AI and Robot Learning](https://mlsanity.com/paper/2405.03164/) (2024)
- [GarmentPile: Point-Level Visual Affordance Guided Retrieval and Adaptation for Cluttered Garments Manipulation](https://mlsanity.com/paper/2503.09243/) (2025)
- [Self-Supervised Interactive Object Segmentation Through a Singulation-and-Grasping Approach](https://mlsanity.com/paper/2207.09314/) (2022)
- [Discovering Generalizable Skills via Automated Generation of Diverse Tasks](https://mlsanity.com/paper/2106.13935/) (2021)
- [LLMs for Robotic Object Disambiguation](https://mlsanity.com/paper/2401.03388/) (2024)
- [Task-Oriented 6-DoF Grasp Pose Detection in Clutters](https://mlsanity.com/paper/2502.16976/) (2025)
- [Mechanical Search on Shelves using a Novel "Bluction" Tool](https://mlsanity.com/paper/2201.08968/) (2022)
- [SafePicking: Learning Safe Object Extraction via Object-Level Mapping](https://mlsanity.com/paper/2202.05832/) (2022)
- [Efficient Alignment of Unconditioned Action Prior for Language-conditioned Pick and Place in Clutter](https://mlsanity.com/paper/2503.09423/) (2025)
- [LAC-Net: Linear-Fusion Attention-Guided Convolutional Network for Accurate Robotic Grasping Under the Occlusion](https://mlsanity.com/paper/2408.03238/) (2024)
- [XPG-RL: Reinforcement Learning with Explainable Priority Guidance for Efficiency-Boosted Mechanical Search](https://mlsanity.com/paper/2504.20969/) (2025)
- [Gentle Object Retraction in Dense Clutter Using Multimodal Force Sensing and Imitation Learning](https://mlsanity.com/paper/2508.19476/) (2025)

## Abstract (arXiv)

When searching for objects in cluttered environments, it is often necessary to perform complex interactions in order to move occluding objects out of the way and fully reveal the object of interest and make it graspable. Due to the complexity of the physics involved and the lack of accurate models of the clutter, planning and controlling precise predefined interactions with accurate outcome is extremely hard, when not impossible. In problems where accurate (forward) models are lacking, Deep Reinforcement Learning (RL) has shown to be a viable solution to map observations (e.g. images) to good interactions in the form of close-loop visuomotor policies. However, Deep RL is sample inefficient and fails when applied directly to the problem of unoccluding objects based on images. In this work we present a novel Deep RL procedure that combines i) teacher-aided exploration, ii) a critic with privileged information, and iii) mid-level representations, resulting in sample efficient and effective learning for the problem of uncovering a target object occluded by a heap of unknown objects. Our experiments show that our approach trains faster and converges to more efficient uncovering solutions than baselines and ablations, and that our uncovering policies lead to an average improvement in the graspability of the target object, facilitating downstream retrieval applications.