# Mechanical Search: Multi-Step Retrieval of a Target Object Occluded by Clutter

arXiv: https://arxiv.org/abs/1903.01588 · HTML: https://mlsanity.com/paper/1903.01588/

## Key facts

- Cited by 30 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 4 Mar 2019.

## Built on by (most cited first, 30 of 30)

- [LEGO-Net: Learning Regular Rearrangements of Objects in Rooms](https://mlsanity.com/paper/2301.09629/) (2023)
- [Multi-Stage Cable Routing through Hierarchical Imitation Learning](https://mlsanity.com/paper/2307.08927/) (2023)
- [RT-Sketch: Goal-Conditioned Imitation Learning from Hand-Drawn Sketches](https://mlsanity.com/paper/2403.02709/) (2024)
- [INVIGORATE: Interactive Visual Grounding and Grasping in Clutter](https://mlsanity.com/paper/2108.11092/) (2021)
- [A Joint Network for Grasp Detection Conditioned on Natural Language Commands](https://mlsanity.com/paper/2104.00492/) (2021)
- [Play it by Ear: Learning Skills amidst Occlusion through Audio-Visual Imitation Learning](https://mlsanity.com/paper/2205.14850/) (2022)
- [Robotic Visual Instruction](https://mlsanity.com/paper/2505.00693/) (2025)
- [Learning Deformable Object Manipulation from Expert Demonstrations](https://mlsanity.com/paper/2207.10148/) (2022)
- [Orienting Novel 3D Objects Using Self-Supervised Learning of Rotation Transforms](https://mlsanity.com/paper/2105.14246/) (2021)
- [ICGNet: A Unified Approach for Instance-Centric Grasping](https://mlsanity.com/paper/2401.09939/) (2024)
- [Collision-Aware Target-Driven Object Grasping in Constrained Environments](https://mlsanity.com/paper/2104.00776/) (2021)
- [Uniform Object Rearrangement: From Complete Monotone Primitives to Efficient Non-Monotone Informed Search](https://mlsanity.com/paper/2101.12241/) (2021)
- [Attribute-Based Robotic Grasping with Data-Efficient Adaptation](https://mlsanity.com/paper/2501.02149/) (2025)
- [Attribute-Based Robotic Grasping with One-Grasp Adaptation](https://mlsanity.com/paper/2104.02271/) (2021)
- [Efficient and High-quality Prehensile Rearrangement in Cluttered and Confined Spaces](https://mlsanity.com/paper/2110.02814/) (2021)
- [A Hybrid Approach for Learning to Shift and Grasp with Elaborate Motion Primitives](https://mlsanity.com/paper/2111.01510/) (2021)
- [Discovering Generalizable Skills via Automated Generation of Diverse Tasks](https://mlsanity.com/paper/2106.13935/) (2021)
- [Smart Explorer: Recognizing Objects in Dense Clutter via Interactive Exploration](https://mlsanity.com/paper/2208.03496/) (2022)
- [SGTM 2.0: Autonomously Untangling Long Cables using Interactive Perception](https://mlsanity.com/paper/2209.13706/) (2022)
- [Grasp, See, and Place: Efficient Unknown Object Rearrangement with Policy Structure Prior](https://mlsanity.com/paper/2402.15402/) (2024)
- [Visionary: Vision architecture discovery for robot learning](https://mlsanity.com/paper/2103.14633/) (2021)
- [LLMs for Robotic Object Disambiguation](https://mlsanity.com/paper/2401.03388/) (2024)
- [Mechanical Search on Shelves using a Novel "Bluction" Tool](https://mlsanity.com/paper/2201.08968/) (2022)
- [Self-Supervised Learning for Interactive Perception of Surgical Thread for Autonomous Suture Tail-Shortening](https://mlsanity.com/paper/2307.06845/) (2023)
- [Language-Conditioned Observation Models for Visual Object Search](https://mlsanity.com/paper/2309.07276/) (2023)
- [Learning Dual-arm Object Rearrangement for Cartesian Robots](https://mlsanity.com/paper/2402.13634/) (2024)
- [Map Space Belief Prediction for Manipulation-Enhanced Mapping](https://mlsanity.com/paper/2502.20606/) (2025)
- [XPG-RL: Reinforcement Learning with Explainable Priority Guidance for Efficiency-Boosted Mechanical Search](https://mlsanity.com/paper/2504.20969/) (2025)
- [RUMI: Rummaging Using Mutual Information](https://mlsanity.com/paper/2408.10450/) (2024)
- [RoboEye: Enhancing 2D Robotic Object Identification with Selective 3D Geometric Keypoint Matching](https://mlsanity.com/paper/2509.14966/) (2025)

## Abstract (arXiv)

When operating in unstructured environments such as warehouses, homes, and retail centers, robots are frequently required to interactively search for and retrieve specific objects from cluttered bins, shelves, or tables. Mechanical Search describes the class of tasks where the goal is to locate and extract a known target object. In this paper, we formalize Mechanical Search and study a version where distractor objects are heaped over the target object in a bin. The robot uses an RGBD perception system and control policies to iteratively select, parameterize, and perform one of 3 actions -- push, suction, grasp -- until the target object is extracted, or either a time limit is exceeded, or no high confidence push or grasp is available. We present a study of 5 algorithmic policies for mechanical search, with 15,000 simulated trials and 300 physical trials for heaps ranging from 10 to 20 objects. Results suggest that success can be achieved in this long-horizon task with algorithmic policies in over 95% of instances and that the number of actions required scales approximately linearly with the size of the heap. Code and supplementary material can be found at http://ai.stanford.edu/mech-search .