# Object Finding in Cluttered Scenes Using Interactive Perception

arXiv: https://arxiv.org/abs/1911.07482 · HTML: https://mlsanity.com/paper/1911.07482/

## Key facts

- Cited by 18 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 18 Nov 2019.

## Built on by (most cited first, 18 of 18)

- [Coarse-to-Fine Q-attention: Efficient Learning for Visual Robotic Manipulation via Discretisation](https://mlsanity.com/paper/2106.12534/) (2021)
- [Occlusion Handling in Generic Object Detection: A Review](https://mlsanity.com/paper/2101.08845/) (2021)
- [SAGCI-System: Towards Sample-Efficient, Generalizable, Compositional, and Incremental Robot Learning](https://mlsanity.com/paper/2111.14693/) (2021)
- [Towards Optimal Correlational Object Search](https://mlsanity.com/paper/2110.09991/) (2021)
- [From Screens to Scenes: A Survey of Embodied AI in Healthcare](https://mlsanity.com/paper/2501.07468/) (2025)
- [DoRO: Disambiguation of referred object for embodied agents](https://mlsanity.com/paper/2207.14205/) (2022)
- [Smart Explorer: Recognizing Objects in Dense Clutter via Interactive Exploration](https://mlsanity.com/paper/2208.03496/) (2022)
- [SGTM 2.0: Autonomously Untangling Long Cables using Interactive Perception](https://mlsanity.com/paper/2209.13706/) (2022)
- [SO-NeRF: Active View Planning for NeRF using Surrogate Objectives](https://mlsanity.com/paper/2312.03266/) (2023)
- [Grasp, See, and Place: Efficient Unknown Object Rearrangement with Policy Structure Prior](https://mlsanity.com/paper/2402.15402/) (2024)
- [Tactile-Filter: Interactive Tactile Perception for Part Mating](https://mlsanity.com/paper/2303.06034/) (2023)
- [Mechanical Search on Shelves using a Novel "Bluction" Tool](https://mlsanity.com/paper/2201.08968/) (2022)
- [Self-Supervised Learning for Interactive Perception of Surgical Thread for Autonomous Suture Tail-Shortening](https://mlsanity.com/paper/2307.06845/) (2023)
- [Real-World Reinforcement Learning of Active Perception Behaviors](https://mlsanity.com/paper/2512.01188/) (2025)
- [Fusing RGBD Tracking and Segmentation Tree Sampling for Multi-Hypothesis Volumetric Segmentation](https://mlsanity.com/paper/2104.00205/) (2021)
- [Online Refinement of a Scene Recognition Model for Mobile Robots by Observing Human's Interaction with Environments](https://mlsanity.com/paper/2208.06636/) (2022)
- [Zero-shot Interactive Perception](https://mlsanity.com/paper/2602.18374/) (2026)
- [GoStop: Reinforcement Learning for Adaptive Temporal Aggregation in Event-Based Feature Tracking](https://mlsanity.com/paper/2607.15699/) (2026)

## Abstract (arXiv)

Object finding in clutter is a skill that requires perception of the environment and in many cases physical interaction. In robotics, interactive perception defines a set of algorithms that leverage actions to improve the perception of the environment, and vice versa use perception to guide the next action. Scene interactions are difficult to model, therefore, most of the current systems use predefined heuristics. This limits their ability to efficiently search for the target object in a complex environment. In order to remove heuristics and the need for explicit models of the interactions, in this work we propose a reinforcement learning based active and interactive perception system for scene exploration and object search. We evaluate our work both in simulated and in real-world experiments using a robotic manipulator equipped with an RGB and a depth camera, and compare our system to two baselines. The results indicate that our approach, trained in simulation only, transfers smoothly to reality and can solve the object finding task efficiently and with more than 88% success rate.