# Smart Explorer: Recognizing Objects in Dense Clutter via Interactive Exploration

arXiv: https://arxiv.org/abs/2208.03496 · HTML: https://mlsanity.com/paper/2208.03496/

## Key facts

- Cited by 5 papers in the map (data through 14 Oct 2026).
- Builds on 25 of its 38 references that are in the map.
- First posted on arXiv 6 Aug 2022.

## Builds on (most cited first, 25 of 25)

- [YOLACT: Real-time Instance Segmentation](https://mlsanity.com/paper/1904.02689/) (2019)
- [6-DOF GraspNet: Variational Grasp Generation for Object Manipulation](https://mlsanity.com/paper/1905.10520/) (2019)
- [Contact-GraspNet: Efficient 6-DoF Grasp Generation in Cluttered Scenes](https://mlsanity.com/paper/2103.14127/) (2021)
- [Benchmarking in Manipulation Research: The YCB Object and Model Set and Benchmarking Protocols](https://mlsanity.com/paper/1502.03143/) (2015)
- [Learning RGB-D Feature Embeddings for Unseen Object Instance Segmentation](https://mlsanity.com/paper/2007.15157/) (2020)
- [Object Rearrangement Using Learned Implicit Collision Functions](https://mlsanity.com/paper/2011.10726/) (2020)
- [The Best of Both Modes: Separately Leveraging RGB and Depth for Unseen Object Instance Segmentation](https://mlsanity.com/paper/1907.13236/) (2019)
- [Mechanical Search: Multi-Step Retrieval of a Target Object Occluded by Clutter](https://mlsanity.com/paper/1903.01588/) (2019)
- [RGB-D Object Detection and Semantic Segmentation for Autonomous Manipulation in Clutter](https://mlsanity.com/paper/1810.00818/) (2018)
- [A Deep Learning Approach to Grasping the Invisible](https://mlsanity.com/paper/1909.04840/) (2019)
- [Object Finding in Cluttered Scenes Using Interactive Perception](https://mlsanity.com/paper/1911.07482/) (2019)
- [Where to relocate?: Object rearrangement inside cluttered and confined environments for robotic manipulation](https://mlsanity.com/paper/2003.10863/) (2020)
- [RICE: Refining Instance Masks in Cluttered Environments with Graph Neural Networks](https://mlsanity.com/paper/2106.15711/) (2021)
- [Self-supervised Transfer Learning for Instance Segmentation through Physical Interaction](https://mlsanity.com/paper/2005.09484/) (2020)
- [Joint Learning of Instance and Semantic Segmentation for Robotic Pick-and-Place with Heavy Occlusions in Clutter](https://mlsanity.com/paper/2001.07481/) (2020)
- [Task-driven Perception and Manipulation for Constrained Placement of Unknown Objects](https://mlsanity.com/paper/2006.15503/) (2020)
- [Occlusion-Aware Search for Object Retrieval in Clutter](https://mlsanity.com/paper/2011.03334/) (2020)
- [Efficient Robotic Object Search via HIEM: Hierarchical Policy Learning with Intrinsic-Extrinsic Modeling](https://mlsanity.com/paper/2010.08596/) (2020)
- [Uniform Object Rearrangement: From Complete Monotone Primitives to Efficient Non-Monotone Informed Search](https://mlsanity.com/paper/2101.12241/) (2021)
- [GE-Grasp: Efficient Target-Oriented Grasping in Dense Clutter](https://mlsanity.com/paper/2207.11941/) (2022)
- [Multiresolution Mapping and Informative Path Planning for UAV-based Terrain Monitoring](https://mlsanity.com/paper/1703.02854/) (2017)
- [Fast and resilient manipulation planning for target retrieval in clutter](https://mlsanity.com/paper/2003.11420/) (2020)
- [Rotational Subgroup Voting and Pose Clustering for Robust 3D Object Recognition](https://mlsanity.com/paper/1709.02142/) (2017)
- [NudgeSeg: Zero-Shot Object Segmentation by Repeated Physical Interaction](https://mlsanity.com/paper/2109.13859/) (2021)
- [Active and Interactive Mapping with Dynamic Gaussian Process Implicit Surfaces for Mobile Manipulators](https://mlsanity.com/paper/2010.13108/) (2020)

## Built on by (most cited first, 5 of 5)

- [Embodied Task Planning with Large Language Models](https://mlsanity.com/paper/2307.01848/) (2023)
- [Planning Irregular Object Packing via Hierarchical Reinforcement Learning](https://mlsanity.com/paper/2211.09382/) (2022)
- [Category-level Shape Estimation for Densely Cluttered Objects](https://mlsanity.com/paper/2302.11983/) (2023)
- [Learning Accurate Performance Predictors for Ultrafast Automated Model Compression](https://mlsanity.com/paper/2304.06393/) (2023)
- [SegmATRon: Embodied Adaptive Semantic Segmentation for Indoor Environment](https://mlsanity.com/paper/2310.12031/) (2023)

## Abstract (arXiv)

Recognizing objects in dense clutter accurately plays an important role to a wide variety of robotic manipulation tasks including grasping, packing, rearranging and many others. However, conventional visual recognition models usually miss objects because of the significant occlusion among instances and causes incorrect prediction due to the visual ambiguity with the high object crowdedness. In this paper, we propose an interactive exploration framework called Smart Explorer for recognizing all objects in dense clutters. Our Smart Explorer physically interacts with the clutter to maximize the recognition performance while minimize the number of motions, where the false positives and negatives can be alleviated effectively with the optimal accuracy-efficiency trade-offs. Specifically, we first collect the multi-view RGB-D images of the clutter and reconstruct the corresponding point cloud. By aggregating the instance segmentation of RGB images across views, we acquire the instance-wise point cloud partition of the clutter through which the existed classes and the number of objects for each class are predicted. The pushing actions for effective physical interaction are generated to sizably reduce the recognition uncertainty that consists of the instance segmentation entropy and multi-view object disagreement. Therefore, the optimal accuracy-efficiency trade-off of object recognition in dense clutter is achieved via iterative instance prediction and physical interaction. Extensive experiments demonstrate that our Smart Explorer acquires promising recognition accuracy with only a few actions, which also outperforms the random pushing by a large margin.