# SupeRGB-D: Zero-shot Instance Segmentation in Cluttered Indoor Environments

arXiv: https://arxiv.org/abs/2212.11922 · HTML: https://mlsanity.com/paper/2212.11922/

## Key facts

- Cited by 4 papers in the map (data through 14 Oct 2026).
- Builds on 33 of its 52 references that are in the map.
- First posted on arXiv 22 Dec 2022.

## Builds on (most cited first, 30 of 33)

- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [Mask R-CNN](https://mlsanity.com/paper/1703.06870/) (2017)
- [Emerging Properties in Self-Supervised Vision Transformers](https://mlsanity.com/paper/2104.14294/) (2021)
- [ShapeNet: An Information-Rich 3D Model Repository](https://mlsanity.com/paper/1512.03012/) (2015)
- [Semantic Scene Completion from a Single Depth Image](https://mlsanity.com/paper/1611.08974/) (2016)
- [Point-Voxel CNN for Efficient 3D Deep Learning](https://mlsanity.com/paper/1907.03739/) (2019)
- [Zero-Shot Semantic Segmentation](https://mlsanity.com/paper/1906.00817/) (2019)
- [3D-SIS: 3D Semantic Instance Segmentation of RGB-D Scans](https://mlsanity.com/paper/1812.07003/) (2018)
- [Zero-Shot Object Detection](https://mlsanity.com/paper/1804.04340/) (2018)
- [SGPN: Similarity Group Proposal Network for 3D Point Cloud Instance Segmentation](https://mlsanity.com/paper/1711.08588/) (2017)
- [Multimodal Token Fusion for Vision Transformers](https://mlsanity.com/paper/2204.08721/) (2022)
- [Learning Open-World Object Proposals without Learning to Classify](https://mlsanity.com/paper/2108.06753/) (2021)
- [A Survey on Instance Segmentation: State of the art](https://mlsanity.com/paper/2007.00047/) (2020)
- [Unseen Object Instance Segmentation for Robotic Environments](https://mlsanity.com/paper/2007.08073/) (2020)
- [Learning to Segment Object Candidates](https://mlsanity.com/paper/1506.06204/) (2015)
- [Few-shot 3D Point Cloud Semantic Segmentation](https://mlsanity.com/paper/2006.12052/) (2020)
- [Learning RGB-D Feature Embeddings for Unseen Object Instance Segmentation](https://mlsanity.com/paper/2007.15157/) (2020)
- [Generalized Few-shot Semantic Segmentation](https://mlsanity.com/paper/2010.05210/) (2020)
- [EasyLabel: A Semi-Automatic Pixel-wise Object Annotation Tool for Creating Robotic RGB-D Datasets](https://mlsanity.com/paper/1902.01626/) (2019)
- [Generative Zero-Shot Learning for Semantic Segmentation of 3D Point Clouds](https://mlsanity.com/paper/2108.06230/) (2021)
- [Incremental Few-Shot Instance Segmentation](https://mlsanity.com/paper/2105.05312/) (2021)
- [Unseen Object Amodal Instance Segmentation via Hierarchical Occlusion Modeling](https://mlsanity.com/paper/2109.11103/) (2021)
- [Open-World Instance Segmentation: Exploiting Pseudo Ground Truth From Learned Pairwise Affinity](https://mlsanity.com/paper/2204.06107/) (2022)
- [Zero-Shot Instance Segmentation](https://mlsanity.com/paper/2104.06601/) (2021)
- [Novel Class Discovery in Semantic Segmentation](https://mlsanity.com/paper/2112.01900/) (2021)
- [Learning to Group: A Bottom-Up Framework for 3D Part Discovery in Unseen Categories](https://mlsanity.com/paper/2002.06478/) (2020)
- [From 2D to 3D: Re-thinking Benchmarking of Monocular Depth Prediction](https://mlsanity.com/paper/2203.08122/) (2022)
- [SimNet: Enabling Robust Unknown Object Manipulation from Pure Synthetic Data via Stereo](https://mlsanity.com/paper/2106.16118/) (2021)
- [Unknown Object Segmentation from Stereo Images](https://mlsanity.com/paper/2103.06796/) (2021)
- [3D Compositional Zero-shot Learning with DeCompositional Consensus](https://mlsanity.com/paper/2111.14673/) (2021)

## Built on by (most cited first, 4 of 4)

- [FoundPose: Unseen Object Pose Estimation with Foundation Features](https://mlsanity.com/paper/2311.18809/) (2023)
- [ZeroPose: CAD-Prompted Zero-shot Object 6D Pose Estimation in Cluttered Scenes](https://mlsanity.com/paper/2305.17934/) (2023)
- [Configurable Embodied Data Generation for Class-Agnostic RGB-D Video Segmentation](https://mlsanity.com/paper/2410.12995/) (2024)
- [Clutt3R-Seg: Sparse-view 3D Instance Segmentation for Language-grounded Grasping in Cluttered Scenes](https://mlsanity.com/paper/2602.11660/) (2026)

## Abstract (arXiv)

Object instance segmentation is a key challenge for indoor robots navigating cluttered environments with many small objects. Limitations in 3D sensing capabilities often make it difficult to detect every possible object. While deep learning approaches may be effective for this problem, manually annotating 3D data for supervised learning is time-consuming. In this work, we explore zero-shot instance segmentation (ZSIS) from RGB-D data to identify unseen objects in a semantic category-agnostic manner. We introduce a zero-shot split for Tabletop Objects Dataset (TOD-Z) to enable this study and present a method that uses annotated objects to learn the ``objectness'' of pixels and generalize to unseen object categories in cluttered indoor environments. Our method, SupeRGB-D, groups pixels into small patches based on geometric cues and learns to merge the patches in a deep agglomerative clustering fashion. SupeRGB-D outperforms existing baselines on unseen objects while achieving similar performance on seen objects. We further show competitive results on the real dataset OCID. With its lightweight design (0.4 MB memory requirement), our method is extremely suitable for mobile and robotic applications. Additional DINO features can increase performance with a higher memory requirement. The dataset split and code are available at https://github.com/evinpinar/supergb-d.