# Unseen Object Instance Segmentation for Robotic Environments

arXiv: https://arxiv.org/abs/2007.08073 · HTML: https://mlsanity.com/paper/2007.08073/

## Key facts

- Cited by 85 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 16 Jul 2020.

## Built on by (most cited first, 30 of 85)

- [MegaPose: 6D Pose Estimation of Novel Objects via Render & Compare](https://mlsanity.com/paper/2212.06870/) (2022)
- [Superpoint Transformer for 3D Scene Instance Segmentation](https://mlsanity.com/paper/2211.15766/) (2022)
- [Learning Open-World Object Proposals without Learning to Classify](https://mlsanity.com/paper/2108.06753/) (2021)
- [CNOS: A Strong Baseline for CAD-based Novel Object Segmentation](https://mlsanity.com/paper/2307.11067/) (2023)
- [SE(3)-Equivariant Relational Rearrangement with Neural Descriptor Fields](https://mlsanity.com/paper/2211.09786/) (2022)
- [CaTGrasp: Learning Category-Level Task-Relevant Grasping in Clutter from Simulation](https://mlsanity.com/paper/2109.09163/) (2021)
- [Unseen Object Amodal Instance Segmentation via Hierarchical Occlusion Modeling](https://mlsanity.com/paper/2109.11103/) (2021)
- [Long-Horizon Manipulation of Unknown Objects via Task and Motion Planning with Estimated Affordances](https://mlsanity.com/paper/2108.04145/) (2021)
- [Affordance Learning from Play for Sample-Efficient Policy Learning](https://mlsanity.com/paper/2203.00352/) (2022)
- [Towards Scale Balanced 6-DoF Grasp Detection in Cluttered Scenes](https://mlsanity.com/paper/2212.05275/) (2022)
- [ZeroPose: CAD-Prompted Zero-shot Object 6D Pose Estimation in Cluttered Scenes](https://mlsanity.com/paper/2305.17934/) (2023)
- [Mean Shift Mask Transformer for Unseen Object Instance Segmentation](https://mlsanity.com/paper/2211.11679/) (2022)
- [ICM-3D: Instantiated Category Modeling for 3D Instance Segmentation](https://mlsanity.com/paper/2108.11771/) (2021)
- [ARMBench: An Object-centric Benchmark Dataset for Robotic Manipulation](https://mlsanity.com/paper/2303.16382/) (2023)
- [Adapting Pre-Trained Vision Models for Novel Instance Detection and Segmentation](https://mlsanity.com/paper/2405.17859/) (2024)
- [Unknown Object Segmentation from Stereo Images](https://mlsanity.com/paper/2103.06796/) (2021)
- [RICE: Refining Instance Masks in Cluttered Environments with Graph Neural Networks](https://mlsanity.com/paper/2106.15711/) (2021)
- [Towards Open-World Grasping with Large Vision-Language Models](https://mlsanity.com/paper/2406.18722/) (2024)
- [CenterGrasp: Object-Aware Implicit Representation Learning for Simultaneous Shape Reconstruction and 6-DoF Grasp Estimation](https://mlsanity.com/paper/2312.08240/) (2023)
- [NeuralBF: Neural Bilateral Filtering for Top-down Instance Segmentation on Point Clouds](https://mlsanity.com/paper/2207.09978/) (2022)
- [Self-Supervised Unseen Object Instance Segmentation via Long-Term Robot Interaction](https://mlsanity.com/paper/2302.03793/) (2023)
- [USIS16K: High-Quality Dataset for Underwater Salient Instance Segmentation](https://mlsanity.com/paper/2506.19472/) (2025)
- [Radar Instance Transformer: Reliable Moving Instance Segmentation in Sparse Radar Point Clouds](https://mlsanity.com/paper/2309.16435/) (2023)
- [Benchmarking the Robustness of Instance Segmentation Models](https://mlsanity.com/paper/2109.01123/) (2021)
- [Unseen Object Instance Segmentation with Fully Test-time RGB-D Embeddings Adaptation](https://mlsanity.com/paper/2204.09847/) (2022)
- [Self-Supervised Interactive Object Segmentation Through a Singulation-and-Grasping Approach](https://mlsanity.com/paper/2207.09314/) (2022)
- [Open-world Instance Segmentation: Top-down Learning with Bottom-up Supervision](https://mlsanity.com/paper/2303.05503/) (2023)
- [GraspClutter6D: A Large-scale Real-world Dataset for Robust Perception and Grasping in Cluttered Scenes](https://mlsanity.com/paper/2504.06866/) (2025)
- [A Neuromorphic Dataset for Object Segmentation in Indoor Cluttered Environment](https://mlsanity.com/paper/2302.06301/) (2023)
- [Sim-Suction: Learning a Suction Grasp Policy for Cluttered Environments Using a Synthetic Benchmark](https://mlsanity.com/paper/2305.16378/) (2023)

## Abstract (arXiv)

In order to function in unstructured environments, robots need the ability to recognize unseen objects. We take a step in this direction by tackling the problem of segmenting unseen object instances in tabletop environments. However, the type of large-scale real-world dataset required for this task typically does not exist for most robotic settings, which motivates the use of synthetic data. Our proposed method, UOIS-Net, separately leverages synthetic RGB and synthetic depth for unseen object instance segmentation. UOIS-Net is comprised of two stages: first, it operates only on depth to produce object instance center votes in 2D or 3D and assembles them into rough initial masks. Secondly, these initial masks are refined using RGB. Surprisingly, our framework is able to learn from synthetic RGB-D data where the RGB is non-photorealistic. To train our method, we introduce a large-scale synthetic dataset of random objects on tabletops. We show that our method can produce sharp and accurate segmentation masks, outperforming state-of-the-art methods on unseen object instance segmentation. We also show that our method can segment unseen objects for robot grasping.