# RICE: Refining Instance Masks in Cluttered Environments with Graph Neural Networks

arXiv: https://arxiv.org/abs/2106.15711 · HTML: https://mlsanity.com/paper/2106.15711/

## Key facts

- Cited by 15 papers in the map (data through 14 Oct 2026).
- Builds on 42 of its 55 references that are in the map.
- First posted on arXiv 29 Jun 2021.

## Builds on (most cited first, 30 of 42)

- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [U-Net: Convolutional Networks for Biomedical Image Segmentation](https://mlsanity.com/paper/1505.04597/) (2015)
- [Microsoft COCO: Common Objects in Context](https://mlsanity.com/paper/1405.0312/) (2014)
- [Mask R-CNN](https://mlsanity.com/paper/1703.06870/) (2017)
- [Feature Pyramid Networks for Object Detection](https://mlsanity.com/paper/1612.03144/) (2016)
- [ShapeNet: An Information-Rich 3D Model Repository](https://mlsanity.com/paper/1512.03012/) (2015)
- [Group Normalization](https://mlsanity.com/paper/1803.08494/) (2018)
- [Relational inductive biases, deep learning, and graph networks](https://mlsanity.com/paper/1806.01261/) (2018)
- [Interaction Networks for Learning about Objects, Relations and Physics](https://mlsanity.com/paper/1612.00222/) (2016)
- [Scene Graph Generation by Iterative Message Passing](https://mlsanity.com/paper/1701.02426/) (2017)
- [PointRend: Image Segmentation as Rendering](https://mlsanity.com/paper/1912.08193/) (2019)
- [Neural Motifs: Scene Graph Parsing with Global Context](https://mlsanity.com/paper/1711.06640/) (2017)
- [A simple neural network module for relational reasoning](https://mlsanity.com/paper/1706.01427/) (2017)
- [Few-Shot Learning with Graph Neural Networks](https://mlsanity.com/paper/1711.04043/) (2017)
- [PointGroup: Dual-Set Point Grouping for 3D Instance Segmentation](https://mlsanity.com/paper/2004.01658/) (2020)
- [Relation Networks for Object Detection](https://mlsanity.com/paper/1711.11575/) (2017)
- [6-DOF GraspNet: Variational Grasp Generation for Object Manipulation](https://mlsanity.com/paper/1905.10520/) (2019)
- [Contact-GraspNet: Efficient 6-DoF Grasp Generation in Cluttered Scenes](https://mlsanity.com/paper/2103.14127/) (2021)
- [Zero-shot Recognition via Semantic Embeddings and Knowledge Graphs](https://mlsanity.com/paper/1803.08035/) (2018)
- [Fully Convolutional Instance-aware Semantic Segmentation](https://mlsanity.com/paper/1611.07709/) (2016)
- [A Compositional Object-Based Approach to Learning Physical Dynamics](https://mlsanity.com/paper/1612.00341/) (2016)
- [Semantic Instance Segmentation with a Discriminative Loss Function](https://mlsanity.com/paper/1708.02551/) (2017)
- [Instance Segmentation by Jointly Optimizing Spatial Embeddings and Clustering Bandwidth](https://mlsanity.com/paper/1906.11109/) (2019)
- [Unseen Object Instance Segmentation for Robotic Environments](https://mlsanity.com/paper/2007.08073/) (2020)
- [Learning to Segment Object Candidates](https://mlsanity.com/paper/1506.06204/) (2015)
- [6-DOF Grasping for Target-driven Object Manipulation in Clutter](https://mlsanity.com/paper/1912.03628/) (2019)
- [Learning to Refine Object Segments](https://mlsanity.com/paper/1603.08695/) (2016)
- [Learning RGB-D Feature Embeddings for Unseen Object Instance Segmentation](https://mlsanity.com/paper/2007.15157/) (2020)
- [Recurrent Pixel Embedding for Instance Grouping](https://mlsanity.com/paper/1712.08273/) (2017)

## Built on by (most cited first, 15 of 15)

- [Mean Shift Mask Transformer for Unseen Object Instance Segmentation](https://mlsanity.com/paper/2211.11679/) (2022)
- [ICM-3D: Instantiated Category Modeling for 3D Instance Segmentation](https://mlsanity.com/paper/2108.11771/) (2021)
- [Unseen Object Instance Segmentation with Fully Test-time RGB-D Embeddings Adaptation](https://mlsanity.com/paper/2204.09847/) (2022)
- [Smart Explorer: Recognizing Objects in Dense Clutter via Interactive Exploration](https://mlsanity.com/paper/2208.03496/) (2022)
- [Real-time Mapping of Physical Scene Properties with an Autonomous Robot Experimenter](https://mlsanity.com/paper/2210.17325/) (2022)
- [SupeRGB-D: Zero-shot Instance Segmentation in Cluttered Indoor Environments](https://mlsanity.com/paper/2212.11922/) (2022)
- [Reliable Image Dehazing by NeRF](https://mlsanity.com/paper/2303.09153/) (2023)
- [DoUnseen: Tuning-Free Class-Adaptive Object Detection of Unseen Objects for Robotic Grasping](https://mlsanity.com/paper/2304.02833/) (2023)
- [ENInst: Enhancing Weakly-supervised Low-shot Instance Segmentation](https://mlsanity.com/paper/2302.09765/) (2023)
- [Embodied Uncertainty-Aware Object Segmentation](https://mlsanity.com/paper/2408.04760/) (2024)
- [Category-level Shape Estimation for Densely Cluttered Objects](https://mlsanity.com/paper/2302.11983/) (2023)
- [High-Quality Unknown Object Instance Segmentation via Quadruple Boundary Error Refinement](https://mlsanity.com/paper/2306.16132/) (2023)
- [TMI: Text-to-Image Meets Image-to-Image for Complementary Data Synthesis to Boost Long-Tailed Instance Segmentation](https://mlsanity.com/paper/2607.08201/) (2026)
- [Self-Supervised Learning of Object Segmentation from Unlabeled RGB-D Videos](https://mlsanity.com/paper/2304.04325/) (2023)
- [Clutt3R-Seg: Sparse-view 3D Instance Segmentation for Language-grounded Grasping in Cluttered Scenes](https://mlsanity.com/paper/2602.11660/) (2026)

## Abstract (arXiv)

Segmenting unseen object instances in cluttered environments is an important capability that robots need when functioning in unstructured environments. While previous methods have exhibited promising results, they still tend to provide incorrect results in highly cluttered scenes. We postulate that a network architecture that encodes relations between objects at a high-level can be beneficial. Thus, in this work, we propose a novel framework that refines the output of such methods by utilizing a graph-based representation of instance masks. We train deep networks capable of sampling smart perturbations to the segmentations, and a graph neural network, which can encode relations between objects, to evaluate the perturbed segmentations. Our proposed method is orthogonal to previous works and achieves state-of-the-art performance when combined with them. We demonstrate an application that uses uncertainty estimates generated by our method to guide a manipulator, leading to efficient understanding of cluttered scenes. Code, models, and video can be found at https://github.com/chrisdxie/rice .