# RISeg: Robot Interactive Object Segmentation via Body Frame-Invariant Features

arXiv: https://arxiv.org/abs/2403.01731 · HTML: https://mlsanity.com/paper/2403.01731/

## Key facts

- Cited by 4 papers in the map (data through 14 Oct 2026).
- Builds on 15 of its 32 references that are in the map.
- First posted on arXiv 4 Mar 2024.

## Builds on (most cited first, 15 of 15)

- [Segment Anything](https://mlsanity.com/paper/2304.02643/) (2023)
- [RAFT: Recurrent All-Pairs Field Transforms for Optical Flow](https://mlsanity.com/paper/2003.12039/) (2020)
- [Unseen Object Instance Segmentation for Robotic Environments](https://mlsanity.com/paper/2007.08073/) (2020)
- [Interactive Perception: Leveraging Action in Perception and Perception in Action](https://mlsanity.com/paper/1604.03670/) (2016)
- [Multi-view Self-supervised Deep Learning for 6D Pose Estimation in the Amazon Picking Challenge](https://mlsanity.com/paper/1609.09475/) (2016)
- [Learning RGB-D Feature Embeddings for Unseen Object Instance Segmentation](https://mlsanity.com/paper/2007.15157/) (2020)
- [Unseen Object Amodal Instance Segmentation via Hierarchical Occlusion Modeling](https://mlsanity.com/paper/2109.11103/) (2021)
- [The Best of Both Modes: Separately Leveraging RGB and Depth for Unseen Object Instance Segmentation](https://mlsanity.com/paper/1907.13236/) (2019)
- [Mean Shift Mask Transformer for Unseen Object Instance Segmentation](https://mlsanity.com/paper/2211.11679/) (2022)
- [A Self-supervised Learning System for Object Detection using Physics Simulation and Multi-view Pose Estimation](https://mlsanity.com/paper/1703.03347/) (2017)
- [Self-Supervised Unseen Object Instance Segmentation via Long-Term Robot Interaction](https://mlsanity.com/paper/2302.03793/) (2023)
- [SceneCut: Joint Geometric and Object Segmentation for Indoor Scenes](https://mlsanity.com/paper/1709.07158/) (2017)
- [Unseen Object Instance Segmentation with Fully Test-time RGB-D Embeddings Adaptation](https://mlsanity.com/paper/2204.09847/) (2022)
- [Segmenting Unknown 3D Objects from Real Depth Images using Mask R-CNN Trained on Synthetic Data](https://mlsanity.com/paper/1809.05825/) (2018)
- [Unbiasing Semantic Segmentation For Robot Perception using Synthetic Data Feature Transfer](https://mlsanity.com/paper/1809.03676/) (2018)

## Built on by (most cited first, 4 of 4)

- [Embodied Uncertainty-Aware Object Segmentation](https://mlsanity.com/paper/2408.04760/) (2024)
- [rt-RISeg: Real-Time Model-Free Robot Interactive Segmentation for Active Instance-Level Object Understanding](https://mlsanity.com/paper/2507.10776/) (2025)
- [Proactive Scene Decomposition and Reconstruction](https://mlsanity.com/paper/2510.16272/) (2025)
- [MotionBits: Video Segmentation through Motion-Level Analysis of Rigid Bodies](https://mlsanity.com/paper/2603.06846/) (2026)

## Abstract (arXiv)

In order to successfully perform manipulation tasks in new environments, such as grasping, robots must be proficient in segmenting unseen objects from the background and/or other objects. Previous works perform unseen object instance segmentation (UOIS) by training deep neural networks on large-scale data to learn RGB/RGB-D feature embeddings, where cluttered environments often result in inaccurate segmentations. We build upon these methods and introduce a novel approach to correct inaccurate segmentation, such as under-segmentation, of static image-based UOIS masks by using robot interaction and a designed body frame-invariant feature. We demonstrate that the relative linear and rotational velocities of frames randomly attached to rigid bodies due to robot interactions can be used to identify objects and accumulate corrected object-level segmentation masks. By introducing motion to regions of segmentation uncertainty, we are able to drastically improve segmentation accuracy in an uncertainty-driven manner with minimal, non-disruptive interactions (ca. 2-3 per scene). We demonstrate the effectiveness of our proposed interactive perception pipeline in accurately segmenting cluttered scenes by achieving an average object segmentation accuracy rate of 80.7%, an increase of 28.2% when compared with other state-of-the-art UOIS methods.