# rt-RISeg: Real-Time Model-Free Robot Interactive Segmentation for Active Instance-Level Object Understanding

arXiv: https://arxiv.org/abs/2507.10776 · HTML: https://mlsanity.com/paper/2507.10776/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Builds on 16 of its 39 references that are in the map.
- First posted on arXiv 14 Jul 2025.

## Builds on (most cited first, 16 of 16)

- [Segment Anything](https://mlsanity.com/paper/2304.02643/) (2023)
- [RAFT: Recurrent All-Pairs Field Transforms for Optical Flow](https://mlsanity.com/paper/2003.12039/) (2020)
- [Unseen Object Instance Segmentation for Robotic Environments](https://mlsanity.com/paper/2007.08073/) (2020)
- [Interactive Perception: Leveraging Action in Perception and Perception in Action](https://mlsanity.com/paper/1604.03670/) (2016)
- [Multi-view Self-supervised Deep Learning for 6D Pose Estimation in the Amazon Picking Challenge](https://mlsanity.com/paper/1609.09475/) (2016)
- [Learning RGB-D Feature Embeddings for Unseen Object Instance Segmentation](https://mlsanity.com/paper/2007.15157/) (2020)
- [Unseen Object Amodal Instance Segmentation via Hierarchical Occlusion Modeling](https://mlsanity.com/paper/2109.11103/) (2021)
- [The Best of Both Modes: Separately Leveraging RGB and Depth for Unseen Object Instance Segmentation](https://mlsanity.com/paper/1907.13236/) (2019)
- [Mean Shift Mask Transformer for Unseen Object Instance Segmentation](https://mlsanity.com/paper/2211.11679/) (2022)
- [A Self-supervised Learning System for Object Detection using Physics Simulation and Multi-view Pose Estimation](https://mlsanity.com/paper/1703.03347/) (2017)
- [Self-Supervised Unseen Object Instance Segmentation via Long-Term Robot Interaction](https://mlsanity.com/paper/2302.03793/) (2023)
- [SceneCut: Joint Geometric and Object Segmentation for Indoor Scenes](https://mlsanity.com/paper/1709.07158/) (2017)
- [Unseen Object Instance Segmentation with Fully Test-time RGB-D Embeddings Adaptation](https://mlsanity.com/paper/2204.09847/) (2022)
- [Segmenting Unknown 3D Objects from Real Depth Images using Mask R-CNN Trained on Synthetic Data](https://mlsanity.com/paper/1809.05825/) (2018)
- [Unbiasing Semantic Segmentation For Robot Perception using Synthetic Data Feature Transfer](https://mlsanity.com/paper/1809.03676/) (2018)
- [RISeg: Robot Interactive Object Segmentation via Body Frame-Invariant Features](https://mlsanity.com/paper/2403.01731/) (2024)

## Built on by (most cited first, 1 of 1)

- [MotionBits: Video Segmentation through Motion-Level Analysis of Rigid Bodies](https://mlsanity.com/paper/2603.06846/) (2026)

## Abstract (arXiv)

Successful execution of dexterous robotic manipulation tasks in new environments, such as grasping, depends on the ability to proficiently segment unseen objects from the background and other objects. Previous works in unseen object instance segmentation (UOIS) train models on large-scale datasets, which often leads to overfitting on static visual features. This dependency results in poor generalization performance when confronted with out-of-distribution scenarios. To address this limitation, we rethink the task of UOIS based on the principle that vision is inherently interactive and occurs over time. We propose a novel real-time interactive perception framework, rt-RISeg, that continuously segments unseen objects by robot interactions and analysis of a designed body frame-invariant feature (BFIF). We demonstrate that the relative rotational and linear velocities of randomly sampled body frames, resulting from selected robot interactions, can be used to identify objects without any learned segmentation model. This fully self-contained segmentation pipeline generates and updates object segmentation masks throughout each robot interaction without the need to wait for an action to finish. We showcase the effectiveness of our proposed interactive perception method by achieving an average object segmentation accuracy rate 27.5% greater than state-of-the-art UOIS methods. Furthermore, although rt-RISeg is a standalone framework, we show that the autonomously generated segmentation masks can be used as prompts to vision foundation models for significantly improved performance.