# Self-supervised Transfer Learning for Instance Segmentation through Physical Interaction

arXiv: https://arxiv.org/abs/2005.09484 · HTML: https://mlsanity.com/paper/2005.09484/

## Key facts

- Cited by 13 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 19 May 2020.

## Built on by (most cited first, 13 of 13)

- [SEAL: Self-supervised Embodied Active Learning using Exploration and 3D Consistency](https://mlsanity.com/paper/2112.01001/) (2021)
- [O2O-Afford: Annotation-Free Large-Scale Object-Object Affordance Learning](https://mlsanity.com/paper/2106.15087/) (2021)
- [Act the Part: Learning Interaction Strategies for Articulated Object Part Discovery](https://mlsanity.com/paper/2105.01047/) (2021)
- [Interactron: Embodied Adaptive Object Detection](https://mlsanity.com/paper/2202.00660/) (2022)
- [Self-Supervised Unseen Object Instance Segmentation via Long-Term Robot Interaction](https://mlsanity.com/paper/2302.03793/) (2023)
- [IFR-Explore: Learning Inter-object Functional Relationships in 3D Indoor Scenes](https://mlsanity.com/paper/2112.05298/) (2021)
- [Self-Supervised Interactive Object Segmentation Through a Singulation-and-Grasping Approach](https://mlsanity.com/paper/2207.09314/) (2022)
- [Smart Explorer: Recognizing Objects in Dense Clutter via Interactive Exploration](https://mlsanity.com/paper/2208.03496/) (2022)
- [ZISVFM: Zero-Shot Object Instance Segmentation in Indoor Robotic Environments with Vision Foundation Models](https://mlsanity.com/paper/2502.03266/) (2025)
- [ALP: Action-Aware Embodied Learning for Perception](https://mlsanity.com/paper/2306.10190/) (2023)
- [Learn Fast, Segment Well: Fast Object Segmentation Learning on the iCub Robot](https://mlsanity.com/paper/2206.13462/) (2022)
- [Automatically Prepare Training Data for YOLO Using Robotic In-Hand Observation and Synthesis](https://mlsanity.com/paper/2301.01441/) (2023)
- [Self-Supervised Instance Segmentation by Grasping](https://mlsanity.com/paper/2305.06305/) (2023)

## Abstract (arXiv)

Instance segmentation of unknown objects from images is regarded as relevant for several robot skills including grasping, tracking and object sorting. Recent results in computer vision have shown that large hand-labeled datasets enable high segmentation performance. To overcome the time-consuming process of manually labeling data for new environments, we present a transfer learning approach for robots that learn to segment objects by interacting with their environment in a self-supervised manner. Our robot pushes unknown objects on a table and uses information from optical flow to create training labels in the form of object masks. To achieve this, we fine-tune an existing DeepMask network for instance segmentation on the self-labeled training data acquired by the robot. We evaluate our trained network (SelfDeepMask) on a set of real images showing challenging and cluttered scenes with novel objects. Here, SelfDeepMask outperforms the DeepMask network trained on the COCO dataset by 9.5% in average precision. Furthermore, we combine our approach with recent approaches for training with noisy labels in order to better cope with induced label noise.