# Learning to Better Segment Objects from Unseen Classes with Unlabeled Videos

arXiv: https://arxiv.org/abs/2104.12276 · HTML: https://mlsanity.com/paper/2104.12276/

## Key facts

- Cited by 9 papers in the map (data through 14 Oct 2026).
- Builds on 47 of its 77 references that are in the map.
- First posted on arXiv 25 Apr 2021.

## Builds on (most cited first, 30 of 47)

- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [Mask R-CNN](https://mlsanity.com/paper/1703.06870/) (2017)
- [Feature Pyramid Networks for Object Detection](https://mlsanity.com/paper/1612.03144/) (2016)
- [ShapeNet: An Information-Rich 3D Model Repository](https://mlsanity.com/paper/1512.03012/) (2015)
- [Pyramid Scene Parsing Network](https://mlsanity.com/paper/1612.01105/) (2016)
- [RAFT: Recurrent All-Pairs Field Transforms for Optical Flow](https://mlsanity.com/paper/2003.12039/) (2020)
- [The Kinetics Human Action Video Dataset](https://mlsanity.com/paper/1705.06950/) (2017)
- [A Large Dataset to Train Convolutional Networks for Disparity, Optical Flow, and Scene Flow Estimation](https://mlsanity.com/paper/1512.02134/) (2015)
- [Object-Centric Learning with Slot Attention](https://mlsanity.com/paper/2006.15055/) (2020)
- [COCO-Stuff: Thing and Stuff Classes in Context](https://mlsanity.com/paper/1612.03716/) (2016)
- [Panoptic Segmentation](https://mlsanity.com/paper/1801.00868/) (2018)
- [ResNeSt: Split-Attention Networks](https://mlsanity.com/paper/2004.08955/) (2020)
- [Video Object Segmentation using Space-Time Memory Networks](https://mlsanity.com/paper/1904.00607/) (2019)
- [Video Instance Segmentation](https://mlsanity.com/paper/1905.04804/) (2019)
- [MONet: Unsupervised Scene Decomposition and Representation](https://mlsanity.com/paper/1901.11390/) (2019)
- [Tracking without bells and whistles](https://mlsanity.com/paper/1903.05625/) (2019)
- [Multi-Object Representation Learning with Iterative Variational Inference](https://mlsanity.com/paper/1903.00450/) (2019)
- [Conditional Convolutions for Instance Segmentation](https://mlsanity.com/paper/2003.05664/) (2020)
- [SOLO: Segmenting Objects by Locations](https://mlsanity.com/paper/1912.04488/) (2019)
- [Learning Correspondence from the Cycle-Consistency of Time](https://mlsanity.com/paper/1903.07593/) (2019)
- [One-Shot Video Object Segmentation](https://mlsanity.com/paper/1611.05198/) (2016)
- [Meta R-CNN : Towards General Solver for Instance-level Few-shot Learning](https://mlsanity.com/paper/1909.13032/) (2019)
- [FEELVOS: Fast End-to-End Embedding Learning for Video Object Segmentation](https://mlsanity.com/paper/1902.09513/) (2019)
- [Learning Features by Watching Objects Move](https://mlsanity.com/paper/1612.06370/) (2016)
- [Data Distillation: Towards Omni-Supervised Learning](https://mlsanity.com/paper/1712.04440/) (2017)
- [Visual Interaction Networks](https://mlsanity.com/paper/1706.01433/) (2017)
- [SegFlow: Joint Learning for Video Object Segmentation and Optical Flow](https://mlsanity.com/paper/1709.06750/) (2017)
- [RVOS: End-to-End Recurrent Network for Video Object Segmentation](https://mlsanity.com/paper/1903.05612/) (2019)
- [Joint-task Self-supervised Learning for Temporal Correspondence](https://mlsanity.com/paper/1909.11895/) (2019)
- [FusionSeg: Learning to combine motion and appearance for fully automatic segmention of generic objects in videos](https://mlsanity.com/paper/1701.05384/) (2017)

## Built on by (most cited first, 9 of 9)

- [CNOS: A Strong Baseline for CAD-based Novel Object Segmentation](https://mlsanity.com/paper/2307.11067/) (2023)
- [PoseContrast: Class-Agnostic Object Viewpoint Estimation in the Wild with Pose-Aware Contrastive Learning](https://mlsanity.com/paper/2105.05643/) (2021)
- [PIZZA: A Powerful Image-only Zero-Shot Zero-CAD Approach to 6 DoF Tracking](https://mlsanity.com/paper/2209.07589/) (2022)
- [A Simple and Powerful Global Optimization for Unsupervised Video Object Segmentation](https://mlsanity.com/paper/2209.09341/) (2022)
- [Unifying Global-Local Representations in Salient Object Detection with Transformer](https://mlsanity.com/paper/2108.02759/) (2021)
- [Open-world Instance Segmentation: Top-down Learning with Bottom-up Supervision](https://mlsanity.com/paper/2303.05503/) (2023)
- [Conditional Latent Diffusion Models for Zero-Shot Instance Segmentation](https://mlsanity.com/paper/2508.04122/) (2025)
- [Configurable Embodied Data Generation for Class-Agnostic RGB-D Video Segmentation](https://mlsanity.com/paper/2410.12995/) (2024)
- [Template-based Object Detection Using a Foundation Model](https://mlsanity.com/paper/2603.19773/) (2026)

## Abstract (arXiv)

The ability to localize and segment objects from unseen classes would open the door to new applications, such as autonomous object learning in active vision. Nonetheless, improving the performance on unseen classes requires additional training data, while manually annotating the objects of the unseen classes can be labor-extensive and expensive. In this paper, we explore the use of unlabeled video sequences to automatically generate training data for objects of unseen classes. It is in principle possible to apply existing video segmentation methods to unlabeled videos and automatically obtain object masks, which can then be used as a training set even for classes with no manual labels available. However, our experiments show that these methods do not perform well enough for this purpose. We therefore introduce a Bayesian method that is specifically designed to automatically create such a training set: Our method starts from a set of object proposals and relies on (non-realistic) analysis-by-synthesis to select the correct ones by performing an efficient optimization over all the frames simultaneously. Through extensive experiments, we show that our method can generate a high-quality training set which significantly boosts the performance of segmenting objects of unseen classes. We thus believe that our method could open the door for open-world instance segmentation using abundant Internet videos.