# VideoCutLER: Surprisingly Simple Unsupervised Video Instance Segmentation

arXiv: https://arxiv.org/abs/2308.14710 · HTML: https://mlsanity.com/paper/2308.14710/

## Key facts

- Cited by 37 papers in the map (data through 14 Oct 2026).
- Builds on 33 of its 43 references that are in the map.
- First posted on arXiv 28 Aug 2023.

## Builds on (most cited first, 30 of 33)

- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale](https://mlsanity.com/paper/2010.11929/) (2020)
- [Microsoft COCO: Common Objects in Context](https://mlsanity.com/paper/1405.0312/) (2014)
- [A Simple Framework for Contrastive Learning of Visual Representations](https://mlsanity.com/paper/2002.05709/) (2020)
- [Momentum Contrast for Unsupervised Visual Representation Learning](https://mlsanity.com/paper/1911.05722/) (2019)
- [End-to-End Object Detection with Transformers](https://mlsanity.com/paper/2005.12872/) (2020)
- [Masked Autoencoders Are Scalable Vision Learners](https://mlsanity.com/paper/2111.06377/) (2021)
- [Mask R-CNN](https://mlsanity.com/paper/1703.06870/) (2017)
- [Emerging Properties in Self-Supervised Vision Transformers](https://mlsanity.com/paper/2104.14294/) (2021)
- [Bootstrap your own latent: A new approach to self-supervised Learning](https://mlsanity.com/paper/2006.07733/) (2020)
- [Unsupervised Learning of Visual Features by Contrasting Cluster Assignments](https://mlsanity.com/paper/2006.09882/) (2020)
- [RAFT: Recurrent All-Pairs Field Transforms for Optical Flow](https://mlsanity.com/paper/2003.12039/) (2020)
- [Masked-attention Mask Transformer for Universal Image Segmentation](https://mlsanity.com/paper/2112.01527/) (2021)
- [FlowNet: Learning Optical Flow with Convolutional Networks](https://mlsanity.com/paper/1504.06852/) (2015)
- [The 2017 DAVIS Challenge on Video Object Segmentation](https://mlsanity.com/paper/1704.00675/) (2017)
- [Video Instance Segmentation](https://mlsanity.com/paper/1905.04804/) (2019)
- [Space-Time Correspondence as a Contrastive Random Walk](https://mlsanity.com/paper/2006.14613/) (2020)
- [Localizing Objects with Self-Supervised Transformers and no Labels](https://mlsanity.com/paper/2109.14279/) (2021)
- [Cut and Learn for Unsupervised Object Detection and Instance Segmentation](https://mlsanity.com/paper/2301.11320/) (2023)
- [Mask2Former for Video Instance Segmentation](https://mlsanity.com/paper/2112.10764/) (2021)
- [Self-supervised Video Object Segmentation by Motion Grouping](https://mlsanity.com/paper/2104.07658/) (2021)
- [A Survey on Deep Learning Technique for Video Segmentation](https://mlsanity.com/paper/2107.01153/) (2021)
- [TokenCut: Segmenting Objects in Images and Videos with Self-supervised Transformer and Normalized Cut](https://mlsanity.com/paper/2209.00383/) (2022)
- [FreeSOLO: Learning to Segment Objects without Annotations](https://mlsanity.com/paper/2202.12181/) (2022)
- [RVOS: End-to-End Recurrent Network for Video Object Segmentation](https://mlsanity.com/paper/1903.05612/) (2019)
- [Joint-task Self-supervised Learning for Temporal Correspondence](https://mlsanity.com/paper/1909.11895/) (2019)
- [Unsupervised Moving Object Detection via Contextual Information Separation](https://mlsanity.com/paper/1901.03360/) (2019)
- [Segmenting Moving Objects via an Object-Centric Layered Representation](https://mlsanity.com/paper/2207.02206/) (2022)
- [Motion-Attentive Transition for Zero-Shot Video Object Segmentation](https://mlsanity.com/paper/2003.04253/) (2020)
- [Unsupervised Feature Learning by Cross-Level Instance-Group Discrimination](https://mlsanity.com/paper/2008.03813/) (2020)

## Built on by (most cited first, 30 of 37)

- [Moving Object Segmentation: All You Need Is SAM (and Flow)](https://mlsanity.com/paper/2404.12389/) (2024)
- [Segment Anything without Supervision](https://mlsanity.com/paper/2406.20081/) (2024)
- [RoMo: Robust Motion Segmentation Improves Structure from Motion](https://mlsanity.com/paper/2411.18650/) (2024)
- [Unsupervised Object Localization in the Era of Self-Supervised ViTs: A Survey](https://mlsanity.com/paper/2310.12904/) (2023)
- [Learning segmentation from point trajectories](https://mlsanity.com/paper/2501.12392/) (2025)
- [Can Visual Foundation Models Achieve Long-term Point Tracking?](https://mlsanity.com/paper/2408.13575/) (2024)
- [SANGRIA: Surgical Video Scene Graph Optimization for Surgical Workflow Prediction](https://mlsanity.com/paper/2407.20214/) (2024)
- [Appearance-Based Refinement for Object-Centric Motion Segmentation](https://mlsanity.com/paper/2312.11463/) (2023)
- [Vector-Quantized Vision Foundation Models for Object-Centric Learning](https://mlsanity.com/paper/2502.20263/) (2025)
- [Betrayed by Attention: A Simple yet Effective Approach for Self-supervised Video Object Segmentation](https://mlsanity.com/paper/2311.17893/) (2023)
- [Scene-Centric Unsupervised Panoptic Segmentation](https://mlsanity.com/paper/2504.01955/) (2025)
- [Rethinking Image-to-Video Adaptation: An Object-centric Perspective](https://mlsanity.com/paper/2407.06871/) (2024)
- [Multi-Scale Fusion for Object Representation](https://mlsanity.com/paper/2410.01539/) (2024)
- [HiERO: understanding the hierarchy of human behavior enhances reasoning on egocentric videos](https://mlsanity.com/paper/2505.12911/) (2025)
- [Trokens: Semantic-Aware Relational Trajectory Tokens for Few-Shot Action Recognition](https://mlsanity.com/paper/2508.03695/) (2025)
- [TRACE: Your Diffusion Model is Secretly an Instance Edge Detector](https://mlsanity.com/paper/2503.07982/) (2025)
- [Motion-Refined DINOSAUR for Unsupervised Multi-Object Discovery](https://mlsanity.com/paper/2509.02545/) (2025)
- [WildRayZer: Self-supervised Large View Synthesis in Dynamic Environments](https://mlsanity.com/paper/2601.10716/) (2026)
- [PyraTok: Language-Aligned Pyramidal Tokenizer for Video Understanding and Generation](https://mlsanity.com/paper/2601.16210/) (2026)
- [FlowCut: Unsupervised Video Instance Segmentation via Temporal Mask Matching](https://mlsanity.com/paper/2505.13174/) (2025)
- [A Comprehensive Survey on Video Scene Parsing:Advances, Challenges, and Prospects](https://mlsanity.com/paper/2506.13552/) (2025)
- [Hierarchical Visual Prompt Learning for Continual Video Instance Segmentation](https://mlsanity.com/paper/2508.08612/) (2025)
- [Unsupervised Part Discovery via Descriptor-Based Masked Image Restoration with Optimized Constraints](https://mlsanity.com/paper/2507.11985/) (2025)
- [AutoQ-VIS: Improving Unsupervised Video Instance Segmentation via Automatic Quality Assessment](https://mlsanity.com/paper/2508.19808/) (2025)
- [MapSAM2: Adapting SAM2 for Automatic Segmentation of Historical Map Images and Time Series](https://mlsanity.com/paper/2510.27547/) (2025)
- [GMOS: Grounding Moving Object Segmentation in 3D Space and Time](https://mlsanity.com/paper/2605.30352/) (2026)
- [Scene-Centric Unsupervised Video Panoptic Segmentation](https://mlsanity.com/paper/2606.04925/) (2026)
- [Improving Weakly-supervised Video Instance Segmentation by Leveraging Spatio-temporal Consistency](https://mlsanity.com/paper/2408.16661/) (2024)
- [Unsupervised Online 3D Instance Segmentation with Synthetic Sequences and Dynamic Loss](https://mlsanity.com/paper/2509.23194/) (2025)
- [The Dynamic Prior: Understanding 3D Structures for Casual Dynamic Videos](https://mlsanity.com/paper/2512.05398/) (2025)

## Abstract (arXiv)

Existing approaches to unsupervised video instance segmentation typically rely on motion estimates and experience difficulties tracking small or divergent motions. We present VideoCutLER, a simple method for unsupervised multi-instance video segmentation without using motion-based learning signals like optical flow or training on natural videos. Our key insight is that using high-quality pseudo masks and a simple video synthesis method for model training is surprisingly sufficient to enable the resulting video model to effectively segment and track multiple instances across video frames. We show the first competitive unsupervised learning results on the challenging YouTubeVIS-2019 benchmark, achieving 50.7% APvideo^50 , surpassing the previous state-of-the-art by a large margin. VideoCutLER can also serve as a strong pretrained model for supervised video instance segmentation tasks, exceeding DINO by 15.9% on YouTubeVIS-2019 in terms of APvideo.