# Dense Unsupervised Learning for Video Segmentation

arXiv: https://arxiv.org/abs/2111.06265 · HTML: https://mlsanity.com/paper/2111.06265/

## Key facts

- Cited by 26 papers in the map (data through 14 Oct 2026).
- Builds on 46 of its 50 references that are in the map.
- First posted on arXiv 11 Nov 2021.

## Builds on (most cited first, 30 of 46)

- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [A Simple Framework for Contrastive Learning of Visual Representations](https://mlsanity.com/paper/2002.05709/) (2020)
- [Representation Learning with Contrastive Predictive Coding](https://mlsanity.com/paper/1807.03748/) (2018)
- [Momentum Contrast for Unsupervised Visual Representation Learning](https://mlsanity.com/paper/1911.05722/) (2019)
- [Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift](https://mlsanity.com/paper/1502.03167/) (2015)
- [Layer Normalization](https://mlsanity.com/paper/1607.06450/) (2016)
- [Fully Convolutional Networks for Semantic Segmentation](https://mlsanity.com/paper/1411.4038/) (2014)
- [Bootstrap your own latent: A new approach to self-supervised Learning](https://mlsanity.com/paper/2006.07733/) (2020)
- [Quo Vadis, Action Recognition? A New Model and the Kinetics Dataset](https://mlsanity.com/paper/1705.07750/) (2017)
- [Exploring Simple Siamese Representation Learning](https://mlsanity.com/paper/2011.10566/) (2020)
- [Unsupervised Learning of Visual Features by Contrasting Cluster Assignments](https://mlsanity.com/paper/2006.09882/) (2020)
- [RAFT: Recurrent All-Pairs Field Transforms for Optical Flow](https://mlsanity.com/paper/2003.12039/) (2020)
- [Barlow Twins: Self-Supervised Learning via Redundancy Reduction](https://mlsanity.com/paper/2103.03230/) (2021)
- [Deep Clustering for Unsupervised Learning of Visual Features](https://mlsanity.com/paper/1807.05520/) (2018)
- [PWC-Net: CNNs for Optical Flow Using Pyramid, Warping, and Cost Volume](https://mlsanity.com/paper/1709.02371/) (2017)
- [Self-Supervised Learning of Pretext-Invariant Representations](https://mlsanity.com/paper/1912.01991/) (2019)
- [Data-Efficient Image Recognition with Contrastive Predictive Coding](https://mlsanity.com/paper/1905.09272/) (2019)
- [Time-Contrastive Networks: Self-Supervised Learning from Video](https://mlsanity.com/paper/1704.06888/) (2017)
- [Video Object Segmentation using Space-Time Memory Networks](https://mlsanity.com/paper/1904.00607/) (2019)
- [Spatiotemporal Contrastive Video Representation Learning](https://mlsanity.com/paper/2008.03800/) (2020)
- [The Effectiveness of Data Augmentation in Image Classification using Deep Learning](https://mlsanity.com/paper/1712.04621/) (2017)
- [TrackingNet: A Large-Scale Dataset and Benchmark for Object Tracking in the Wild](https://mlsanity.com/paper/1803.10794/) (2018)
- [Self-supervised Equivariant Attention Mechanism for Weakly Supervised Semantic Segmentation](https://mlsanity.com/paper/2004.04581/) (2020)
- [Learning Correspondence from the Cycle-Consistency of Time](https://mlsanity.com/paper/1903.07593/) (2019)
- [Space-Time Correspondence as a Contrastive Random Walk](https://mlsanity.com/paper/2006.14613/) (2020)
- [One-Shot Video Object Segmentation](https://mlsanity.com/paper/1611.05198/) (2016)
- [YouTube-VOS: Sequence-to-Sequence Video Object Segmentation](https://mlsanity.com/paper/1809.00461/) (2018)
- [Self-supervised Co-training for Video Representation Learning](https://mlsanity.com/paper/2010.09709/) (2020)
- [A Large-Scale Study on Unsupervised Spatiotemporal Representation Learning](https://mlsanity.com/paper/2104.14558/) (2021)
- [UnFlow: Unsupervised Learning of Optical Flow with a Bidirectional Census Loss](https://mlsanity.com/paper/1711.07837/) (2017)

## Built on by (most cited first, 26 of 26)

- [A survey on deep learning in medical image registration: new technologies, uncertainty, evaluation metrics, and beyond](https://mlsanity.com/paper/2307.15615/) (2023)
- [Advances in Deep Concealed Scene Understanding](https://mlsanity.com/paper/2304.11234/) (2023)
- [Time Does Tell: Self-Supervised Time-Tuning of Dense Image Representations](https://mlsanity.com/paper/2308.11796/) (2023)
- [DropMAE: Learning Representations via Masked Autoencoders with Spatial-Attention Dropout for Temporal Matching Tasks](https://mlsanity.com/paper/2304.00571/) (2023)
- [Boosting Video Object Segmentation via Space-time Correspondence Learning](https://mlsanity.com/paper/2304.06211/) (2023)
- [Unified Mask Embedding and Correspondence Learning for Self-Supervised Video Segmentation](https://mlsanity.com/paper/2303.10100/) (2023)
- [Near, far: Patch-ordering enhances vision foundation models' scene understanding](https://mlsanity.com/paper/2408.11054/) (2024)
- [UVOSAM: A Mask-free Paradigm for Unsupervised Video Object Segmentation via Segment Anything Model](https://mlsanity.com/paper/2305.12659/) (2023)
- [Spatial-then-Temporal Self-Supervised Learning for Video Correspondence](https://mlsanity.com/paper/2209.07778/) (2022)
- [Demystifying Unsupervised Semantic Correspondence Estimation](https://mlsanity.com/paper/2207.05054/) (2022)
- [In-N-Out Generative Learning for Dense Unsupervised Video Segmentation](https://mlsanity.com/paper/2203.15312/) (2022)
- [Transfer of Representations to Video Label Propagation: Implementation Factors Matter](https://mlsanity.com/paper/2203.05553/) (2022)
- [MoSiC: Optimal-Transport Motion Trajectory for Dense Self-Supervised Learning](https://mlsanity.com/paper/2506.08694/) (2025)
- [Learning Fine-Grained Features for Pixel-wise Video Correspondences](https://mlsanity.com/paper/2308.03040/) (2023)
- [Unsupervised CD in satellite image time series by contrastive learning and feature tracking](https://mlsanity.com/paper/2304.11375/) (2023)
- [Overcoming Annotation Bottlenecks in Underwater Fish Segmentation: A Robust Self-Supervised Learning Approach](https://mlsanity.com/paper/2206.05390/) (2022)
- [MomentaMorph: Unsupervised Spatial-Temporal Registration with Momenta, Shooting, and Correction](https://mlsanity.com/paper/2308.02949/) (2023)
- [Unsupervised Cognition](https://mlsanity.com/paper/2409.18624/) (2024)
- [FlowFeat: Pixel-Dense Embedding of Motion Profiles](https://mlsanity.com/paper/2511.07696/) (2025)
- [How to Track and Segment Fish without Human Annotations: A Self-Supervised Deep Learning Approach](https://mlsanity.com/paper/2208.10662/) (2022)
- [AutoQ-VIS: Improving Unsupervised Video Instance Segmentation via Automatic Quality Assessment](https://mlsanity.com/paper/2508.19808/) (2025)
- [Dynamic in Static: Hybrid Visual Correspondence for Self-Supervised Video Object Segmentation](https://mlsanity.com/paper/2404.13505/) (2024)
- [One-shot Training for Video Object Segmentation](https://mlsanity.com/paper/2405.14010/) (2024)
- [Boosting Unsupervised Video Instance Segmentation with Automatic Quality-Guided Self-Training](https://mlsanity.com/paper/2512.06864/) (2025)
- [Featurising Pixels from Dynamic 3D Scenes with Linear In-Context Learners](https://mlsanity.com/paper/2604.26488/) (2026)
- [`Attention-Guided Cross-Temporal Clustering for Self-Supervised Video Object Segmentation](https://mlsanity.com/paper/2607.07230/) (2026)

## Abstract (arXiv)

We present a novel approach to unsupervised learning for video object segmentation (VOS). Unlike previous work, our formulation allows to learn dense feature representations directly in a fully convolutional regime. We rely on uniform grid sampling to extract a set of anchors and train our model to disambiguate between them on both inter- and intra-video levels. However, a naive scheme to train such a model results in a degenerate solution. We propose to prevent this with a simple regularisation scheme, accommodating the equivariance property of the segmentation task to similarity transformations. Our training objective admits efficient implementation and exhibits fast training convergence. On established VOS benchmarks, our approach exceeds the segmentation accuracy of previous work despite using significantly less training data and compute power.