# Self-supervised Learning for Video Correspondence Flow

arXiv: https://arxiv.org/abs/1905.00875 · HTML: https://mlsanity.com/paper/1905.00875/

## Key facts

- Cited by 21 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 2 May 2019.

## Built on by (most cited first, 21 of 21)

- [CATs++: Boosting Cost Aggregation with Convolutions and Transformers](https://mlsanity.com/paper/2202.06817/) (2022)
- [Progressive Temporal Feature Alignment Network for Video Inpainting](https://mlsanity.com/paper/2104.03507/) (2021)
- [Dense Unsupervised Learning for Video Segmentation](https://mlsanity.com/paper/2111.06265/) (2021)
- [The Functional Correspondence Problem](https://mlsanity.com/paper/2109.01097/) (2021)
- [Contrastive Learning of Image Representations with Cross-Video Cycle-Consistency](https://mlsanity.com/paper/2105.06463/) (2021)
- [Modelling Neighbor Relation in Joint Space-Time Graph for Video Correspondence Learning](https://mlsanity.com/paper/2109.13499/) (2021)
- [Semantic-Aware Fine-Grained Correspondence](https://mlsanity.com/paper/2207.10456/) (2022)
- [Bringing Generalization to Deep Multi-View Pedestrian Detection](https://mlsanity.com/paper/2109.12227/) (2021)
- [GCoNet+: A Stronger Group Collaborative Co-Salient Object Detector](https://mlsanity.com/paper/2205.15469/) (2022)
- [Breaking Shortcut: Exploring Fully Convolutional Cycle-Consistency for Video Correspondence Learning](https://mlsanity.com/paper/2105.05838/) (2021)
- [Self-supervised Video Representation Learning with Cross-Stream Prototypical Contrasting](https://mlsanity.com/paper/2106.10137/) (2021)
- [Bridging Images and Videos: A Simple Learning Framework for Large Vocabulary Video Object Detection](https://mlsanity.com/paper/2212.10147/) (2022)
- [Transfer of Representations to Video Label Propagation: Implementation Factors Matter](https://mlsanity.com/paper/2203.05553/) (2022)
- [Self-Supervised Correspondence Estimation via Multiview Registration](https://mlsanity.com/paper/2212.03236/) (2022)
- [Box Supervised Video Segmentation Proposal Network](https://mlsanity.com/paper/2202.07025/) (2022)
- [Overcoming Annotation Bottlenecks in Underwater Fish Segmentation: A Robust Self-Supervised Learning Approach](https://mlsanity.com/paper/2206.05390/) (2022)
- [Boost Video Frame Interpolation via Motion Adaptation](https://mlsanity.com/paper/2306.13933/) (2023)
- [FlowFeat: Pixel-Dense Embedding of Motion Profiles](https://mlsanity.com/paper/2511.07696/) (2025)
- [Unveiling the Power of Self-supervision for Multi-view Multi-human Association and Tracking](https://mlsanity.com/paper/2401.17617/) (2024)
- [Personalized Video Summarization using Text-Based Queries and Conditional Modeling](https://mlsanity.com/paper/2408.14743/) (2024)
- [SWT: Self-Supervised Video Object Segmentation via Sliding, Wavelet and Transportation](https://mlsanity.com/paper/2609.31725/) (2026)

## Abstract (arXiv)

The objective of this paper is self-supervised learning of feature embeddings that are suitable for matching correspondences along the videos, which we term correspondence flow. By leveraging the natural spatial-temporal coherence in videos, we propose to train a ``pointer'' that reconstructs a target frame by copying pixels from a reference frame. We make the following contributions: First, we introduce a simple information bottleneck that forces the model to learn robust features for correspondence matching, and prevent it from learning trivial solutions, eg matching based on low-level colour information. Second, to tackle the challenges from tracker drifting, due to complex object deformations, illumination changes and occlusions, we propose to train a recursive model over long temporal windows with scheduled sampling and cycle consistency. Third, we achieve state-of-the-art performance on DAVIS 2017 video segmentation and JHMDB keypoint tracking tasks, outperforming all previous self-supervised learning approaches by a significant margin. Fourth, in order to shed light on the potential of self-supervised learning on the task of video correspondence flow, we probe the upper bound by training on additional data, ie more diverse videos, further demonstrating significant improvements on video segmentation.