# Joint-task Self-supervised Learning for Temporal Correspondence

arXiv: https://arxiv.org/abs/1909.11895 · HTML: https://mlsanity.com/paper/1909.11895/

## Key facts

- Cited by 80 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 26 Sep 2019.

## Built on by (most cited first, 30 of 80)

- [Emergent Correspondence from Image Diffusion](https://mlsanity.com/paper/2306.03881/) (2023)
- [TAP-Vid: A Benchmark for Tracking Any Point in a Video](https://mlsanity.com/paper/2211.03726/) (2022)
- [Conditional Object-Centric Learning from Video](https://mlsanity.com/paper/2111.12594/) (2021)
- [A Survey on Self-supervised Learning: Algorithms, Applications, and Future Trends](https://mlsanity.com/paper/2301.05712/) (2023)
- [Tracking Everything Everywhere All at Once](https://mlsanity.com/paper/2306.05422/) (2023)
- [A Survey on Deep Learning Technique for Video Segmentation](https://mlsanity.com/paper/2107.01153/) (2021)
- [Region Similarity Representation Learning](https://mlsanity.com/paper/2103.12902/) (2021)
- [Rethinking Self-supervised Correspondence Learning: A Video Frame-level Similarity Perspective](https://mlsanity.com/paper/2103.17263/) (2021)
- [Siamese Masked Autoencoders](https://mlsanity.com/paper/2305.14344/) (2023)
- [DINO-Tracker: Taming DINO for Self-Supervised Point Tracking in a Single Video](https://mlsanity.com/paper/2403.14548/) (2024)
- [Segmenting Moving Objects via an Object-Centric Layered Representation](https://mlsanity.com/paper/2207.02206/) (2022)
- [Do Different Tracking Tasks Require Different Appearance Models?](https://mlsanity.com/paper/2107.02156/) (2021)
- [ProposalContrast: Unsupervised Pre-training for LiDAR-based 3D Object Detection](https://mlsanity.com/paper/2207.12654/) (2022)
- [Is ImageNet worth 1 video? Learning strong image encoders from 1 long unlabelled video](https://mlsanity.com/paper/2310.08584/) (2023)
- [VideoCutLER: Surprisingly Simple Unsupervised Video Instance Segmentation](https://mlsanity.com/paper/2308.14710/) (2023)
- [Learning Pixel Trajectories with Multiscale Contrastive Random Walks](https://mlsanity.com/paper/2201.08379/) (2022)
- [Time Does Tell: Self-Supervised Time-Tuning of Dense Image Representations](https://mlsanity.com/paper/2308.11796/) (2023)
- [Self-Supervised Representation Learning from Flow Equivariance](https://mlsanity.com/paper/2101.06553/) (2021)
- [Enhancing Self-supervised Video Representation Learning via Multi-level Feature Optimization](https://mlsanity.com/paper/2108.02183/) (2021)
- [Dense Unsupervised Learning for Video Segmentation](https://mlsanity.com/paper/2111.06265/) (2021)
- [Locality-Aware Inter-and Intra-Video Reconstruction for Self-Supervised Correspondence Learning](https://mlsanity.com/paper/2203.14333/) (2022)
- [Learning to Associate Every Segment for Video Panoptic Segmentation](https://mlsanity.com/paper/2106.09453/) (2021)
- [Self-Supervised Multi-Object Tracking with Cross-Input Consistency](https://mlsanity.com/paper/2111.05943/) (2021)
- [Static and Dynamic Concepts for Self-supervised Video Representation Learning](https://mlsanity.com/paper/2207.12795/) (2022)
- [Self-Supervised Geometric Correspondence for Category-Level 6D Object Pose Estimation in the Wild](https://mlsanity.com/paper/2210.07199/) (2022)
- [DropMAE: Learning Representations via Masked Autoencoders with Spatial-Attention Dropout for Temporal Matching Tasks](https://mlsanity.com/paper/2304.00571/) (2023)
- [Learning to Track Objects from Unlabeled Videos](https://mlsanity.com/paper/2108.12711/) (2021)
- [Boosting Video Object Segmentation via Space-time Correspondence Learning](https://mlsanity.com/paper/2304.06211/) (2023)
- [Unsupervised Visual Representation Learning by Tracking Patches in Video](https://mlsanity.com/paper/2105.02545/) (2021)
- [Partially Does It: Towards Scene-Level FG-SBIR with Partial Input](https://mlsanity.com/paper/2203.14804/) (2022)

## Abstract (arXiv)

This paper proposes to learn reliable dense correspondence from videos in a self-supervised manner. Our learning process integrates two highly related tasks: tracking large image regions and establishing fine-grained pixel-level associations between consecutive video frames. We exploit the synergy between both tasks through a shared inter-frame affinity matrix, which simultaneously models transitions between video frames at both the region- and pixel-levels. While region-level localization helps reduce ambiguities in fine-grained matching by narrowing down search regions; fine-grained matching provides bottom-up features to facilitate region-level localization. Our method outperforms the state-of-the-art self-supervised methods on a variety of visual correspondence tasks, including video-object and part-segmentation propagation, keypoint tracking, and object tracking. Our self-supervised method even surpasses the fully-supervised affinity feature representation obtained from a ResNet-18 pre-trained on the ImageNet.