mlsanityData to 7 Oct 2026

Rethinking Self-supervised Correspondence Learning: A Video Frame-level Similarity Perspective

arXiv 2103.17263 ↗First posted 31 Mar 2021cs.CVcited by 81

Not on the map. Why?

Abstract (arXiv)

Learning a good representation for space-time correspondence is the key for various computer vision tasks, including tracking object bounding boxes and performing video object pixel segmentation. To learn generalizable representation for correspondence in large-scale, a variety of self-supervised pretext tasks are proposed to explicitly perform object-level or patch-level similarity learning. Instead of following the previous literature, we propose to learn correspondence using Video Frame-level Similarity (VFS) learning, i.e, simply learning from comparing video frames. Our work is inspired by the recent success in image-level contrastive learning and similarity learning for visual recognition. Our hypothesis is that if the representation is good for recognition, it requires the convolutional features to find correspondence between similar objects or parts. Our experiments show surprising results that VFS surpasses state-of-the-art self-supervised approaches for both OTB visual object tracking and DAVIS video object segmentation. We perform detailed analysis on what matters in VFS and reveals new properties on image and frame level similarity learning. Project page with code is available at https://jerryxu.net/VFS

Key facts. Cited by 81 papers in the map (data through 7 Oct 2026). Builds on 54 of its 85 references that are in the map. First posted on arXiv 31 Mar 2021.

Builds on

54 of 85 references.

References · 54

  1. 2015Deep Residual Learning for Image Recognitioncited by 51,809
  2. 2020A Simple Framework for Contrastive Learning of Visual Representationscited by 11,979
  3. 2018Representation Learning with Contrastive Predictive Codingcited by 7,662
  4. 2019Momentum Contrast for Unsupervised Visual Representation Learningcited by 7,482
  5. 2016SGDR: Stochastic Gradient Descent with Warm Restartscited by 4,993
  6. 2020Bootstrap your own latent: A new approach to self-supervised Learningcited by 4,712
  7. 2017Quo Vadis, Action Recognition? A New Model and the Kinetics Datasetcited by 3,259
  8. 2020Exploring Simple Siamese Representation Learningcited by 2,853
  9. 2020Unsupervised Learning of Visual Features by Contrasting Cluster Assignmentscited by 2,845
  10. 2020Improved Baselines with Momentum Contrastive Learningcited by 2,282
  11. 2013Visualizing and Understanding Convolutional Networkscited by 2,278
  12. 2017The Kinetics Human Action Video Datasetcited by 1,894
38 more
  1. 2018Unsupervised Feature Learning via Non-Parametric Instance-level Discriminationcited by 1,838
  2. 2018Unsupervised Representation Learning by Predicting Image Rotationscited by 1,586
  3. 2016Context Encoders: Feature Learning by Inpaintingcited by 1,424
  4. 2016Unsupervised Learning of Visual Representations by Solving Jigsaw Puzzlescited by 1,379
  5. 2019Contrastive Multiview Codingcited by 1,297
  6. 2016Colorful Image Colorizationcited by 1,113
  7. 2015Unsupervised Visual Representation Learning by Context Predictioncited by 1,062
  8. 2015FlowNet: Learning Optical Flow with Convolutional Networkscited by 1,054
  9. 2017PWC-Net: CNNs for Optical Flow Using Pyramid, Warping, and Cost Volumecited by 966
  10. 2017The 2017 DAVIS Challenge on Video Object Segmentationcited by 942
  11. 2020What Makes for Good Views for Contrastive Learning?cited by 843
  12. 2019Self-Supervised Learning of Pretext-Invariant Representationscited by 806
  13. 2016FlowNet 2.0: Evolution of Optical Flow Estimation with Deep Networkscited by 780
  14. 2019Learning Representations by Maximizing Mutual Information Across Viewscited by 733
  15. 2019Data-Efficient Image Recognition with Contrastive Predictive Codingcited by 672
  16. 2016Fully-Convolutional Siamese Networks for Object Trackingcited by 566
  17. 2017Rethinking Spatiotemporal Feature Learning: Speed-Accuracy Trade-offs in Video Classificationcited by 521
  18. 2018GOT-10k: A Large High-Diversity Benchmark for Generic Object Tracking in the Wildcited by 471
  19. 2015Unsupervised Learning of Video Representations using LSTMscited by 434
  20. 2016Optical Flow Estimation using a Spatial Pyramid Networkcited by 421
  21. 2019Tracking without bells and whistlescited by 382
  22. 2015Deep multi-scale video prediction beyond mean square errorcited by 297
  23. 2019Learning Correspondence from the Cycle-Consistency of Timecited by 269
  24. 2020Space-Time Correspondence as a Contrastive Random Walkcited by 239
  25. 2016One-Shot Video Object Segmentationcited by 233
  26. 2019Unsupervised Embedding Learning via Invariant and Spreading Instance Featurecited by 231
  27. 2018Tracking Emerges by Colorizing Videoscited by 166
  28. 2020Demystifying Contrastive Self-Supervised Learning: Invariances, Augmentations and Dataset Biasescited by 143
  29. 2016Learning Features by Watching Objects Movecited by 138
  30. 2020MAST: A Memory-Augmented Self-supervised Trackercited by 109
  31. 2015Learning to See by Movingcited by 101
  32. 2018Efficient Video Object Segmentation via Network Modulationcited by 98
  33. 2017End-to-end representation learning for Correlation Filter based trackingcited by 90
  34. 2019Joint-task Self-supervised Learning for Temporal Correspondencecited by 80
  35. 2019Unsupervised Deep Trackingcited by 74
  36. 2020SeCo: Exploring Sequence Supervision for Unsupervised Representation Learningcited by 65
  37. 2020Watching the World Go By: Representation Learning from Unlabeled Videoscited by 59
  38. 2014Unsupervised Learning of Spatiotemporally Coherent Metricscited by 45

50 most cited of 54 All 54 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 7; 2022: 20; 2023: 20; 2024: 13; 2025: 18; 2026: 3.

81 papers.

Sort:

By area:

  1. 2023Emergent Correspondence from Image Diffusioncited by 437
  2. 2022Kubric: A scalable dataset generatorcited by 373
  3. 2023TAPIR: Tracking Any Point with per-frame Initialization and temporal Refinementcited by 281
  4. 2022TAP-Vid: A Benchmark for Tracking Any Point in a Videocited by 258
  5. 2022Particle Video Revisited: Tracking Through Occlusions Using Point Trajectoriescited by 215
  6. 2023Tracking Everything Everywhere All at Oncecited by 190
  7. 2024Streaming Long Video Understanding with Large Language Modelscited by 164
  8. 2023Siamese Masked Autoencoderscited by 75
  9. 2024DINO-Tracker: Taming DINO for Self-Supervised Point Tracking in a Single Videocited by 64
  10. 2021Do Different Tracking Tasks Require Different Appearance Models?cited by 62
  11. 2023MC-JEPA: A Joint-Embedding Predictive Architecture for Self-Supervised Learning of Motion and Content Featurescited by 53
  12. 2024Matching Anything by Segmenting Anythingcited by 49
38 more
  1. 2024TAPTR: Tracking Any Point with Transformers as Detectioncited by 43
  2. 2021TransFGU: A Top-down Approach to Fine-Grained Unsupervised Semantic Segmentationcited by 38
  3. 2022Learning Pixel Trajectories with Multiscale Contrastive Random Walkscited by 36
  4. 2025Emergent Temporal Correspondences from Video Diffusion Transformerscited by 30
  5. 2022Locality-Aware Inter-and Intra-Video Reconstruction for Self-Supervised Correspondence Learningcited by 26
  6. 2022Self-Supervised Geometric Correspondence for Category-Level 6D Object Pose Estimation in the Wildcited by 22
  7. 2023DropMAE: Learning Representations via Masked Autoencoders with Spatial-Attention Dropout for Temporal Matching Taskscited by 22
  8. 2023For Pre-Trained Vision Models in Motor Control, Not All Policy Learning Methods are Created Equalcited by 22
  9. 2023Boosting Video Object Segmentation via Space-time Correspondence Learningcited by 21
  10. 2022Self-supervised video pretraining yields robust and more human-aligned visual representationscited by 20
  11. 2023Semantics Meets Temporal Correspondence: Self-supervised Object-centric Learning in Videoscited by 20
  12. 2021Contrastive Learning of Image Representations with Cross-Video Cycle-Consistencycited by 19
  13. 2023Deep Learning Technique for Human Parsing: A Survey and Outlookcited by 18
  14. 2022CoordGAN: Self-Supervised Dense Correspondences Emerge from GANscited by 17
  15. 2023Unified Mask Embedding and Correspondence Learning for Self-Supervised Video Segmentationcited by 16
  16. 2021Contextualized Spatio-Temporal Contrastive Learning with Self-Supervisioncited by 15
  17. 2022Time to augment self-supervised visual representation learningcited by 15
  18. 2025Exploring Temporally-Aware Features for Point Trackingcited by 15
  19. 2022Motion-inductive Self-supervised Object Discovery in Videoscited by 14
  20. 2023ViC-MAE: Self-Supervised Representation Learning from Images and Video with Contrastive Masked Autoencoderscited by 14
  21. 2022Semantic-Aware Fine-Grained Correspondencecited by 13
  22. 2022Spatial-then-Temporal Self-Supervised Learning for Video Correspondencecited by 13
  23. 2023HyRSM++: Hybrid Relation Guided Temporal Set Matching for Few-shot Action Recognitioncited by 12
  24. 2025Predicting Video Slot Attention Queries from Random Slot-Feature Pairscited by 12
  25. 2023Betrayed by Attention: A Simple yet Effective Approach for Self-supervised Video Object Segmentationcited by 10
  26. 2024Visual Representation Learning with Stochastic Frame Predictioncited by 10
  27. 2024Self-supervised visual learning from interactions with objectscited by 10
  28. 2024Self-Supervised Any-Point Tracking by Contrastive Random Walkscited by 10
  29. 2022Scene Consistency Representation Learning for Video Scene Segmentationcited by 9
  30. 2022Learning Shadow Correspondence for Video Shadow Detectioncited by 9
  31. 2023Self-Supervised Representation Learning from Temporal Ordering of Automated Driving Sequencescited by 9
  32. 2023Context-PIPs: Persistent Independent Particles Demands Spatial Context Featurescited by 9
  33. 2025Recurrent Video Masked Autoencoderscited by 9
  34. 2022Correspondence Distillation from NeRF-based GANcited by 8
  35. 2022In-N-Out Generative Learning for Dense Unsupervised Video Segmentationcited by 7
  36. 2022Bridging Images and Videos: A Simple Learning Framework for Large Vocabulary Video Object Detectioncited by 6
  37. 2023Doduo: Learning Dense Visual Correspondence from Unsupervised Semantic-Aware Flowcited by 6
  38. 2021Breaking Shortcut: Exploring Fully Convolutional Cycle-Consistency for Video Correspondence Learningcited by 5

50 most cited of 81 All 81 (JSON)