# When the Future Becomes the Past: Taming Temporal Correspondence for Self-supervised Video Representation Learning

arXiv: https://arxiv.org/abs/2503.15096 · HTML: https://mlsanity.com/paper/2503.15096/

## Key facts

- Cited by 14 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 19 Mar 2025.

## Built on by (most cited first, 14 of 14)

- [Fostering Video Reasoning via Next-Event Prediction](https://mlsanity.com/paper/2505.22457/) (2025)
- [Rethinking Reward Signals in Video GRPO: When Scores Become Targets](https://mlsanity.com/paper/2511.19356/) (2025)
- [PATS: Proficiency-Aware Temporal Sampling for Multi-View Sports Skill Assessment](https://mlsanity.com/paper/2506.04996/) (2025)
- [Exploring Structural Degradation in Dense Representations for Self-supervised Learning](https://mlsanity.com/paper/2510.17299/) (2025)
- [Semantic Concentration for Self-Supervised Dense Representations Learning](https://mlsanity.com/paper/2509.09429/) (2025)
- [Guiding Diffusion-based Reconstruction with Contrastive Signals for Balanced Visual Representation](https://mlsanity.com/paper/2603.04803/) (2026)
- [Self-supervised Representation Learning with Local Aggregation for Image-based Profiling](https://mlsanity.com/paper/2506.14265/) (2025)
- [Self-Supervised Learning from Structural Invariance](https://mlsanity.com/paper/2602.02381/) (2026)
- [Training-Free Composed Video Retrieval via Visual Representation-Guided Video-LLM Reasoning](https://mlsanity.com/paper/2606.02321/) (2026)
- [A Unified Framework for Stealthy Adversarial Generation via Latent Optimization and Transferability Enhancement](https://mlsanity.com/paper/2506.23676/) (2025)
- [A Stitch in Time: Learning Procedural Workflow via Self-Supervised Plackett-Luce Ranking](https://mlsanity.com/paper/2511.17805/) (2025)
- [VINO: Video-driven Invariance for Non-contextual Objects via Structural Prior Guided De-contextualization](https://mlsanity.com/paper/2603.07222/) (2026)
- [Focus-to-Perceive Representation Learning: A Cognition-Inspired Hierarchical Framework for Endoscopic Video Analysis](https://mlsanity.com/paper/2603.25778/) (2026)
- [ViTAL-X: Video-Text Alignment with Cross-Modal Temporal Edits](https://mlsanity.com/paper/2609.00505/) (2026)

## Abstract (arXiv)

The past decade has witnessed notable achievements in self-supervised learning for video tasks. Recent efforts typically adopt the Masked Video Modeling (MVM) paradigm, leading to significant progress on multiple video tasks. However, two critical challenges remain: 1) Without human annotations, the random temporal sampling introduces uncertainty, increasing the difficulty of model training. 2) Previous MVM methods primarily recover the masked patches in the pixel space, leading to insufficient information compression for downstream tasks. To address these challenges jointly, we propose a self-supervised framework that leverages Temporal Correspondence for video Representation learning (T-CoRe). For challenge 1), we propose a sandwich sampling strategy that selects two auxiliary frames to reduce reconstruction uncertainty in a two-side-squeezing manner. Addressing challenge 2), we introduce an auxiliary branch into a self-distillation architecture to restore representations in the latent space, generating high-level semantic representations enriched with temporal information. Experiments of T-CoRe consistently present superior performance across several downstream tasks, demonstrating its effectiveness for video representation learning. The code is available at https://github.com/yafeng19/T-CORE.