# Time Does Tell: Self-Supervised Time-Tuning of Dense Image Representations

arXiv: https://arxiv.org/abs/2308.11796 · HTML: https://mlsanity.com/paper/2308.11796/

## Key facts

- Cited by 35 papers in the map (data through 14 Oct 2026).
- Builds on 63 of its 76 references that are in the map.
- First posted on arXiv 22 Aug 2023.

## Builds on (most cited first, 30 of 63)

- [An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale](https://mlsanity.com/paper/2010.11929/) (2020)
- [PyTorch: An Imperative Style, High-Performance Deep Learning Library](https://mlsanity.com/paper/1912.01703/) (2019)
- [A Simple Framework for Contrastive Learning of Visual Representations](https://mlsanity.com/paper/2002.05709/) (2020)
- [DINOv2: Learning Robust Visual Features without Supervision](https://mlsanity.com/paper/2304.07193/) (2023)
- [Momentum Contrast for Unsupervised Visual Representation Learning](https://mlsanity.com/paper/1911.05722/) (2019)
- [Masked Autoencoders Are Scalable Vision Learners](https://mlsanity.com/paper/2111.06377/) (2021)
- [Emerging Properties in Self-Supervised Vision Transformers](https://mlsanity.com/paper/2104.14294/) (2021)
- [Gaussian Error Linear Units (GELUs)](https://mlsanity.com/paper/1606.08415/) (2016)
- [Unsupervised Learning of Visual Features by Contrasting Cluster Assignments](https://mlsanity.com/paper/2006.09882/) (2020)
- [Billion-scale similarity search with GPUs](https://mlsanity.com/paper/1702.08734/) (2017)
- [Sinkhorn Distances: Lightspeed Computation of Optimal Transportation Distances](https://mlsanity.com/paper/1306.0895/) (2013)
- [An Empirical Study of Training Self-Supervised Vision Transformers](https://mlsanity.com/paper/2104.02057/) (2021)
- [PWC-Net: CNNs for Optical Flow Using Pyramid, Warping, and Cost Volume](https://mlsanity.com/paper/1709.02371/) (2017)
- [The 2017 DAVIS Challenge on Video Object Segmentation](https://mlsanity.com/paper/1704.00675/) (2017)
- [iBOT: Image BERT Pre-Training with Online Tokenizer](https://mlsanity.com/paper/2111.07832/) (2021)
- [Rescaling Egocentric Vision](https://mlsanity.com/paper/2006.13256/) (2020)
- [Self-labelling via simultaneous clustering and representation learning](https://mlsanity.com/paper/1911.05371/) (2019)
- [Dense Contrastive Learning for Self-Supervised Visual Pre-Training](https://mlsanity.com/paper/2011.09157/) (2020)
- [Invariant Information Clustering for Unsupervised Image Classification and Segmentation](https://mlsanity.com/paper/1807.06653/) (2018)
- [YouTube-VOS: A Large-Scale Video Object Segmentation Benchmark](https://mlsanity.com/paper/1809.03327/) (2018)
- [SCAN: Learning to Classify Images without Labels](https://mlsanity.com/paper/2005.12320/) (2020)
- [Masked Siamese Networks for Label-Efficient Learning](https://mlsanity.com/paper/2204.07141/) (2022)
- [Propagate Yourself: Exploring Pixel-Level Consistency for Unsupervised Visual Representation Learning](https://mlsanity.com/paper/2011.10043/) (2020)
- [Learning Correspondence from the Cycle-Consistency of Time](https://mlsanity.com/paper/1903.07593/) (2019)
- [Conditional Object-Centric Learning from Video](https://mlsanity.com/paper/2111.12594/) (2021)
- [Unsupervised Semantic Segmentation by Distilling Feature Correspondences](https://mlsanity.com/paper/2203.08414/) (2022)
- [Space-Time Correspondence as a Contrastive Random Walk](https://mlsanity.com/paper/2006.14613/) (2020)
- [Bridging the Gap to Real-World Object-Centric Learning](https://mlsanity.com/paper/2209.14860/) (2022)
- [A Large-Scale Study on Unsupervised Spatiotemporal Representation Learning](https://mlsanity.com/paper/2104.14558/) (2021)
- [Unsupervised Semantic Segmentation by Contrasting Object Mask Proposals](https://mlsanity.com/paper/2102.06191/) (2021)

## Built on by (most cited first, 30 of 35)

- [V-JEPA 2.1: Unlocking Dense Features in Video Self-Supervised Learning](https://mlsanity.com/paper/2603.14482/) (2026)
- [DINO-Tracker: Taming DINO for Self-Supervised Point Tracking in a Single Video](https://mlsanity.com/paper/2403.14548/) (2024)
- [Is ImageNet worth 1 video? Learning strong image encoders from 1 long unlabelled video](https://mlsanity.com/paper/2310.08584/) (2023)
- [GeneralAD: Anomaly Detection Across Domains by Attending to Distorted Features](https://mlsanity.com/paper/2407.12427/) (2024)
- [Zero-Shot Object-Centric Representation Learning](https://mlsanity.com/paper/2408.09162/) (2024)
- [CrIBo: Self-Supervised Learning via Cross-Image Object-Level Bootstrapping](https://mlsanity.com/paper/2310.07855/) (2023)
- [Near, far: Patch-ordering enhances vision foundation models' scene understanding](https://mlsanity.com/paper/2408.11054/) (2024)
- [Exploring Temporally-Aware Features for Point Tracking](https://mlsanity.com/paper/2501.12218/) (2025)
- [Detect Everything with Few Examples](https://mlsanity.com/paper/2309.12969/) (2023)
- [SIGMA: Sinkhorn-Guided Masked Video Modeling](https://mlsanity.com/paper/2407.15447/) (2024)
- [LesionLocator: Zero-Shot Universal Tumor Segmentation and Tracking in 3D Whole-Body Imaging](https://mlsanity.com/paper/2502.20985/) (2025)
- [CTRL-O: Language-Controllable Object-Centric Visual Representation Learning](https://mlsanity.com/paper/2503.21747/) (2025)
- [INSID3: Training-Free In-Context Segmentation with DINOv3](https://mlsanity.com/paper/2603.28480/) (2026)
- [Betrayed by Attention: A Simple yet Effective Approach for Self-supervised Video Object Segmentation](https://mlsanity.com/paper/2311.17893/) (2023)
- [SMILE: Infusing Spatial and Motion Semantics in Masked Video Learning](https://mlsanity.com/paper/2504.00527/) (2025)
- [No time to train! Training-Free Reference-Based Instance Segmentation](https://mlsanity.com/paper/2507.02798/) (2025)
- [Bootstrap Masked Visual Modeling via Hard Patches Mining](https://mlsanity.com/paper/2312.13714/) (2023)
- [MoSiC: Optimal-Transport Motion Trajectory for Dense Self-Supervised Learning](https://mlsanity.com/paper/2506.08694/) (2025)
- [LayerLock: Non-collapsing Representation Learning with Progressive Freezing](https://mlsanity.com/paper/2509.10156/) (2025)
- [Crane: Context-Guided Prompt Learning and Attention Refinement for Zero-Shot Anomaly Detection](https://mlsanity.com/paper/2504.11055/) (2025)
- [DIP: Unsupervised Dense In-Context Post-training of Visual Representations](https://mlsanity.com/paper/2506.18463/) (2025)
- [FRAME: Pre-Training Video Feature Representations via Anticipation and Memory](https://mlsanity.com/paper/2506.05543/) (2025)
- [Human Gaze Boosts Object-Centered Representation Learning](https://mlsanity.com/paper/2501.02966/) (2025)
- [Structured-Noise Masked Modeling for Video, Audio and Beyond](https://mlsanity.com/paper/2503.16311/) (2025)
- [Evaluating Foundation Models' 3D Understanding Through Multi-View Correspondence Analysis](https://mlsanity.com/paper/2512.11574/) (2025)
- [Label-supervised surgical instrument segmentation using temporal equivariance and semantic continuity](https://mlsanity.com/paper/2403.09551/) (2024)
- [Redefining Normal: A Novel Object-Level Approach for Multi-Object Novelty Detection](https://mlsanity.com/paper/2412.11148/) (2024)
- [Object Concepts Emerge from Motion](https://mlsanity.com/paper/2505.21635/) (2025)
- [VESSA: Video-based objEct-centric Self-Supervised Adaptation for Visual Foundation Models](https://mlsanity.com/paper/2510.20994/) (2025)
- [Temporal Slowness in Central Vision Drives Semantic Object Learning](https://mlsanity.com/paper/2602.04462/) (2026)

## Abstract (arXiv)

Spatially dense self-supervised learning is a rapidly growing problem domain with promising applications for unsupervised segmentation and pretraining for dense downstream tasks. Despite the abundance of temporal data in the form of videos, this information-rich source has been largely overlooked. Our paper aims to address this gap by proposing a novel approach that incorporates temporal consistency in dense self-supervised learning. While methods designed solely for images face difficulties in achieving even the same performance on videos, our method improves not only the representation quality for videos-but also images. Our approach, which we call time-tuning, starts from image-pretrained models and fine-tunes them with a novel self-supervised temporal-alignment clustering loss on unlabeled videos. This effectively facilitates the transfer of high-level information from videos to image representations. Time-tuning improves the state-of-the-art by 8-10% for unsupervised semantic segmentation on videos and matches it for images. We believe this method paves the way for further self-supervised scaling by leveraging the abundant availability of videos. The implementation can be found here : https://github.com/SMSD75/Timetuning