# FRAME: Pre-Training Video Feature Representations via Anticipation and Memory

arXiv: https://arxiv.org/abs/2506.05543 · HTML: https://mlsanity.com/paper/2506.05543/

## Key facts

- Cited by 2 papers in the map (data through 14 Oct 2026).
- Builds on 41 of its 49 references that are in the map.
- First posted on arXiv 5 Jun 2025.

## Builds on (most cited first, 30 of 41)

- [Learning Transferable Visual Models From Natural Language Supervision](https://mlsanity.com/paper/2103.00020/) (2021)
- [A Simple Framework for Contrastive Learning of Visual Representations](https://mlsanity.com/paper/2002.05709/) (2020)
- [Momentum Contrast for Unsupervised Visual Representation Learning](https://mlsanity.com/paper/1911.05722/) (2019)
- [Masked Autoencoders Are Scalable Vision Learners](https://mlsanity.com/paper/2111.06377/) (2021)
- [Emerging Properties in Self-Supervised Vision Transformers](https://mlsanity.com/paper/2104.14294/) (2021)
- [Exploring Simple Siamese Representation Learning](https://mlsanity.com/paper/2011.10566/) (2020)
- [Ego4D: Around the World in 3,000 Hours of Egocentric Video](https://mlsanity.com/paper/2110.07058/) (2021)
- [VideoMAE: Masked Autoencoders are Data-Efficient Learners for Self-Supervised Video Pre-Training](https://mlsanity.com/paper/2203.12602/) (2022)
- [Unsupervised Visual Representation Learning by Context Prediction](https://mlsanity.com/paper/1505.05192/) (2015)
- [The 2017 DAVIS Challenge on Video Object Segmentation](https://mlsanity.com/paper/1704.00675/) (2017)
- [Fully-Convolutional Siamese Networks for Object Tracking](https://mlsanity.com/paper/1606.09549/) (2016)
- [Time-Contrastive Networks: Self-Supervised Learning from Video](https://mlsanity.com/paper/1704.06888/) (2017)
- [YouTube-VOS: A Large-Scale Video Object Segmentation Benchmark](https://mlsanity.com/paper/1809.03327/) (2018)
- [ActionCLIP: A New Paradigm for Video Action Recognition](https://mlsanity.com/paper/2109.08472/) (2021)
- [Learning Correspondence from the Cycle-Consistency of Time](https://mlsanity.com/paper/1903.07593/) (2019)
- [Space-Time Correspondence as a Contrastive Random Walk](https://mlsanity.com/paper/2006.14613/) (2020)
- [What Should Not Be Contrastive in Contrastive Learning](https://mlsanity.com/paper/2008.05659/) (2020)
- [A Large-Scale Study on Unsupervised Spatiotemporal Representation Learning](https://mlsanity.com/paper/2104.14558/) (2021)
- [Tracking Anything with Decoupled Video Segmentation](https://mlsanity.com/paper/2309.03903/) (2023)
- [Putting the Object Back into Video Object Segmentation](https://mlsanity.com/paper/2310.12982/) (2023)
- [Fine-tuned CLIP Models are Efficient Video Learners](https://mlsanity.com/paper/2212.03640/) (2022)
- [AM-RADIO: Agglomerative Vision Foundation Model -- Reduce All Domains Into One](https://mlsanity.com/paper/2312.06709/) (2023)
- [TokenCut: Segmenting Objects in Images and Videos with Self-supervised Transformer and Normalized Cut](https://mlsanity.com/paper/2209.00383/) (2022)
- [End-to-end representation learning for Correlation Filter based tracking](https://mlsanity.com/paper/1704.06036/) (2017)
- [PIDNet: A Real-time Semantic Segmentation Network Inspired by PID Controllers](https://mlsanity.com/paper/2206.02066/) (2022)
- [Rethinking Self-supervised Correspondence Learning: A Video Frame-level Similarity Perspective](https://mlsanity.com/paper/2103.17263/) (2021)
- [Joint-task Self-supervised Learning for Temporal Correspondence](https://mlsanity.com/paper/1909.11895/) (2019)
- [Siamese Masked Autoencoders](https://mlsanity.com/paper/2305.14344/) (2023)
- [Self-supervised Video Transformer](https://mlsanity.com/paper/2112.01514/) (2021)
- [DINO-Tracker: Taming DINO for Self-Supervised Point Tracking in a Single Video](https://mlsanity.com/paper/2403.14548/) (2024)

## Built on by (most cited first, 2 of 2)

- [A Comparative Study of Vision Transformers and CNNs for Few-Shot Rigid Transformation and Fundamental Matrix Estimation](https://mlsanity.com/paper/2510.04794/) (2025)
- [TT-VidT: Decoupling the Temporal Axis for Efficient Motion-Centric Video Pretraining](https://mlsanity.com/paper/2609.33419/) (2026)

## Abstract (arXiv)

Dense video prediction tasks, such as object tracking and semantic segmentation, require video encoders that generate temporally consistent, spatially dense features for every frame. However, existing approaches fall short: image encoders like DINO or CLIP lack temporal awareness, while video models such as VideoMAE underperform compared to image encoders on dense prediction tasks. We address this gap with FRAME, a self-supervised video frame encoder tailored for dense video understanding. FRAME learns to predict current and future DINO patch features from past and present RGB frames, leading to spatially precise and temporally coherent representations. To our knowledge, FRAME is the first video encoder to leverage image-based models for dense prediction while outperforming them on tasks requiring fine-grained visual correspondence. As an auxiliary capability, FRAME aligns its class token with CLIP's semantic space, supporting language-driven tasks such as video classification. We evaluate FRAME across six dense prediction tasks on seven datasets, where it consistently outperforms image encoders and existing self-supervised video models. Despite its versatility, FRAME maintains a compact architecture suitable for a range of downstream applications.