# Advancing Video Self-Supervised Learning via Image Foundation Models

arXiv: https://arxiv.org/abs/2505.19218 · HTML: https://mlsanity.com/paper/2505.19218/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Builds on 38 of its 46 references that are in the map.
- First posted on arXiv 25 May 2025.

## Builds on (most cited first, 30 of 38)

- [Learning Transferable Visual Models From Natural Language Supervision](https://mlsanity.com/paper/2103.00020/) (2021)
- [A Simple Framework for Contrastive Learning of Visual Representations](https://mlsanity.com/paper/2002.05709/) (2020)
- [DINOv2: Learning Robust Visual Features without Supervision](https://mlsanity.com/paper/2304.07193/) (2023)
- [Momentum Contrast for Unsupervised Visual Representation Learning](https://mlsanity.com/paper/1911.05722/) (2019)
- [Masked Autoencoders Are Scalable Vision Learners](https://mlsanity.com/paper/2111.06377/) (2021)
- [Bootstrap your own latent: A new approach to self-supervised Learning](https://mlsanity.com/paper/2006.07733/) (2020)
- [Going Deeper with Convolutions](https://mlsanity.com/paper/1409.4842/) (2014)
- [Quo Vadis, Action Recognition? A New Model and the Kinetics Dataset](https://mlsanity.com/paper/1705.07750/) (2017)
- [Decoupled Weight Decay Regularization](https://mlsanity.com/paper/1711.05101/) (2017)
- [Is Space-Time Attention All You Need for Video Understanding?](https://mlsanity.com/paper/2102.05095/) (2021)
- [ViViT: A Video Vision Transformer](https://mlsanity.com/paper/2103.15691/) (2021)
- [VideoMAE: Masked Autoencoders are Data-Efficient Learners for Self-Supervised Video Pre-Training](https://mlsanity.com/paper/2203.12602/) (2022)
- [A Closer Look at Spatiotemporal Convolutions for Action Recognition](https://mlsanity.com/paper/1711.11248/) (2017)
- [The "something something" video database for learning and evaluating visual common sense](https://mlsanity.com/paper/1706.04261/) (2017)
- [Video Swin Transformer](https://mlsanity.com/paper/2106.13230/) (2021)
- [VideoMAE V2: Scaling Video Masked Autoencoders with Dual Masking](https://mlsanity.com/paper/2303.16727/) (2023)
- [Rethinking Spatiotemporal Feature Learning: Speed-Accuracy Trade-offs in Video Classification](https://mlsanity.com/paper/1712.04851/) (2017)
- [Learning Spatio-Temporal Representation with Pseudo-3D Residual Networks](https://mlsanity.com/paper/1711.10305/) (2017)
- [Video Transformer Network](https://mlsanity.com/paper/2102.00719/) (2021)
- [Beyond Short Snippets: Deep Networks for Video Classification](https://mlsanity.com/paper/1503.08909/) (2015)
- [A Large-Scale Study on Unsupervised Spatiotemporal Representation Learning](https://mlsanity.com/paper/2104.14558/) (2021)
- [Training Very Deep Networks](https://mlsanity.com/paper/1507.06228/) (2015)
- [BEVT: BERT Pretraining of Video Transformers](https://mlsanity.com/paper/2112.01529/) (2021)
- [AIM: Adapting Image Models for Efficient Video Action Recognition](https://mlsanity.com/paper/2302.03024/) (2023)
- [TCLR: Temporal Contrastive Learning for Video Representation](https://mlsanity.com/paper/2101.07974/) (2021)
- [Video Playback Rate Perception for Self-supervisedSpatio-Temporal Representation Learning](https://mlsanity.com/paper/2006.11476/) (2020)
- [Self-Supervised Learning for Videos: A Survey](https://mlsanity.com/paper/2207.00419/) (2022)
- [RSPNet: Relative Speed Perception for Unsupervised Video Representation Learning](https://mlsanity.com/paper/2011.07949/) (2020)
- [Dual-path Adaptation from Image to Video Transformers](https://mlsanity.com/paper/2303.09857/) (2023)
- [ASCNet: Self-supervised Video Representation Learning with Appearance-Speed Consistency](https://mlsanity.com/paper/2106.02342/) (2021)

## Built on by (most cited first, 1 of 1)

- [TT-VidT: Decoupling the Temporal Axis for Efficient Motion-Centric Video Pretraining](https://mlsanity.com/paper/2609.33419/) (2026)

## Abstract (arXiv)

In the past decade, image foundation models (IFMs) have achieved unprecedented progress. However, the potential of directly using IFMs for video self-supervised representation learning has largely been overlooked. In this study, we propose an advancing video self-supervised learning (AdViSe) approach, aimed at significantly reducing the training overhead of video representation models using pre-trained IFMs. Specifically, we first introduce temporal modeling modules (ResNet3D) to IFMs, constructing a video representation model. We then employ a video self-supervised learning approach, playback rate perception, to train temporal modules while freezing the IFM components. Experiments on UCF101 demonstrate that AdViSe achieves performance comparable to state-of-the-art methods while reducing training time by 3.4× and GPU memory usage by 8.2×. This study offers fresh insights into low-cost video self-supervised learning based on pre-trained IFMs. Code is available at https://github.com/JingwWu/advise-video-ssl.