mlsanityData to 14 Oct 2026

A Controlled Study of Self-Supervised Image and Video Pretraining under Limited Resources

arXiv 2608.13183 ↗First posted 13 Aug 2026cs.CVcited by 0

Not on the map. Why?

Abstract (arXiv)

Visual foundation models are a cornerstone of image and video understanding but typically require large amounts of data and computation. The current scale required for pretraining visual foundation models may be unsustainable or unnecessary, and significant benefits arise when effective models can be obtained with fewer resources. To better understand how self-supervised learning (SSL) objectives behave under resource constraints, we conduct a controlled study of image and video SSL objectives under matched data, architecture, and compute budgets. We compare contrastive, reconstruction, feature-prediction, and diffusion objectives and evaluate both standalone and jointly trained image-video SSL formulations across a diverse set of image and video understanding tasks. Our results show that DINOv2-style pretraining consistently provides the strongest overall performance under limited resources. Furthermore, combining DINOv2 with video SSL objectives such as VideoMAE substantially improves image classification and segmentation performance, but degrades video tracking and camera-pose estimation performance, revealing an important tradeoff between semantic and geometric representation learning. These findings suggest that combining image and video SSL objectives can be beneficial in resource-limited settings, while highlighting the need for improved methods that better balance semantic, temporal, and geometric supervision.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Builds on 35 of its 39 references that are in the map. First posted on arXiv 13 Aug 2026.

Builds on

35 of 39 references.

References · 35

  1. 2020A Simple Framework for Contrastive Learning of Visual Representationscited by 12,049
  2. 2023DINOv2: Learning Robust Visual Features without Supervisioncited by 7,974
  3. 2019Momentum Contrast for Unsupervised Visual Representation Learningcited by 7,513
  4. 2021Masked Autoencoders Are Scalable Vision Learnerscited by 7,403
  5. 2021Emerging Properties in Self-Supervised Vision Transformerscited by 6,968
  6. 2022Scalable Diffusion Models with Transformerscited by 6,054
  7. 2020Bootstrap your own latent: A new approach to self-supervised Learningcited by 4,750
  8. 2021RoFormer: Enhanced Transformer with Rotary Position Embeddingcited by 4,703
  9. 2017Quo Vadis, Action Recognition? A New Model and the Kinetics Datasetcited by 3,270
  10. 2020Exploring Simple Siamese Representation Learningcited by 2,866
  11. 2024SAM 2: Segment Anything in Images and Videoscited by 2,863
  12. 2020Unsupervised Learning of Visual Features by Contrasting Cluster Assignmentscited by 2,857
23 more
  1. 2021BEiT: BERT Pre-Training of Image Transformerscited by 2,375
  2. 2021Barlow Twins: Self-Supervised Learning via Redundancy Reductioncited by 1,838
  3. 2021An Empirical Study of Training Self-Supervised Vision Transformerscited by 1,616
  4. 2022VideoMAE: Masked Autoencoders are Data-Efficient Learners for Self-Supervised Video Pre-Trainingcited by 1,322
  5. 2021SimMIM: A Simple Framework for Masked Image Modelingcited by 1,153
  6. 2017The "something something" video database for learning and evaluating visual common sensecited by 1,103
  7. 2021VICReg: Variance-Invariance-Covariance Regularization for Self-Supervised Learningcited by 1,057
  8. 2023Self-Supervised Learning from Images with a Joint-Embedding Predictive Architecturecited by 1,039
  9. 2021iBOT: Image BERT Pre-Training with Online Tokenizercited by 903
  10. 2022data2vec: A General Framework for Self-supervised Learning in Speech, Vision and Languagecited by 727
  11. 2018Taskonomy: Disentangling Task Transfer Learningcited by 634
  12. 2021Masked Feature Prediction for Self-Supervised Visual Pre-Trainingcited by 576
  13. 2024Revisiting Feature Prediction for Learning Visual Representations from Videocited by 531
  14. 2024InternVideo2: Scaling Foundation Models for Multimodal Video Understandingcited by 317
  15. 2019A Short Note on the Kinetics-700 Human Action Datasetcited by 310
  16. 2018Spreading vectors for similarity searchcited by 118
  17. 2022Masked Video Distillation: Rethinking Masked Feature Modeling for Self-supervised Video Representation Learningcited by 90
  18. 2024Scaling 4D Representationscited by 30
  19. 2024DepthCues: Evaluating Monocular Depth Perception in Large Vision Modelscited by 19
  20. 2025From Image to Video: An Empirical Study of Diffusion Representationscited by 15
  21. 2023ViC-MAE: Self-Supervised Representation Learning from Images and Video with Contrastive Masked Autoencoderscited by 14
  22. 2026VidEoMT: Your ViT is Secretly Also a Video Segmentation Modelcited by 8
  23. 2025CrossVideoMAE: Self-Supervised Image-Video Representation Learning with Masked Autoencoderscited by 2

Built on by

Not cited yet.