mlsanityData to 14 Oct 2026

Unsupervised Transformer Pre-Training for Images: Self-Distillation, Mean Teachers, and Random Crops

arXiv 2510.03606 ↗First posted 4 Oct 2025cs.CV cs.LG eess.IVcited by 0

On the map: Knowledge distillation → General knowledge distillation methods

Abstract (arXiv)

Recent advances in self-supervised learning (SSL) have made it possible to learn general-purpose visual features that capture both the high-level semantics and the fine-grained spatial structure of images. Most notably, the recent DINOv2 has established a new state of the art by surpassing weakly supervised methods (WSL) like OpenCLIP on most benchmarks. In this survey, we examine the core ideas behind its approach, multi-crop view augmentation and self-distillation with a mean teacher, and trace their development in previous work. We then compare the performance of DINO and DINOv2 with other SSL and WSL methods across various downstream tasks, and highlight some remarkable emergent properties of their learned features with transformer backbones. We conclude by briefly discussing DINOv2's limitations, its impact, and future research directions.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Placed in General knowledge distillation methods, one of 12 directions in Knowledge distillation. Builds on 60 of its 91 references that are in the map. First posted on arXiv 4 Oct 2025.

Builds on

60 of 91 references.

References · 60

  1. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 30,186
  2. 2021Swin Transformer: Hierarchical Vision Transformer using Shifted Windowscited by 12,139
  3. 2020A Simple Framework for Contrastive Learning of Visual Representationscited by 12,049
  4. 2015Distilling the Knowledge in a Neural Networkcited by 10,324
  5. 2018Representation Learning with Contrastive Predictive Codingcited by 7,777
  6. 2019Momentum Contrast for Unsupervised Visual Representation Learningcited by 7,513
  7. 2021Masked Autoencoders Are Scalable Vision Learnerscited by 7,403
  8. 2021Emerging Properties in Self-Supervised Vision Transformerscited by 6,968
  9. 2018Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actorcited by 4,885
  10. 2020Bootstrap your own latent: A new approach to self-supervised Learningcited by 4,750
  11. 2020Training data-efficient image transformers & distillation through attentioncited by 4,586
  12. 2021Zero-Shot Text-to-Image Generationcited by 4,288
38 more
  1. 2022LAION-5B: An open large-scale dataset for training next generation image-text modelscited by 4,152
  2. 2015Continuous control with deep reinforcement learningcited by 3,407
  3. 2020Unsupervised Learning of Visual Features by Contrasting Cluster Assignmentscited by 2,857
  4. 2021BEiT: BERT Pre-Training of Image Transformerscited by 2,375
  5. 2020Improved Baselines with Momentum Contrastive Learningcited by 2,287
  6. 2021Barlow Twins: Self-Supervised Learning via Redundancy Reductioncited by 1,838
  7. 2021An Empirical Study of Training Self-Supervised Vision Transformerscited by 1,616
  8. 2018Unsupervised Representation Learning by Predicting Image Rotationscited by 1,591
  9. 2016Context Encoders: Feature Learning by Inpaintingcited by 1,428
  10. 2016Unsupervised Learning of Visual Representations by Solving Jigsaw Puzzlescited by 1,382
  11. 2020Big Self-Supervised Models are Strong Semi-Supervised Learnerscited by 1,380
  12. 2022VideoMAE: Masked Autoencoders are Data-Efficient Learners for Self-Supervised Video Pre-Trainingcited by 1,322
  13. 2019Self-training with Noisy Student improves ImageNet classificationcited by 1,311
  14. 2018Averaging Weights Leads to Wider Optima and Better Generalizationcited by 1,209
  15. 2022Reproducible scaling laws for contrastive language-image learningcited by 1,173
  16. 2016Temporal Ensembling for Semi-Supervised Learningcited by 1,057
  17. 2023Self-Supervised Learning from Images with a Joint-Embedding Predictive Architecturecited by 1,039
  18. 2018Deep Clustering for Unsupervised Learning of Visual Featurescited by 1,028
  19. 2019Self-Supervised Learning of Pretext-Invariant Representationscited by 808
  20. 2019Data-Efficient Image Recognition with Contrastive Predictive Codingcited by 673
  21. 2021RegionCLIP: Region-based Language-Image Pretrainingcited by 616
  22. 2018Exploring the Limits of Weakly Supervised Pretrainingcited by 605
  23. 2021LiT: Zero-Shot Transfer with Locked-image text Tuningcited by 577
  24. 2016Weight Normalization: A Simple Reparameterization to Accelerate Training of Deep Neural Networkscited by 543
  25. 2024Revisiting Feature Prediction for Learning Visual Representations from Videocited by 531
  26. 2019Self-labelling via simultaneous clustering and representation learningcited by 528
  27. 2019Be Your Own Teacher: Improve the Performance of Convolutional Neural Networks via Self Distillationcited by 497
  28. 2023Learning Interactive Real-World Simulatorscited by 458
  29. 2015Deep Metric Learning via Lifted Structured Feature Embeddingcited by 449
  30. 2024DINO-WM: World Models on Pre-trained Visual Features enable Zero-shot Planningcited by 386
  31. 2021Understanding Dimensional Collapse in Contrastive Self-supervised Learningcited by 378
  32. 2016Entropy-SGD: Biasing Gradient Descent Into Wide Valleyscited by 359
  33. 2022Masked Siamese Networks for Label-Efficient Learningcited by 302
  34. 2017Multi-task Self-Supervised Visual Learningcited by 200
  35. 2015Learning Visual Features from Large Weakly Supervised Datacited by 165
  36. 2021On Feature Decorrelation in Self-Supervised Learningcited by 160
  37. 2018Spreading vectors for similarity searchcited by 118
  38. 2018Noise Contrastive Estimation and Negative Sampling for Conditional Models: Consistency and Statistical Efficiencycited by 105

50 most cited of 60 All 60 (JSON)

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2026Self-supervised pretraining for an iterative image size agnostic vision transformer
  2. 2023A Simple Recipe for Competitive Low-compute Self supervised Vision Models
  3. 2022Attention Distillation: self-supervised vision transformer students need more guidance
  4. 2021Efficient Self-supervised Vision Transformers for Representation Learning
  5. 2024Weight Copy and Low-Rank Adaptation for Few-Shot Distillation of Vision Transformers
  6. 2025Object-level Self-Distillation for Vision Pretraining
  7. 2023Supervised Masked Knowledge Distillation for Few-Shot Transformers
  8. 2026Distilling Vision Transformers for Distortion-Robust Representation Learning