mlsanityData to 7 Oct 2026

An Empirical Study of Training Self-Supervised Vision Transformers

arXiv 2104.02057 ↗First posted 5 Apr 2021cs.CV cs.LGcited by 1,610

Not on the map. Why?

Cited by 45 in Diffusion models, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

This paper does not describe a novel method. Instead, it studies a straightforward, incremental, yet must-know baseline given the recent progress in computer vision: self-supervised learning for Vision Transformers (ViT). While the training recipes for standard convolutional networks have been highly mature and robust, the recipes for ViT are yet to be built, especially in the self-supervised scenarios where training becomes more challenging. In this work, we go back to basics and investigate the effects of several fundamental components for training self-supervised ViT. We observe that instability is a major issue that degrades accuracy, and it can be hidden by apparently good results. We reveal that these results are indeed partial failure, and they can be improved when training is made more stable. We benchmark ViT results in MoCo v3 and several other self-supervised frameworks, with ablations in various aspects. We discuss the currently positive evidence as well as challenges and open questions. We hope that this work will provide useful data points and experience for future research.

Key facts. Cited by 1,610 papers in the map (data through 7 Oct 2026). Builds on 35 of its 50 references that are in the map. First posted on arXiv 5 Apr 2021.

Builds on

35 of 50 references.

References · 35

  1. 2017Attention Is All You Needcited by 59,932
  2. 2015Deep Residual Learning for Image Recognitioncited by 51,809
  3. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  4. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  5. 2020Language Models are Few-Shot Learnerscited by 32,199
  6. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 29,915
  7. 2020A Simple Framework for Contrastive Learning of Visual Representationscited by 11,979
  8. 2018Representation Learning with Contrastive Predictive Codingcited by 7,662
  9. 2019Momentum Contrast for Unsupervised Visual Representation Learningcited by 7,482
  10. 2020End-to-End Object Detection with Transformerscited by 7,384
  11. 2015Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shiftcited by 6,979
  12. 2016Layer Normalizationcited by 5,579
23 more
  1. 2016SGDR: Stochastic Gradient Descent with Warm Restartscited by 4,993
  2. 2020Bootstrap your own latent: A new approach to self-supervised Learningcited by 4,712
  3. 2020Training data-efficient image transformers & distillation through attentioncited by 4,560
  4. 2020Exploring Simple Siamese Representation Learningcited by 2,853
  5. 2020Unsupervised Learning of Visual Features by Contrasting Cluster Assignmentscited by 2,845
  6. 2017Non-local Neural Networkscited by 2,290
  7. 2020Improved Baselines with Momentum Contrastive Learningcited by 2,282
  8. 2018Unsupervised Feature Learning via Non-Parametric Instance-level Discriminationcited by 1,838
  9. 2017Accurate, Large Minibatch SGD: Training ImageNet in 1 Hourcited by 1,479
  10. 2020Big Self-Supervised Models are Strong Semi-Supervised Learnerscited by 1,375
  11. 2018Learning deep representations by mutual information estimation and maximizationcited by 1,265
  12. 2019Big Transfer (BiT): General Visual Representation Learningcited by 766
  13. 2019Learning Representations by Maximizing Mutual Information Across Viewscited by 733
  14. 2019Large Batch Optimization for Deep Learning: Training BERT in 76 minutescited by 671
  15. 2019Stand-Alone Self-Attention in Vision Modelscited by 542
  16. 2016Weight Normalization: A Simple Reparameterization to Accelerate Training of Deep Neural Networkscited by 541
  17. 2017Large Batch Training of Convolutional Networkscited by 529
  18. 2019Selective Kernel Networkscited by 401
  19. 2019GCNet: Non-local Networks Meet Squeeze-Excitation Networks and Beyondcited by 331
  20. 2020Exploring Self-attention for Image Recognitioncited by 313
  21. 2014One weird trick for parallelizing convolutional neural networkscited by 264
  22. 2019Unsupervised Embedding Learning via Invariant and Spreading Instance Featurecited by 231
  23. 2017Batch Renormalization: Towards Reducing Minibatch Dependence in Batch-Normalized Modelscited by 127

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 91; 2022: 370; 2023: 398; 2024: 331; 2025: 261; 2026: 159.

1,610 papers.

Sort:

By area:

  1. 2023DINOv2: Learning Robust Visual Features without Supervisioncited by 7,764
  2. 2021Masked Autoencoders Are Scalable Vision Learnerscited by 7,339
  3. 2021BEiT: BERT Pre-Training of Image Transformerscited by 2,360
  4. 2022Visual Prompt Tuningcited by 1,674
  5. 2022VideoMAE: Masked Autoencoders are Data-Efficient Learners for Self-Supervised Video Pre-Trainingcited by 1,305
  6. 2021SimMIM: A Simple Framework for Masked Image Modelingcited by 1,147
  7. 2023Self-Supervised Learning from Images with a Joint-Embedding Predictive Architecturecited by 982
  8. 2021iBOT: Image BERT Pre-Training with Online Tokenizercited by 891
  9. 2022GLM-130B: An Open Bilingual Pre-trained Modelcited by 837
  10. 2024Cambrian-1: A Fully Open, Vision-Centric Exploration of Multimodal LLMscited by 822
  11. 2022Fine-Tuning can Distort Pretrained Features and Underperform Out-of-Distributioncited by 722
  12. 2022data2vec: A General Framework for Self-supervised Learning in Speech, Vision and Languagecited by 718
38 more
  1. 2022AdaptFormer: Adapting Vision Transformers for Scalable Visual Recognitioncited by 714
  2. 2023ConvNeXt V2: Co-designing and Scaling ConvNets with Masked Autoencoderscited by 703
  3. 2021CogView: Mastering Text-to-Image Generation via Transformerscited by 691
  4. 2022EVA: Exploring the Limits of Masked Visual Representation Learning at Scalecited by 685
  5. 2021Do Vision Transformers See Like Convolutional Neural Networks?cited by 671
  6. 2024Representation Alignment for Generation: Training Diffusion Transformers Is Easier Than You Thinkcited by 666
  7. 2024Parameter-Efficient Fine-Tuning for Large Models: A Comprehensive Surveycited by 641
  8. 2021ImageNet-21K Pretraining for the Massescited by 607
  9. 2021Masked Feature Prediction for Self-Supervised Visual Pre-Trainingcited by 573
  10. 2021LiT: Zero-Shot Transfer with Locked-image text Tuningcited by 572
  11. 2022GroupViT: Semantic Segmentation Emerges from Text Supervisioncited by 529
  12. 2022All are Worth Words: A ViT Backbone for Diffusion Modelscited by 498
  13. 2024Revisiting Feature Prediction for Learning Visual Representations from Videocited by 490
  14. 2021SLIP: Self-supervision meets Language-Image Pre-trainingcited by 434
  15. 2021VATT: Transformers for Multimodal Self-Supervised Learning from Raw Video, Audio and Textcited by 427
  16. 2021Self-Supervised Pre-Training of Swin Transformers for 3D Medical Image Analysiscited by 418
  17. 2022Diffusion Art or Digital Forgery? Investigating Data Replication in Diffusion Modelscited by 418
  18. 2022SatMAE: Pre-training Transformers for Temporal and Multi-Spectral Satellite Imagerycited by 402
  19. 2022Multimodal Learning with Transformers: A Surveycited by 399
  20. 2021Early Convolutions Help Transformers See Bettercited by 395
  21. 2023Open-Vocabulary Panoptic Segmentation with Text-to-Image Diffusion Modelscited by 371
  22. 2022Decomposing NeRF for Editing via Feature Field Distillationcited by 370
  23. 2022Masked Siamese Networks for Label-Efficient Learningcited by 301
  24. 2022BEiT v2: Masked Image Modeling with Vector-Quantized Visual Tokenizerscited by 298
  25. 2022Context Autoencoder for Self-Supervised Representation Learningcited by 297
  26. 2022MultiMAE: Multi-modal Multi-task Masked Autoencoderscited by 287
  27. 2021Focal Self-attention for Local-Global Interactions in Vision Transformerscited by 286
  28. 2022X-CLIP: End-to-End Multi-grained Contrastive Learning for Video-Text Retrievalcited by 279
  29. 2024Video Prediction Policy: A Generalist Robot Policy with Predictive Visual Representationscited by 279
  30. 2022Masked Autoencoders that Listencited by 271
  31. 2021When Vision Transformers Outperform ResNets without Pre-training or Strong Data Augmentationscited by 259
  32. 2023A Cookbook of Self-Supervised Learningcited by 248
  33. 2022Masked Visual Pre-training for Motor Controlcited by 246
  34. 2021Attention Mechanisms in Computer Vision: A Surveycited by 232
  35. 2022TinyViT: Fast Pretraining Distillation for Small Vision Transformerscited by 232
  36. 2025LeJEPA: Provable and Scalable Self-Supervised Learning Without the Heuristicscited by 227
  37. 2022MAGE: MAsked Generative Encoder to Unify Representation Learning and Image Synthesiscited by 222
  38. 2022Bridging the Gap to Real-World Object-Centric Learningcited by 219

50 most cited of 1,610 All 1,610 (JSON)