mlsanityData to 7 Oct 2026

Better plain ViT baselines for ImageNet-1k

arXiv 2205.01580 ↗First posted 3 May 2022cs.CVcited by 97

Not on the map. Why?

Abstract (arXiv)

It is commonly accepted that the Vision Transformer model requires sophisticated regularization techniques to excel at ImageNet-1k scale data. Surprisingly, we find this is not the case and standard data augmentation is sufficient. This note presents a few minor modifications to the original Vision Transformer (ViT) vanilla training setting that dramatically improve the performance of plain ViT models. Notably, 90 epochs of training surpass 76% top-1 accuracy in under seven hours on a TPUv3-8, similar to the classic ResNet50 baseline, and 300 epochs of training reach 80% in less than one day.

Key facts. Cited by 97 papers in the map (data through 7 Oct 2026). Builds on 19 of its 21 references that are in the map. First posted on arXiv 3 May 2022.

Builds on

19 of 21 references.

References · 19

  1. 2015Deep Residual Learning for Image Recognitioncited by 51,809
  2. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 29,915
  3. 2021Masked Autoencoders Are Scalable Vision Learnerscited by 7,339
  4. 2017mixup: Beyond Empirical Risk Minimizationcited by 5,155
  5. 2020Training data-efficient image transformers & distillation through attentioncited by 4,560
  6. 2014Going Deeper with Convolutionscited by 4,231
  7. 2019CutMix: Regularization Strategy to Train Strong Classifiers with Localizable Featurescited by 2,875
  8. 2021An Empirical Study of Training Self-Supervised Vision Transformerscited by 1,610
  9. 2021MLP-Mixer: An all-MLP Architecture for Visioncited by 1,441
  10. 2021Scaling Vision Transformerscited by 993
  11. 2016Deep Networks with Stochastic Depthcited by 973
  12. 2019Big Transfer (BiT): General Visual Representation Learningcited by 766
7 more
  1. 2022Exploring Plain Vision Transformer Backbones for Object Detectioncited by 723
  2. 2021Do Vision Transformers See Like Convolutional Neural Networks?cited by 671
  3. 2021LiT: Zero-Shot Transfer with Locked-image text Tuningcited by 572
  4. 2021How to train your ViT? Data, Augmentation, and Regularization in Vision Transformerscited by 562
  5. 2022DeiT III: Revenge of the ViTcited by 417
  6. 2021ResNet strikes back: An improved training procedure in timmcited by 365
  7. 2022Towards Efficient and Scalable Sharpness-Aware Minimizationcited by 129

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2022: 8; 2023: 23; 2024: 31; 2025: 20; 2026: 15.

97 papers.

Sort:

By area:

  1. 2023Sigmoid Loss for Language Image Pre-Trainingcited by 3,048
  2. 2023Synthetic Data from Diffusion Models Improves ImageNet Classificationcited by 326
  3. 2022FlexiViT: One Model for All Patch Sizescited by 151
  4. 2023Getting ViT in Shape: Scaling Laws for Compute-Optimal Model Designcited by 87
  5. 2024Heavy-Tailed Class Imbalance and Why Adam Outperforms Gradient Descent on Language Modelscited by 78
  6. 2023SILC: Improving Vision Language Pretraining with Self-Distillationcited by 66
  7. 2022VeLO: Training Versatile Learned Optimizers by Scaling Upcited by 57
  8. 2022Learning to Estimate Shapley Values with Vision Transformerscited by 44
  9. 2023Replacing softmax with ReLU in Vision Transformerscited by 43
  10. 2024LocCa: Visual Pretraining with Location-aware Captionerscited by 35
  11. 2024An Image is Worth More Than 16x16 Patches: Exploring Transformers on Individual Pixelscited by 32
  12. 2023B-cos Alignment for Inherently Interpretable CNNs and Vision Transformerscited by 31
38 more
  1. 2023On the Implicit Bias of Adamcited by 25
  2. 2022No Reason for No Supervision: Improved Generalization in Supervised Modelscited by 23
  3. 2023Joint Training of Deep Ensembles Fails Due to Learner Collusioncited by 23
  4. 2022Exploring Long-Sequence Masked Autoencoderscited by 22
  5. 2023Adaptive Computation with Elastic Input Sequencecited by 22
  6. 2025Purifying Shampoo: Investigating Shampoo's Heuristics by Decomposing its Preconditionercited by 21
  7. 2022Spikeformer: A Novel Architecture for Training High-Performance Low-Latency Spiking Neural Networkcited by 19
  8. 2022RGB no more: Minimally-decoded JPEG Vision Transformerscited by 18
  9. 2023RECLIP: Resource-efficient CLIP by Training with Small Imagescited by 16
  10. 2025SapiensID: Foundation for Human Recognitioncited by 14
  11. 2023JaxPruner: A concise library for sparsity researchcited by 13
  12. 2024Momentum-SAM: Sharpness Aware Minimization without Computational Overheadcited by 13
  13. 2024Transformers Learn Low Sensitivity Functions: Investigations and Implicationscited by 13
  14. 2024Frozen Feature Augmentation for Few-Shot Image Classificationcited by 13
  15. 2024What Variables Affect Out-of-Distribution Generalization in Pretrained Models?cited by 13
  16. 2023Learning to (Learn at Test Time)cited by 12
  17. 2024B-cosification: Transforming Deep Neural Networks to be Inherently Interpretablecited by 12
  18. 2022A new hope for network model generalizationcited by 11
  19. 2025Controlling Neural Collapse Enhances Out-of-Distribution Detection and Transfer Learningcited by 11
  20. 2023A Study of Autoregressive Decoders for Multi-Tasking in Computer Visioncited by 10
  21. 2024Solving Masked Jigsaw Puzzles with Diffusion Vision Transformerscited by 10
  22. 2024Understanding Adam Requires Better Rotation Dependent Assumptionscited by 10
  23. 2023Dual PatchNormcited by 9
  24. 2024LookHere: Vision Transformers with Directed Attention Generalize and Extrapolatecited by 9
  25. 2025Recursive Inference Scaling: A Winning Path to Scalable Inference in Language and Multimodal Systemscited by 8
  26. 2023Are Vision Transformers More Data Hungry Than Newborn Visual Systems?cited by 7
  27. 2024Quantum Vision Transformers for Quark-Gluon Classificationcited by 7
  28. 2024Improving robustness to corruptions with multiplicative weight perturbationscited by 7
  29. 2025DeepCrossAttention: Supercharging Transformer Residual Connectionscited by 7
  30. 2024Orchid: Flexible and Data-Dependent Convolution for Sequence Modelingcited by 6
  31. 2024Adaptive Parametric Activation: Unifying and Generalising Activation Functions Across Taskscited by 6
  32. 2024Snuffy: Efficient Whole Slide Image Classifiercited by 6
  33. 2024MatMamba: A Matryoshka State Space Modelcited by 6
  34. 2024No More Adam: Learning Rate Scaling at Initialization is All You Needcited by 6
  35. 2023A Quadratic Synchronization Rule for Distributed Deep Learningcited by 5
  36. 2025Contrastive Forward-Forward: A Training Algorithm of Vision Transformercited by 5
  37. 2025DeBackdoor: A Deductive Framework for Detecting Backdoor Attacks on Deep Models with Limited Datacited by 5
  38. 2026Clarifying Shampoo: Adapting Spectral Descent to Stochasticity and the Parameter Trajectorycited by 5

50 most cited of 97 All 97 (JSON)