# Better plain ViT baselines for ImageNet-1k

arXiv: https://arxiv.org/abs/2205.01580 · HTML: https://mlsanity.com/paper/2205.01580/

## Key facts

- Cited by 98 papers in the map (data through 14 Oct 2026).
- Builds on 19 of its 21 references that are in the map.
- First posted on arXiv 3 May 2022.

## Builds on (most cited first, 19 of 19)

- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale](https://mlsanity.com/paper/2010.11929/) (2020)
- [Masked Autoencoders Are Scalable Vision Learners](https://mlsanity.com/paper/2111.06377/) (2021)
- [mixup: Beyond Empirical Risk Minimization](https://mlsanity.com/paper/1710.09412/) (2017)
- [Training data-efficient image transformers & distillation through attention](https://mlsanity.com/paper/2012.12877/) (2020)
- [Going Deeper with Convolutions](https://mlsanity.com/paper/1409.4842/) (2014)
- [CutMix: Regularization Strategy to Train Strong Classifiers with Localizable Features](https://mlsanity.com/paper/1905.04899/) (2019)
- [An Empirical Study of Training Self-Supervised Vision Transformers](https://mlsanity.com/paper/2104.02057/) (2021)
- [MLP-Mixer: An all-MLP Architecture for Vision](https://mlsanity.com/paper/2105.01601/) (2021)
- [Scaling Vision Transformers](https://mlsanity.com/paper/2106.04560/) (2021)
- [Deep Networks with Stochastic Depth](https://mlsanity.com/paper/1603.09382/) (2016)
- [Big Transfer (BiT): General Visual Representation Learning](https://mlsanity.com/paper/1912.11370/) (2019)
- [Exploring Plain Vision Transformer Backbones for Object Detection](https://mlsanity.com/paper/2203.16527/) (2022)
- [Do Vision Transformers See Like Convolutional Neural Networks?](https://mlsanity.com/paper/2108.08810/) (2021)
- [LiT: Zero-Shot Transfer with Locked-image text Tuning](https://mlsanity.com/paper/2111.07991/) (2021)
- [How to train your ViT? Data, Augmentation, and Regularization in Vision Transformers](https://mlsanity.com/paper/2106.10270/) (2021)
- [DeiT III: Revenge of the ViT](https://mlsanity.com/paper/2204.07118/) (2022)
- [ResNet strikes back: An improved training procedure in timm](https://mlsanity.com/paper/2110.00476/) (2021)
- [Towards Efficient and Scalable Sharpness-Aware Minimization](https://mlsanity.com/paper/2203.02714/) (2022)

## Built on by (most cited first, 30 of 98)

- [Sigmoid Loss for Language Image Pre-Training](https://mlsanity.com/paper/2303.15343/) (2023)
- [Synthetic Data from Diffusion Models Improves ImageNet Classification](https://mlsanity.com/paper/2304.08466/) (2023)
- [FlexiViT: One Model for All Patch Sizes](https://mlsanity.com/paper/2212.08013/) (2022)
- [Getting ViT in Shape: Scaling Laws for Compute-Optimal Model Design](https://mlsanity.com/paper/2305.13035/) (2023)
- [Heavy-Tailed Class Imbalance and Why Adam Outperforms Gradient Descent on Language Models](https://mlsanity.com/paper/2402.19449/) (2024)
- [SILC: Improving Vision Language Pretraining with Self-Distillation](https://mlsanity.com/paper/2310.13355/) (2023)
- [VeLO: Training Versatile Learned Optimizers by Scaling Up](https://mlsanity.com/paper/2211.09760/) (2022)
- [Learning to Estimate Shapley Values with Vision Transformers](https://mlsanity.com/paper/2206.05282/) (2022)
- [Replacing softmax with ReLU in Vision Transformers](https://mlsanity.com/paper/2309.08586/) (2023)
- [LocCa: Visual Pretraining with Location-aware Captioners](https://mlsanity.com/paper/2403.19596/) (2024)
- [An Image is Worth More Than 16x16 Patches: Exploring Transformers on Individual Pixels](https://mlsanity.com/paper/2406.09415/) (2024)
- [B-cos Alignment for Inherently Interpretable CNNs and Vision Transformers](https://mlsanity.com/paper/2306.10898/) (2023)
- [On the Implicit Bias of Adam](https://mlsanity.com/paper/2309.00079/) (2023)
- [No Reason for No Supervision: Improved Generalization in Supervised Models](https://mlsanity.com/paper/2206.15369/) (2022)
- [Joint Training of Deep Ensembles Fails Due to Learner Collusion](https://mlsanity.com/paper/2301.11323/) (2023)
- [Purifying Shampoo: Investigating Shampoo's Heuristics by Decomposing its Preconditioner](https://mlsanity.com/paper/2506.03595/) (2025)
- [Exploring Long-Sequence Masked Autoencoders](https://mlsanity.com/paper/2210.07224/) (2022)
- [Adaptive Computation with Elastic Input Sequence](https://mlsanity.com/paper/2301.13195/) (2023)
- [Spikeformer: A Novel Architecture for Training High-Performance Low-Latency Spiking Neural Network](https://mlsanity.com/paper/2211.10686/) (2022)
- [RGB no more: Minimally-decoded JPEG Vision Transformers](https://mlsanity.com/paper/2211.16421/) (2022)
- [Learning to (Learn at Test Time)](https://mlsanity.com/paper/2310.13807/) (2023)
- [RECLIP: Resource-efficient CLIP by Training with Small Images](https://mlsanity.com/paper/2304.06028/) (2023)
- [Momentum-SAM: Sharpness Aware Minimization without Computational Overhead](https://mlsanity.com/paper/2401.12033/) (2024)
- [Transformers Learn Low Sensitivity Functions: Investigations and Implications](https://mlsanity.com/paper/2403.06925/) (2024)
- [SapiensID: Foundation for Human Recognition](https://mlsanity.com/paper/2504.04708/) (2025)
- [JaxPruner: A concise library for sparsity research](https://mlsanity.com/paper/2304.14082/) (2023)
- [Frozen Feature Augmentation for Few-Shot Image Classification](https://mlsanity.com/paper/2403.10519/) (2024)
- [What Variables Affect Out-of-Distribution Generalization in Pretrained Models?](https://mlsanity.com/paper/2405.15018/) (2024)
- [B-cosification: Transforming Deep Neural Networks to be Inherently Interpretable](https://mlsanity.com/paper/2411.00715/) (2024)
- [A new hope for network model generalization](https://mlsanity.com/paper/2207.05843/) (2022)

## Abstract (arXiv)

It is commonly accepted that the Vision Transformer model requires sophisticated regularization techniques to excel at ImageNet-1k scale data. Surprisingly, we find this is not the case and standard data augmentation is sufficient. This note presents a few minor modifications to the original Vision Transformer (ViT) vanilla training setting that dramatically improve the performance of plain ViT models. Notably, 90 epochs of training surpass 76% top-1 accuracy in under seven hours on a TPUv3-8, similar to the classic ResNet50 baseline, and 300 epochs of training reach 80% in less than one day.