# Understanding Why ViT Trains Badly on Small Datasets: An Intuitive Perspective

arXiv: https://arxiv.org/abs/2302.03751 · HTML: https://mlsanity.com/paper/2302.03751/

## Key facts

- Cited by 28 papers in the map (data through 14 Oct 2026).
- Builds on 5 of its 8 references that are in the map.
- First posted on arXiv 7 Feb 2023.

## Builds on (most cited first, 5 of 5)

- [An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale](https://mlsanity.com/paper/2010.11929/) (2020)
- [Do Vision Transformers See Like Convolutional Neural Networks?](https://mlsanity.com/paper/2108.08810/) (2021)
- [Vision Transformer for Small-Size Datasets](https://mlsanity.com/paper/2112.13492/) (2021)
- [Efficient Training of Visual Transformers with Small Datasets](https://mlsanity.com/paper/2106.03746/) (2021)
- [Hybrid BYOL-ViT: Efficient approach to deal with small datasets](https://mlsanity.com/paper/2111.04845/) (2021)

## Built on by (most cited first, 28 of 28)

- [FRoundation: Are Foundation Models Ready for Face Recognition?](https://mlsanity.com/paper/2410.23831/) (2024)
- [UniCat: Crafting a Stronger Fusion Baseline for Multimodal Re-Identification](https://mlsanity.com/paper/2310.18812/) (2023)
- [MIMIR: Masked Image Modeling for Mutual Information-based Adversarial Robustness](https://mlsanity.com/paper/2312.04960/) (2023)
- [Equipping Computational Pathology Systems with Artifact Processing Pipelines: A Showcase for Computation and Performance Trade-offs](https://mlsanity.com/paper/2403.07743/) (2024)
- [Federated Dynamical Low-Rank Training with Global Loss Convergence Guarantees](https://mlsanity.com/paper/2406.17887/) (2024)
- [Self-Supervised Ultrasound-Video Segmentation with Feature Prediction and 3D Localised Loss](https://mlsanity.com/paper/2507.18424/) (2025)
- [ADAPT: Alzheimer Diagnosis through Adaptive Profiling Transformers](https://mlsanity.com/paper/2401.06349/) (2024)
- [Understanding and Mitigating Human-Labelling Errors in Supervised Contrastive Learning](https://mlsanity.com/paper/2403.06289/) (2024)
- [Elucidating and Overcoming the Challenges of Label Noise in Supervised Contrastive Learning](https://mlsanity.com/paper/2311.16481/) (2023)
- [The Expert Knowledge combined with AI outperforms AI Alone in Seizure Onset Zone Localization using resting state fMRI](https://mlsanity.com/paper/2312.09360/) (2023)
- [D4: Text-guided diffusion model-based domain adaptive data augmentation for vineyard shoot detection](https://mlsanity.com/paper/2409.04060/) (2024)
- [An Enhanced Harmonic Densely Connected Hybrid Transformer Network Architecture for Chronic Wound Segmentation Utilising Multi-Colour Space Tensor Merging](https://mlsanity.com/paper/2410.03359/) (2024)
- [Evaluating Self-Supervised Learning in Medical Imaging: A Benchmark for Robustness, Generalizability, and Multi-Domain Impact](https://mlsanity.com/paper/2412.19124/) (2024)
- [Flat Posterior Does Matter For Bayesian Model Averaging](https://mlsanity.com/paper/2406.15664/) (2024)
- [Collaborative Knowledge Distillation via a Learning-by-Education Node Community](https://mlsanity.com/paper/2410.00074/) (2024)
- [Large Kernel MedNeXt for Breast Tumor Segmentation and Self-Normalizing Network for pCR Classification in Magnetic Resonance Images](https://mlsanity.com/paper/2508.01831/) (2025)
- [eMamba: Efficient Acceleration Framework for Mamba Models in Edge Computing](https://mlsanity.com/paper/2508.10370/) (2025)
- [ADAPT: Lightweight, Long-Range Machine Learning Force Fields Without Graphs](https://mlsanity.com/paper/2509.24115/) (2025)
- [On Measuring Localization of Shortcuts in Deep Networks](https://mlsanity.com/paper/2510.26560/) (2025)
- [Progressive Alignment with VLM-LLM Feature to Augment Defect Classification for the ASE Dataset](https://mlsanity.com/paper/2404.05183/) (2024)
- [Hybrid Deep Learning-Based for Enhanced Occlusion Segmentation in PICU Patient Monitoring](https://mlsanity.com/paper/2407.13341/) (2024)
- [Data-driven tool wear prediction in milling, based on a process-integrated single-sensor approach](https://mlsanity.com/paper/2412.19950/) (2024)
- [ConvShareViT: Enhancing Vision Transformers with Convolutional Attention Mechanisms for Free-Space Optical Accelerators](https://mlsanity.com/paper/2504.11517/) (2025)
- [Balancing Accuracy, Calibration, and Efficiency in Active Learning with Vision Transformers Under Label Noise](https://mlsanity.com/paper/2505.04375/) (2025)
- [Warming Up for Zeroth-Order Federated Pre-Training with Low Resource Clients](https://mlsanity.com/paper/2509.03503/) (2025)
- [Detection of Deployment Operational Deviations for Safety and Security of AI-Enabled Human-Centric Cyber Physical Systems](https://mlsanity.com/paper/2601.04605/) (2026)
- [FixationFormer: Direct Utilization of Expert Gaze Trajectories for Chest X-Ray Classification](https://mlsanity.com/paper/2603.22939/) (2026)
- [Forward-Only Convolutional Neural Networks with Learnable Channel-Class Assignment](https://mlsanity.com/paper/2606.09928/) (2026)

## Abstract (arXiv)

Vision transformer (ViT) is an attention neural network architecture that is shown to be effective for computer vision tasks. However, compared to ResNet-18 with a similar number of parameters, ViT has a significantly lower evaluation accuracy when trained on small datasets. To facilitate studies in related fields, we provide a visual intuition to help understand why it is the case. We first compare the performance of the two models and confirm that ViT has less accuracy than ResNet-18 when trained on small datasets. We then interpret the results by showing attention map visualization for ViT and feature map visualization for ResNet-18. The difference is further analyzed through a representation similarity perspective. We conclude that the representation of ViT trained on small datasets is hugely different from ViT trained on large datasets, which may be the reason why the performance drops a lot on small datasets.