mlsanityData to 7 Oct 2026

Understanding Why ViT Trains Badly on Small Datasets: An Intuitive Perspective

arXiv 2302.03751 ↗First posted 7 Feb 2023cs.CV cs.LGcited by 28

Not on the map. Why?

Abstract (arXiv)

Vision transformer (ViT) is an attention neural network architecture that is shown to be effective for computer vision tasks. However, compared to ResNet-18 with a similar number of parameters, ViT has a significantly lower evaluation accuracy when trained on small datasets. To facilitate studies in related fields, we provide a visual intuition to help understand why it is the case. We first compare the performance of the two models and confirm that ViT has less accuracy than ResNet-18 when trained on small datasets. We then interpret the results by showing attention map visualization for ViT and feature map visualization for ResNet-18. The difference is further analyzed through a representation similarity perspective. We conclude that the representation of ViT trained on small datasets is hugely different from ViT trained on large datasets, which may be the reason why the performance drops a lot on small datasets.

Key facts. Cited by 28 papers in the map (data through 7 Oct 2026). Builds on 5 of its 8 references that are in the map. First posted on arXiv 7 Feb 2023.

Builds on

5 of 8 references.

References · 5

  1. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 29,915
  2. 2021Do Vision Transformers See Like Convolutional Neural Networks?cited by 671
  3. 2021Vision Transformer for Small-Size Datasetscited by 122
  4. 2021Efficient Training of Visual Transformers with Small Datasetscited by 107
  5. 2021Hybrid BYOL-ViT: Efficient approach to deal with small datasetscited by 2

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 4; 2024: 13; 2025: 8; 2026: 3.

28 papers.

Sort:

By area:

  1. 2024FRoundation: Are Foundation Models Ready for Face Recognition?cited by 22
  2. 2023UniCat: Crafting a Stronger Fusion Baseline for Multimodal Re-Identificationcited by 18
  3. 2023MIMIR: Masked Image Modeling for Mutual Information-based Adversarial Robustnesscited by 12
  4. 2024Equipping Computational Pathology Systems with Artifact Processing Pipelines: A Showcase for Computation and Performance Trade-offscited by 9
  5. 2024Federated Dynamical Low-Rank Training with Global Loss Convergence Guaranteescited by 7
  6. 2025Self-Supervised Ultrasound-Video Segmentation with Feature Prediction and 3D Localised Losscited by 4
  7. 2024ADAPT: Alzheimer Diagnosis through Adaptive Profiling Transformerscited by 3
  8. 2024Understanding and Mitigating Human-Labelling Errors in Supervised Contrastive Learningcited by 3
  9. 2023Elucidating and Overcoming the Challenges of Label Noise in Supervised Contrastive Learningcited by 2
  10. 2023The Expert Knowledge combined with AI outperforms AI Alone in Seizure Onset Zone Localization using resting state fMRIcited by 2
  11. 2024D4: Text-guided diffusion model-based domain adaptive data augmentation for vineyard shoot detectioncited by 2
  12. 2024An Enhanced Harmonic Densely Connected Hybrid Transformer Network Architecture for Chronic Wound Segmentation Utilising Multi-Colour Space Tensor Mergingcited by 2
16 more
  1. 2024Evaluating Self-Supervised Learning in Medical Imaging: A Benchmark for Robustness, Generalizability, and Multi-Domain Impactcited by 2
  2. 2024Flat Posterior Does Matter For Bayesian Model Averagingcited by 1
  3. 2024Collaborative Knowledge Distillation via a Learning-by-Education Node Communitycited by 1
  4. 2025Large Kernel MedNeXt for Breast Tumor Segmentation and Self-Normalizing Network for pCR Classification in Magnetic Resonance Imagescited by 1
  5. 2025eMamba: Efficient Acceleration Framework for Mamba Models in Edge Computingcited by 1
  6. 2025ADAPT: Lightweight, Long-Range Machine Learning Force Fields Without Graphscited by 1
  7. 2025On Measuring Localization of Shortcuts in Deep Networkscited by 1
  8. 2024Progressive Alignment with VLM-LLM Feature to Augment Defect Classification for the ASE Datasetcited by 0
  9. 2024Hybrid Deep Learning-Based for Enhanced Occlusion Segmentation in PICU Patient Monitoringcited by 0
  10. 2024Data-driven tool wear prediction in milling, based on a process-integrated single-sensor approachcited by 0
  11. 2025ConvShareViT: Enhancing Vision Transformers with Convolutional Attention Mechanisms for Free-Space Optical Acceleratorscited by 0
  12. 2025Balancing Accuracy, Calibration, and Efficiency in Active Learning with Vision Transformers Under Label Noisecited by 0
  13. 2025Warming Up for Zeroth-Order Federated Pre-Training with Low Resource Clientscited by 0
  14. 2026Detection of Deployment Operational Deviations for Safety and Security of AI-Enabled Human-Centric Cyber Physical Systemscited by 0
  15. 2026FixationFormer: Direct Utilization of Expert Gaze Trajectories for Chest X-Ray Classificationcited by 0
  16. 2026Forward-Only Convolutional Neural Networks with Learnable Channel-Class Assignmentcited by 0