# How to Train Vision Transformer on Small-scale Datasets?

arXiv: https://arxiv.org/abs/2210.07240 · HTML: https://mlsanity.com/paper/2210.07240/

## Key facts

- Cited by 35 papers in the map (data through 14 Oct 2026).
- Builds on 58 of its 69 references that are in the map.
- First posted on arXiv 13 Oct 2022.

## Builds on (most cited first, 30 of 58)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale](https://mlsanity.com/paper/2010.11929/) (2020)
- [PyTorch: An Imperative Style, High-Performance Deep Learning Library](https://mlsanity.com/paper/1912.01703/) (2019)
- [Swin Transformer: Hierarchical Vision Transformer using Shifted Windows](https://mlsanity.com/paper/2103.14030/) (2021)
- [A Simple Framework for Contrastive Learning of Visual Representations](https://mlsanity.com/paper/2002.05709/) (2020)
- [End-to-End Object Detection with Transformers](https://mlsanity.com/paper/2005.12872/) (2020)
- [Masked Autoencoders Are Scalable Vision Learners](https://mlsanity.com/paper/2111.06377/) (2021)
- [Emerging Properties in Self-Supervised Vision Transformers](https://mlsanity.com/paper/2104.14294/) (2021)
- [Densely Connected Convolutional Networks](https://mlsanity.com/paper/1608.06993/) (2016)
- [EfficientNet: Rethinking Model Scaling for Convolutional Neural Networks](https://mlsanity.com/paper/1905.11946/) (2019)
- [Rethinking the Inception Architecture for Computer Vision](https://mlsanity.com/paper/1512.00567/) (2015)
- [SGDR: Stochastic Gradient Descent with Warm Restarts](https://mlsanity.com/paper/1608.03983/) (2016)
- [Bootstrap your own latent: A new approach to self-supervised Learning](https://mlsanity.com/paper/2006.07733/) (2020)
- [Training data-efficient image transformers & distillation through attention](https://mlsanity.com/paper/2012.12877/) (2020)
- [Going Deeper with Convolutions](https://mlsanity.com/paper/1409.4842/) (2014)
- [Delving Deep into Rectifiers: Surpassing Human-Level Performance on ImageNet Classification](https://mlsanity.com/paper/1502.01852/) (2015)
- [Deformable DETR: Deformable Transformers for End-to-End Object Detection](https://mlsanity.com/paper/2010.04159/) (2020)
- [CutMix: Regularization Strategy to Train Strong Classifiers with Localizable Features](https://mlsanity.com/paper/1905.04899/) (2019)
- [Benchmarking Neural Network Robustness to Common Corruptions and Perturbations](https://mlsanity.com/paper/1903.12261/) (2019)
- [RandAugment: Practical automated data augmentation with a reduced search space](https://mlsanity.com/paper/1909.13719/) (2019)
- [SwinIR: Image Restoration Using Swin Transformer](https://mlsanity.com/paper/2108.10257/) (2021)
- [Vision Transformers for Dense Prediction](https://mlsanity.com/paper/2103.13413/) (2021)
- [Fine-Grained Visual Classification of Aircraft](https://mlsanity.com/paper/1306.5151/) (2013)
- [An Empirical Study of Training Self-Supervised Vision Transformers](https://mlsanity.com/paper/2104.02057/) (2021)
- [Point Transformer](https://mlsanity.com/paper/2011.00931/) (2020)
- [Deep High-Resolution Representation Learning for Visual Recognition](https://mlsanity.com/paper/1908.07919/) (2019)
- [Random Erasing Data Augmentation](https://mlsanity.com/paper/1708.04896/) (2017)

## Built on by (most cited first, 30 of 35)

- [Mimetic Initialization of Self-Attention Layers](https://mlsanity.com/paper/2305.09828/) (2023)
- [Initializing Models with Larger Ones](https://mlsanity.com/paper/2311.18823/) (2023)
- [Diffusion-Based Neural Network Weights Generation](https://mlsanity.com/paper/2402.18153/) (2024)
- [Learngene: Inheriting Condensed Knowledge from the Ancestry Model to Descendant Models](https://mlsanity.com/paper/2305.02279/) (2023)
- [Supervised Masked Knowledge Distillation for Few-Shot Transformers](https://mlsanity.com/paper/2303.15466/) (2023)
- [MPCViT: Searching for Accurate and Efficient MPC-Friendly Vision Transformer with Heterogeneous Attention](https://mlsanity.com/paper/2211.13955/) (2022)
- [Scalarization for Multi-Task and Multi-Domain Learning at Scale](https://mlsanity.com/paper/2310.08910/) (2023)
- [Limited Data, Unlimited Potential: A Study on ViTs Augmented by Masked Autoencoders](https://mlsanity.com/paper/2310.20704/) (2023)
- [You Don't Need Domain-Specific Data Augmentations When Scaling Self-Supervised Learning](https://mlsanity.com/paper/2406.09294/) (2024)
- [Frozen Feature Augmentation for Few-Shot Image Classification](https://mlsanity.com/paper/2403.10519/) (2024)
- [TULIP: Transformer for Upsampling of LiDAR Point Clouds](https://mlsanity.com/paper/2312.06733/) (2023)
- [Neural Clustering based Visual Representation Learning](https://mlsanity.com/paper/2403.17409/) (2024)
- [Forest2Seq: Revitalizing Order Prior for Sequential Indoor Scene Synthesis](https://mlsanity.com/paper/2407.05388/) (2024)
- [Astroformer: More Data Might not be all you need for Classification](https://mlsanity.com/paper/2304.05350/) (2023)
- [Mitigating the Effect of Incidental Correlations on Part-based Learning](https://mlsanity.com/paper/2310.00377/) (2023)
- [Convolutional Initialization for Data-Efficient Vision Transformers](https://mlsanity.com/paper/2401.12511/) (2024)
- [Approximating Metric Magnitude of Point Sets](https://mlsanity.com/paper/2409.04411/) (2024)
- [CTA-Net: A CNN-Transformer Aggregation Network for Improving Multi-Scale Feature Extraction](https://mlsanity.com/paper/2410.11428/) (2024)
- [YOLOPose V2: Understanding and Improving Transformer-based 6D Pose Estimation](https://mlsanity.com/paper/2307.11550/) (2023)
- [Paired Conditional Generative Adversarial Network for Highly Accelerated Liver 4D MRI](https://mlsanity.com/paper/2405.12357/) (2024)
- [NeurLZ: An Online Neural Learning-Based Method to Enhance Scientific Lossy Compression](https://mlsanity.com/paper/2409.05785/) (2024)
- [Self-supervised Auxiliary Learning for Texture and Model-based Hybrid Robust and Fair Featuring in Face Analysis](https://mlsanity.com/paper/2409.19582/) (2024)
- [CoSwin: Convolution Enhanced Hierarchical Shifted Window Attention For Small-Scale Vision](https://mlsanity.com/paper/2509.08959/) (2025)
- [Do not trust what you trust: Miscalibration in Semi-supervised Learning](https://mlsanity.com/paper/2403.15567/) (2024)
- [An Enhanced Harmonic Densely Connected Hybrid Transformer Network Architecture for Chronic Wound Segmentation Utilising Multi-Colour Space Tensor Merging](https://mlsanity.com/paper/2410.03359/) (2024)
- [NOVO: Unlearning-Compliant Vision Transformers](https://mlsanity.com/paper/2507.03281/) (2025)
- [Explicitly Increasing Input Information Density for Vision Transformers on Small Datasets](https://mlsanity.com/paper/2210.14319/) (2022)
- [Masked autoencoders are effective solution to transformer data-hungry](https://mlsanity.com/paper/2212.05677/) (2022)
- [Powerful Design of Small Vision Transformer on CIFAR10](https://mlsanity.com/paper/2501.06220/) (2025)
- [Unmasking Interstitial Lung Diseases: Leveraging Masked Autoencoders for Diagnosis](https://mlsanity.com/paper/2508.04429/) (2025)

## Abstract (arXiv)

Vision Transformer (ViT), a radically different architecture than convolutional neural networks offers multiple advantages including design simplicity, robustness and state-of-the-art performance on many vision tasks. However, in contrast to convolutional neural networks, Vision Transformer lacks inherent inductive biases. Therefore, successful training of such models is mainly attributed to pre-training on large-scale datasets such as ImageNet with 1.2M or JFT with 300M images. This hinders the direct adaption of Vision Transformer for small-scale datasets. In this work, we show that self-supervised inductive biases can be learned directly from small-scale datasets and serve as an effective weight initialization scheme for fine-tuning. This allows to train these models without large-scale pre-training, changes to model architecture or loss functions. We present thorough experiments to successfully train monolithic and non-monolithic Vision Transformers on five small datasets including CIFAR10/100, CINIC10, SVHN, Tiny-ImageNet and two fine-grained datasets: Aircraft and Cars. Our approach consistently improves the performance of Vision Transformers while retaining their properties such as attention to salient regions and higher robustness. Our codes and pre-trained models are available at: https://github.com/hananshafi/vits-for-small-scale-datasets.