# Vision Transformer for Small-Size Datasets

arXiv: https://arxiv.org/abs/2112.13492 · HTML: https://mlsanity.com/paper/2112.13492/

## Key facts

- Cited by 122 papers in the map (data through 14 Oct 2026).
- Builds on 37 of its 47 references that are in the map.
- First posted on arXiv 27 Dec 2021.

## Builds on (most cited first, 30 of 37)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- [An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale](https://mlsanity.com/paper/2010.11929/) (2020)
- [PyTorch: An Imperative Style, High-Performance Deep Learning Library](https://mlsanity.com/paper/1912.01703/) (2019)
- [Swin Transformer: Hierarchical Vision Transformer using Shifted Windows](https://mlsanity.com/paper/2103.14030/) (2021)
- [Densely Connected Convolutional Networks](https://mlsanity.com/paper/1608.06993/) (2016)
- [EfficientNet: Rethinking Model Scaling for Convolutional Neural Networks](https://mlsanity.com/paper/1905.11946/) (2019)
- [Layer Normalization](https://mlsanity.com/paper/1607.06450/) (2016)
- [Rethinking the Inception Architecture for Computer Vision](https://mlsanity.com/paper/1512.00567/) (2015)
- [mixup: Beyond Empirical Risk Minimization](https://mlsanity.com/paper/1710.09412/) (2017)
- [Squeeze-and-Excitation Networks](https://mlsanity.com/paper/1709.01507/) (2017)
- [Training data-efficient image transformers & distillation through attention](https://mlsanity.com/paper/2012.12877/) (2020)
- [Going Deeper with Convolutions](https://mlsanity.com/paper/1409.4842/) (2014)
- [CutMix: Regularization Strategy to Train Strong Classifiers with Localizable Features](https://mlsanity.com/paper/1905.04899/) (2019)
- [ALBERT: A Lite BERT for Self-supervised Learning of Language Representations](https://mlsanity.com/paper/1909.11942/) (2019)
- [Non-local Neural Networks](https://mlsanity.com/paper/1711.07971/) (2017)
- [RandAugment: Practical automated data augmentation with a reduced search space](https://mlsanity.com/paper/1909.13719/) (2019)
- [Deep High-Resolution Representation Learning for Visual Recognition](https://mlsanity.com/paper/1908.07919/) (2019)
- [Relational inductive biases, deep learning, and graph networks](https://mlsanity.com/paper/1806.01261/) (2018)
- [Random Erasing Data Augmentation](https://mlsanity.com/paper/1708.04896/) (2017)
- [Tokens-to-Token ViT: Training Vision Transformers from Scratch on ImageNet](https://mlsanity.com/paper/2101.11986/) (2021)
- [Deep Networks with Stochastic Depth](https://mlsanity.com/paper/1603.09382/) (2016)
- [CvT: Introducing Convolutions to Vision Transformers](https://mlsanity.com/paper/2103.15808/) (2021)
- [Revisiting Unreasonable Effectiveness of Data in Deep Learning Era](https://mlsanity.com/paper/1707.02968/) (2017)
- [TSM: Temporal Shift Module for Efficient Video Understanding](https://mlsanity.com/paper/1811.08383/) (2018)
- [Self-Attention Generative Adversarial Networks](https://mlsanity.com/paper/1805.08318/) (2018)
- [Going deeper with Image Transformers](https://mlsanity.com/paper/2103.17239/) (2021)

## Built on by (most cited first, 30 of 122)

- [EfficientFormer: Vision Transformers at MobileNet Speed](https://mlsanity.com/paper/2206.01191/) (2022)
- [Rethinking Vision Transformers for MobileNet Size and Speed](https://mlsanity.com/paper/2212.08059/) (2022)
- [The No Free Lunch Theorem, Kolmogorov Complexity, and the Role of Inductive Biases in Machine Learning](https://mlsanity.com/paper/2304.05366/) (2023)
- [Your ViT is Secretly a Hybrid Discriminative-Generative Diffusion Model](https://mlsanity.com/paper/2208.07791/) (2022)
- [Mimetic Initialization of Self-Attention Layers](https://mlsanity.com/paper/2305.09828/) (2023)
- [Few-Shot Diffusion Models](https://mlsanity.com/paper/2205.15463/) (2022)
- [ObjectStitch: Generative Object Compositing](https://mlsanity.com/paper/2212.00932/) (2022)
- [How to Train Vision Transformer on Small-scale Datasets?](https://mlsanity.com/paper/2210.07240/) (2022)
- [Implicit Bias and Fast Convergence Rates for Self-attention](https://mlsanity.com/paper/2402.05738/) (2024)
- [Bridging the Gap Between Vision Transformers and Convolutional Neural Networks on Small Datasets](https://mlsanity.com/paper/2210.05958/) (2022)
- [Understanding Why ViT Trains Badly on Small Datasets: An Intuitive Perspective](https://mlsanity.com/paper/2302.03751/) (2023)
- [Neural Integral Equations](https://mlsanity.com/paper/2209.15190/) (2022)
- [Supervised Masked Knowledge Distillation for Few-Shot Transformers](https://mlsanity.com/paper/2303.15466/) (2023)
- [AA-TransUNet: Attention Augmented TransUNet For Nowcasting Tasks](https://mlsanity.com/paper/2202.04996/) (2022)
- [Spikeformer: A Novel Architecture for Training High-Performance Low-Latency Spiking Neural Network](https://mlsanity.com/paper/2211.10686/) (2022)
- [2-D SSM: A General Spatial Layer for Visual Transformers](https://mlsanity.com/paper/2306.06635/) (2023)
- [4-bit Shampoo for Memory-Efficient Network Training](https://mlsanity.com/paper/2405.18144/) (2024)
- [Limited Data, Unlimited Potential: A Study on ViTs Augmented by Masked Autoencoders](https://mlsanity.com/paper/2310.20704/) (2023)
- [Attention Distillation: self-supervised vision transformer students need more guidance](https://mlsanity.com/paper/2210.00944/) (2022)
- [Accumulated Trivial Attention Matters in Vision Transformers on Small Datasets](https://mlsanity.com/paper/2210.12333/) (2022)
- [TFormer: A throughout fusion transformer for multi-modal skin lesion diagnosis](https://mlsanity.com/paper/2211.11393/) (2022)
- [Augraphy: A Data Augmentation Library for Document Images](https://mlsanity.com/paper/2208.14558/) (2022)
- [FedSOL: Stabilized Orthogonal Learning with Proximal Restrictions in Federated Learning](https://mlsanity.com/paper/2308.12532/) (2023)
- [Transformers Learn Low Sensitivity Functions: Investigations and Implications](https://mlsanity.com/paper/2403.06925/) (2024)
- [Vision Transformers in 2022: An Update on Tiny ImageNet](https://mlsanity.com/paper/2205.10660/) (2022)
- [Temporal Latent Bottleneck: Synthesis of Fast and Slow Processing Mechanisms in Sequence Learning](https://mlsanity.com/paper/2205.14794/) (2022)
- [A Reliable and Interpretable Framework of Multi-view Learning for Liver Fibrosis Staging](https://mlsanity.com/paper/2306.12054/) (2023)
- [Masked Modeling for Self-supervised Representation Learning on Vision and Beyond](https://mlsanity.com/paper/2401.00897/) (2023)
- [Frozen Feature Augmentation for Few-Shot Image Classification](https://mlsanity.com/paper/2403.10519/) (2024)
- [Experts Weights Averaging: A New General Training Scheme for Vision Transformers](https://mlsanity.com/paper/2308.06093/) (2023)

## Abstract (arXiv)

Recently, the Vision Transformer (ViT), which applied the transformer structure to the image classification task, has outperformed convolutional neural networks. However, the high performance of the ViT results from pre-training using a large-size dataset such as JFT-300M, and its dependence on a large dataset is interpreted as due to low locality inductive bias. This paper proposes Shifted Patch Tokenization (SPT) and Locality Self-Attention (LSA), which effectively solve the lack of locality inductive bias and enable it to learn from scratch even on small-size datasets. Moreover, SPT and LSA are generic and effective add-on modules that are easily applicable to various ViTs. Experimental results show that when both SPT and LSA were applied to the ViTs, the performance improved by an average of 2.96% in Tiny-ImageNet, which is a representative small-size dataset. Especially, Swin Transformer achieved an overwhelming performance improvement of 4.08% thanks to the proposed SPT and LSA.