mlsanityData to 14 Oct 2026

Vision Transformer for Small-Size Datasets

arXiv 2112.13492 ↗First posted 27 Dec 2021cs.CVcited by 122

Not on the map. Why?

Abstract (arXiv)

Recently, the Vision Transformer (ViT), which applied the transformer structure to the image classification task, has outperformed convolutional neural networks. However, the high performance of the ViT results from pre-training using a large-size dataset such as JFT-300M, and its dependence on a large dataset is interpreted as due to low locality inductive bias. This paper proposes Shifted Patch Tokenization (SPT) and Locality Self-Attention (LSA), which effectively solve the lack of locality inductive bias and enable it to learn from scratch even on small-size datasets. Moreover, SPT and LSA are generic and effective add-on modules that are easily applicable to various ViTs. Experimental results show that when both SPT and LSA were applied to the ViTs, the performance improved by an average of 2.96% in Tiny-ImageNet, which is a representative small-size dataset. Especially, Swin Transformer achieved an overwhelming performance improvement of 4.08% thanks to the proposed SPT and LSA.

Key facts. Cited by 122 papers in the map (data through 14 Oct 2026). Builds on 37 of its 47 references that are in the map. First posted on arXiv 27 Dec 2021.

Builds on

37 of 47 references.

References · 37

  1. 2017Attention Is All You Needcited by 60,361
  2. 2015Deep Residual Learning for Image Recognitioncited by 52,096
  3. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,399
  4. 2014Adam: A Method for Stochastic Optimizationcited by 37,240
  5. 2020Language Models are Few-Shot Learnerscited by 32,419
  6. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 30,186
  7. 2019PyTorch: An Imperative Style, High-Performance Deep Learning Librarycited by 20,305
  8. 2021Swin Transformer: Hierarchical Vision Transformer using Shifted Windowscited by 12,139
  9. 2016Densely Connected Convolutional Networkscited by 6,720
  10. 2019EfficientNet: Rethinking Model Scaling for Convolutional Neural Networkscited by 6,442
  11. 2016Layer Normalizationcited by 5,669
  12. 2015Rethinking the Inception Architecture for Computer Visioncited by 5,475
25 more
  1. 2017mixup: Beyond Empirical Risk Minimizationcited by 5,183
  2. 2017Squeeze-and-Excitation Networkscited by 4,789
  3. 2020Training data-efficient image transformers & distillation through attentioncited by 4,586
  4. 2014Going Deeper with Convolutionscited by 4,241
  5. 2019CutMix: Regularization Strategy to Train Strong Classifiers with Localizable Featurescited by 2,884
  6. 2019ALBERT: A Lite BERT for Self-supervised Learning of Language Representationscited by 2,790
  7. 2017Non-local Neural Networkscited by 2,293
  8. 2019RandAugment: Practical automated data augmentation with a reduced search spacecited by 2,269
  9. 2019Deep High-Resolution Representation Learning for Visual Recognitioncited by 1,420
  10. 2018Relational inductive biases, deep learning, and graph networkscited by 1,360
  11. 2017Random Erasing Data Augmentationcited by 1,239
  12. 2021Tokens-to-Token ViT: Training Vision Transformers from Scratch on ImageNetcited by 1,101
  13. 2016Deep Networks with Stochastic Depthcited by 979
  14. 2021CvT: Introducing Convolutions to Vision Transformerscited by 947
  15. 2017Revisiting Unreasonable Effectiveness of Data in Deep Learning Eracited by 919
  16. 2018TSM: Temporal Shift Module for Efficient Video Understandingcited by 856
  17. 2018Self-Attention Generative Adversarial Networkscited by 809
  18. 2021Going deeper with Image Transformerscited by 764
  19. 2017Residual Attention Network for Image Classificationcited by 386
  20. 2021Bottleneck Transformers for Visual Recognitioncited by 364
  21. 2019Attention Augmented Convolutional Networkscited by 363
  22. 2021Rethinking Spatial Dimensions of Vision Transformerscited by 361
  23. 2021LocalViT: Analyzing Locality in Vision Transformerscited by 228
  24. 2021Co-Scale Conv-Attentional Image Transformerscited by 224
  25. 2020Towards Learning Convolutions from Scratchcited by 46

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2022: 28; 2023: 31; 2024: 41; 2025: 18; 2026: 4.

122 papers.

Sort:

By area:

  1. 2022EfficientFormer: Vision Transformers at MobileNet Speedcited by 241
  2. 2022Rethinking Vision Transformers for MobileNet Size and Speedcited by 121
  3. 2023The No Free Lunch Theorem, Kolmogorov Complexity, and the Role of Inductive Biases in Machine Learningcited by 63
  4. 2022Your ViT is Secretly a Hybrid Discriminative-Generative Diffusion Modelcited by 58
  5. 2023Mimetic Initialization of Self-Attention Layerscited by 54
  6. 2022Few-Shot Diffusion Modelscited by 38
  7. 2022ObjectStitch: Generative Object Compositingcited by 36
  8. 2022How to Train Vision Transformer on Small-scale Datasets?cited by 35
  9. 2024Implicit Bias and Fast Convergence Rates for Self-attentioncited by 31
  10. 2022Bridging the Gap Between Vision Transformers and Convolutional Neural Networks on Small Datasetscited by 29
  11. 2023Understanding Why ViT Trains Badly on Small Datasets: An Intuitive Perspectivecited by 28
  12. 2022Neural Integral Equationscited by 23
38 more
  1. 2023Supervised Masked Knowledge Distillation for Few-Shot Transformerscited by 23
  2. 2022AA-TransUNet: Attention Augmented TransUNet For Nowcasting Taskscited by 22
  3. 2022Spikeformer: A Novel Architecture for Training High-Performance Low-Latency Spiking Neural Networkcited by 19
  4. 20232-D SSM: A General Spatial Layer for Visual Transformerscited by 18
  5. 20244-bit Shampoo for Memory-Efficient Network Trainingcited by 18
  6. 2023Limited Data, Unlimited Potential: A Study on ViTs Augmented by Masked Autoencoderscited by 17
  7. 2022Attention Distillation: self-supervised vision transformer students need more guidancecited by 16
  8. 2022Accumulated Trivial Attention Matters in Vision Transformers on Small Datasetscited by 15
  9. 2022TFormer: A throughout fusion transformer for multi-modal skin lesion diagnosiscited by 15
  10. 2022Augraphy: A Data Augmentation Library for Document Imagescited by 14
  11. 2023FedSOL: Stabilized Orthogonal Learning with Proximal Restrictions in Federated Learningcited by 14
  12. 2024Transformers Learn Low Sensitivity Functions: Investigations and Implicationscited by 14
  13. 2022Vision Transformers in 2022: An Update on Tiny ImageNetcited by 13
  14. 2022Temporal Latent Bottleneck: Synthesis of Fast and Slow Processing Mechanisms in Sequence Learningcited by 13
  15. 2023A Reliable and Interpretable Framework of Multi-view Learning for Liver Fibrosis Stagingcited by 13
  16. 2023Masked Modeling for Self-supervised Representation Learning on Vision and Beyondcited by 13
  17. 2024Frozen Feature Augmentation for Few-Shot Image Classificationcited by 13
  18. 2023Experts Weights Averaging: A New General Training Scheme for Vision Transformerscited by 12
  19. 2023TULIP: Transformer for Upsampling of LiDAR Point Cloudscited by 12
  20. 2022Continual Hippocampus Segmentation with Transformerscited by 11
  21. 2023Fast offset corrected in-memory trainingcited by 11
  22. 2023Diffusion Models as Artists: Are we Closing the Gap between Humans and Machines?cited by 10
  23. 2024TBConvL-Net: A Hybrid Deep Learning Architecture for Robust Medical Image Segmentationcited by 8
  24. 2024Onboard Satellite Image Classification for Earth Observation: A Comparative Study of ViT Modelscited by 8
  25. 2022Improving Deep Neural Network Random Initialization Through Neuronal Rewiringcited by 7
  26. 2022Class-attention Video Transformer for Engagement Intensity Predictioncited by 7
  27. 2022Neural Attentive Circuitscited by 6
  28. 2022Training a Vision Transformer from scratch in less than 24 hours with 1 GPUcited by 6
  29. 2023V1T: large-scale mouse V1 response prediction using a Vision Transformercited by 6
  30. 2023Astroformer: More Data Might not be all you need for Classificationcited by 6
  31. 2023Deep Classifier Mimicry without Data Accesscited by 6
  32. 2024Weight Copy and Low-Rank Adaptation for Few-Shot Distillation of Vision Transformerscited by 6
  33. 2024GenFormer -- Generated Images are All You Need to Improve Robustness of Transformers on Small Datasetscited by 6
  34. 2023Multi-Dimensional Hyena for Spatial Inductive Biascited by 5
  35. 2024Vision Transformers with Natural Language Semanticscited by 5
  36. 2022Local Learning on Transformers via Feature Reconstructioncited by 4
  37. 2023Denoising Diffusion Probabilistic Model for Retinal Image Generation and Segmentationcited by 4
  38. 2024Convolutional Initialization for Data-Efficient Vision Transformerscited by 4

50 most cited of 122 All 122 (JSON)