mlsanityData to 7 Oct 2026

Training Vision Transformers with Only 2040 Images

arXiv 2201.10728 ↗First posted 26 Jan 2022cs.CV cs.AIcited by 22

Not on the map. Why?

Abstract (arXiv)

Vision Transformers (ViTs) is emerging as an alternative to convolutional neural networks (CNNs) for visual recognition. They achieve competitive results with CNNs but the lack of the typical convolutional inductive bias makes them more data-hungry than common CNNs. They are often pretrained on JFT-300M or at least ImageNet and few works study training ViTs with limited data. In this paper, we investigate how to train ViTs with limited data (e.g., 2040 images). We give theoretical analyses that our method (based on parametric instance discrimination) is superior to other methods in that it can capture both feature alignment and instance similarities. We achieve state-of-the-art results when training from scratch on 7 small datasets under various ViT backbones. We also investigate the transferring ability of small datasets and find that representations learned from small datasets can even improve large-scale ImageNet training.

Key facts. Cited by 22 papers in the map (data through 7 Oct 2026). Builds on 35 of its 47 references that are in the map. First posted on arXiv 26 Jan 2022.

Builds on

35 of 47 references.

References · 35

  1. 2017Attention Is All You Needcited by 59,932
  2. 2015Deep Residual Learning for Image Recognitioncited by 51,809
  3. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  4. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 29,915
  5. 2021Swin Transformer: Hierarchical Vision Transformer using Shifted Windowscited by 12,082
  6. 2020A Simple Framework for Contrastive Learning of Visual Representationscited by 11,979
  7. 2014ImageNet Large Scale Visual Recognition Challengecited by 8,441
  8. 2018Representation Learning with Contrastive Predictive Codingcited by 7,662
  9. 2019Momentum Contrast for Unsupervised Visual Representation Learningcited by 7,482
  10. 2020End-to-End Object Detection with Transformerscited by 7,384
  11. 2021Emerging Properties in Self-Supervised Vision Transformerscited by 6,893
  12. 2015Rethinking the Inception Architecture for Computer Visioncited by 5,451
23 more
  1. 2017mixup: Beyond Empirical Risk Minimizationcited by 5,155
  2. 2020Training data-efficient image transformers & distillation through attentioncited by 4,560
  3. 2019CutMix: Regularization Strategy to Train Strong Classifiers with Localizable Featurescited by 2,875
  4. 2020Unsupervised Learning of Visual Features by Contrasting Cluster Assignmentscited by 2,845
  5. 2020Supervised Contrastive Learningcited by 2,803
  6. 2020Improved Baselines with Momentum Contrastive Learningcited by 2,282
  7. 2013Describing Textures in the Wildcited by 2,110
  8. 2021Pyramid Vision Transformer: A Versatile Backbone for Dense Prediction without Convolutionscited by 1,960
  9. 2018Unsupervised Feature Learning via Non-Parametric Instance-level Discriminationcited by 1,838
  10. 2013Fine-Grained Visual Classification of Aircraftcited by 1,701
  11. 2021An Empirical Study of Training Self-Supervised Vision Transformerscited by 1,610
  12. 2020Understanding Contrastive Representation Learning through Alignment and Uniformity on the Hyperspherecited by 1,393
  13. 2021Tokens-to-Token ViT: Training Vision Transformers from Scratch on ImageNetcited by 1,096
  14. 2018Deep Clustering for Unsupervised Learning of Visual Featurescited by 1,020
  15. 2021CvT: Introducing Convolutions to Vision Transformerscited by 946
  16. 2019Once-for-All: Train One Network and Specialize it for Efficient Deploymentcited by 778
  17. 2021PVT v2: Improved Baselines with Pyramid Vision Transformercited by 749
  18. 2020End-to-End Video Instance Segmentation with Transformerscited by 443
  19. 2014Discriminative Unsupervised Feature Learning with Exemplar Convolutional Neural Networkscited by 299
  20. 2020Rethinking the Value of Labels for Improving Class-Imbalanced Learningcited by 216
  21. 2020Parametric Instance Classification for Unsupervised Visual Feature Learningcited by 41
  22. 2021Rethinking Self-Supervised Learning: Small is Beautifulcited by 13
  23. 2021Train a One-Million-Way Instance Classifier for Unsupervised Visual Representation Learningcited by 2

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2022: 7; 2023: 8; 2024: 5; 2025: 1; 2026: 1.

22 papers.

Sort:

By area:

  1. 2023Zero-TPrune: Zero-Shot Token Pruning through Leveraging of the Attention Graph in Pre-Trained Transformerscited by 61
  2. 2023Mimetic Initialization of Self-Attention Layerscited by 52
  3. 2022Towards Data-Efficient Detection Transformerscited by 34
  4. 2022Self-Promoted Supervision for Few-Shot Transformercited by 22
  5. 2022Spikeformer: A Novel Architecture for Training High-Performance Low-Latency Spiking Neural Networkcited by 19
  6. 2022Augraphy: A Data Augmentation Library for Document Imagescited by 14
  7. 2024Scaling Backwards: Minimal Synthetic Pre-training?cited by 14
  8. 2022GMML is All you Needcited by 12
  9. 2023Pre-training Vision Transformers with Very Limited Synthesized Imagescited by 12
  10. 2022A Survey of Visual Sensory Anomaly Detectioncited by 11
  11. 2023MDViT: Multi-domain Vision Transformer for Small Medical Image Segmentation Datasetscited by 9
  12. 2024RAW-Diffusion: RGB-Guided Diffusion Models for High-Fidelity RAW Image Generationcited by 8
10 more
  1. 2023Are Vision Transformers More Data Hungry Than Newborn Visual Systems?cited by 7
  2. 2023GSB: Group Superposition Binarization for Vision Transformer with Limited Training Samplescited by 4
  3. 2023YOLOPose V2: Understanding and Improving Transformer-based 6D Pose Estimationcited by 3
  4. 2024CloudCast -- Total Cloud Cover Nowcasting with Machine Learningcited by 3
  5. 2022Action Quality Assessment using Transformerscited by 2
  6. 2023Automatic diagnosis of knee osteoarthritis severity using Swin transformercited by 2
  7. 2024On Improving the Algorithm-, Model-, and Data- Efficiency of Self-Supervised Learningcited by 1
  8. 2024uaMix-MAE: Efficient Tuning of Pretrained Audio Transformers with Unsupervised Audio Mixturescited by 0
  9. 2025AMD-HookNet++: Evolution of AMD-HookNet with Hybrid CNN-Transformer Feature Enhancement for Glacier Calving Front Segmentationcited by 0
  10. 2026Training-Efficient Text-to-Music Generation with State-Space Modelingcited by 0