mlsanityData to 7 Oct 2026

ViT-P: Rethinking Data-efficient Vision Transformers from Locality

arXiv 2203.02358 ↗First posted 4 Mar 2022cs.CVcited by 4

Not on the map. Why?

Abstract (arXiv)

Recent advances of Transformers have brought new trust to computer vision tasks. However, on small dataset, Transformers is hard to train and has lower performance than convolutional neural networks. We make vision transformers as data-efficient as convolutional neural networks by introducing multi-focal attention bias. Inspired by the attention distance in a well-trained ViT, we constrain the self-attention of ViT to have multi-scale localized receptive field. The size of receptive field is adaptable during training so that optimal configuration can be learned. We provide empirical evidence that proper constrain of receptive field can reduce the amount of training data for vision transformers. On Cifar100, our ViT-P Base model achieves the state-of-the-art accuracy (83.16%) trained from scratch. We also perform analysis on ImageNet to show our method does not lose accuracy on large data sets.

Key facts. Cited by 4 papers in the map (data through 7 Oct 2026). Builds on 21 of its 28 references that are in the map. First posted on arXiv 4 Mar 2022.

Builds on

21 of 28 references.

References · 21

  1. 2015Deep Residual Learning for Image Recognitioncited by 51,809
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  3. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 29,915
  4. 2021Swin Transformer: Hierarchical Vision Transformer using Shifted Windowscited by 12,082
  5. 2020End-to-End Object Detection with Transformerscited by 7,384
  6. 2021Masked Autoencoders Are Scalable Vision Learnerscited by 7,339
  7. 2019EfficientNet: Rethinking Model Scaling for Convolutional Neural Networkscited by 6,415
  8. 2020Training data-efficient image transformers & distillation through attentioncited by 4,560
  9. 2021BEiT: BERT Pre-Training of Image Transformerscited by 2,360
  10. 2021Pyramid Vision Transformer: A Versatile Backbone for Dense Prediction without Convolutionscited by 1,960
  11. 2021Tokens-to-Token ViT: Training Vision Transformers from Scratch on ImageNetcited by 1,096
  12. 2017Revisiting Unreasonable Effectiveness of Data in Deep Learning Eracited by 916
9 more
  1. 2021CoAtNet: Marrying Convolution and Attention for All Data Sizescited by 566
  2. 2021ConViT: Improving Vision Transformers with Soft Convolutional Inductive Biasescited by 484
  3. 2021Bottleneck Transformers for Visual Recognitioncited by 364
  4. 2019On the Relationship between Self-Attention and Convolutional Layerscited by 299
  5. 2021Escaping the Big Data Paradigm with Compact Transformerscited by 261
  6. 2021LocalViT: Analyzing Locality in Vision Transformerscited by 228
  7. 2021Scaling Local Self-Attention for Parameter Efficient Visual Backbonescited by 220
  8. 2021Visformer: The Vision-friendly Transformercited by 137
  9. 2021Nested Hierarchical Transformer: Towards Accurate, Data-Efficient and Interpretable Visual Understandingcited by 97

Built on by

Cited by year

Citing papers per year: 2022: 1; 2023: 2; 2024: 1.

4 papers.

  1. 2022Hydra Attention: Efficient Attention with Many Headscited by 55
  2. 2024Facing the Elephant in the Room: Visual Prompt Tuning or Full Finetuning?cited by 31
  3. 2023Inference Time Evidences of Adversarial Attacks for Forensic on Transformerscited by 0
  4. 2023Dimension Mixer: Group Mixing of Input Dimensions for Efficient Function Approximationcited by 0