mlsanityData to 14 Oct 2026

Convolutional Initialization for Data-Efficient Vision Transformers

arXiv 2401.12511 ↗First posted 23 Jan 2024cs.CVcited by 4

Not on the map. Why?

Abstract (arXiv)

Training vision transformer networks on small datasets poses challenges. In contrast, convolutional neural networks (CNNs) can achieve state-of-the-art performance by leveraging their architectural inductive bias. In this paper, we investigate whether this inductive bias can be reinterpreted as an initialization bias within a vision transformer network. Our approach is motivated by the finding that random impulse filters can achieve almost comparable performance to learned filters in CNNs. We introduce a novel initialization strategy for transformer networks that can achieve comparable performance to CNNs on small datasets while preserving its architectural flexibility.

Key facts. Cited by 4 papers in the map (data through 14 Oct 2026). Builds on 23 of its 28 references that are in the map. First posted on arXiv 23 Jan 2024.

Builds on

23 of 28 references.

References · 23

  1. 2017Attention Is All You Needcited by 60,361
  2. 2015Deep Residual Learning for Image Recognitioncited by 52,096
  3. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 30,186
  4. 2014ImageNet Large Scale Visual Recognition Challengecited by 8,498
  5. 2020Training data-efficient image transformers & distillation through attentioncited by 4,586
  6. 2021MLP-Mixer: An all-MLP Architecture for Visioncited by 1,446
  7. 2021CvT: Introducing Convolutions to Vision Transformerscited by 947
  8. 2017Revisiting Unreasonable Effectiveness of Data in Deep Learning Eracited by 919
  9. 2021CoAtNet: Marrying Convolution and Attention for All Data Sizescited by 569
  10. 2019On the Relationship between Self-Attention and Convolutional Layerscited by 300
  11. 2021Incorporating Convolution Designs into Visual Transformerscited by 248
  12. 2022UniFormer: Unifying Convolution and Self-attention for Visual Recognitioncited by 206
11 more
  1. 2022Patches Are All You Need?cited by 201
  2. 2021Vision Transformer for Small-Size Datasetscited by 122
  3. 2021Efficient Training of Visual Transformers with Small Datasetscited by 108
  4. 2022Unveiling Transformers with LEGO: a synthetic reasoning taskcited by 106
  5. 2022Better plain ViT baselines for ImageNet-1kcited by 98
  6. 2021Vision Transformer Pruningcited by 79
  7. 2021On the Integration of Self-Attention and Convolutioncited by 74
  8. 2023Mimetic Initialization of Self-Attention Layerscited by 54
  9. 2022How to Train Vision Transformer on Small-scale Datasets?cited by 35
  10. 2022Understanding the Covariance Structure of Convolutional Filterscited by 16
  11. 2019Convolution, attention and structure embeddingcited by 15

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2024: 2; 2025: 0; 2026: 2.

4 papers.

  1. 2024Efficient Learning With Sine-Activated Low-rank Matricescited by 22
  2. 2026Mimetic Initialization of MLPscited by 2
  3. 2024Habaek: High-performance water segmentation through dataset expansion and inductive bias optimizationcited by 1
  4. 2026Few-shot Class-Incremental Learning via Generative Co-Memory Regularizationcited by 1