mlsanityData to 14 Oct 2026

Can Vision Transformers Perform Convolution?

arXiv 2111.01353 ↗First posted 2 Nov 2021cs.CV cs.LGcited by 18

Not on the map. Why?

Abstract (arXiv)

Several recent studies have demonstrated that attention-based networks, such as Vision Transformer (ViT), can outperform Convolutional Neural Networks (CNNs) on several computer vision tasks without using convolutional layers. This naturally leads to the following questions: Can a self-attention layer of ViT express any convolution operation? In this work, we prove that a single ViT layer with image patches as the input can perform any convolution operation constructively, where the multi-head attention mechanism and the relative positional encoding play essential roles. We further provide a lower bound on the number of heads for Vision Transformers to express CNNs. Corresponding with our analysis, experimental results show that the construction in our proof can help inject convolutional bias into Transformers and significantly improve the performance of ViT in low data regimes.

Key facts. Cited by 18 papers in the map (data through 14 Oct 2026). Builds on 28 of its 33 references that are in the map. First posted on arXiv 2 Nov 2021.

Builds on

28 of 33 references.

References · 28

  1. 2017Attention Is All You Needcited by 60,361
  2. 2015Deep Residual Learning for Image Recognitioncited by 52,096
  3. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,399
  4. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 30,186
  5. 2019PyTorch: An Imperative Style, High-Performance Deep Learning Librarycited by 20,305
  6. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,599
  7. 2019RoBERTa: A Robustly Optimized BERT Pretraining Approachcited by 13,372
  8. 2021Swin Transformer: Hierarchical Vision Transformer using Shifted Windowscited by 12,139
  9. 2016Layer Normalizationcited by 5,669
  10. 2020Training data-efficient image transformers & distillation through attentioncited by 4,586
  11. 2019Transformer-XL: Attentive Language Models Beyond a Fixed-Length Contextcited by 1,850
  12. 2017Decoupled Weight Decay Regularizationcited by 1,831
16 more
  1. 2020Big Bird: Transformers for Longer Sequencescited by 1,684
  2. 2015Multi-Scale Context Aggregation by Dilated Convolutionscited by 1,133
  3. 2020On Layer Normalization in the Transformer Architecturecited by 922
  4. 2021ConViT: Improving Vision Transformers with Soft Convolutional Inductive Biasescited by 484
  5. 2021Attention is Not All You Need: Pure Attention Loses Rank Doubly Exponentially with Depthcited by 419
  6. 2019Are Transformers universal approximators of sequence-to-sequence functions?cited by 406
  7. 2019On the Relationship between Self-Attention and Convolutional Layerscited by 300
  8. 2021When Vision Transformers Outperform ResNets without Pre-training or Strong Data Augmentationscited by 260
  9. 2020Rethinking Positional Encoding in Language Pre-trainingcited by 197
  10. 2019On the Turing Completeness of Modern Neural Network Architecturescited by 129
  11. 2021Learnable Fourier Features for Multi-Dimensional Spatial Positional Encodingcited by 93
  12. 2020Low-Rank Bottleneck in Multi-head Attention Modelscited by 89
  13. 2020O(n) Connections are Expressive Enough: Universal Approximability of Sparse Transformerscited by 83
  14. 2021Stable, Fast and Accurate: Kernelized Attention with Relative Positional Encodingcited by 51
  15. 2021On the Expressive Power of Self-Attention Matricescited by 47
  16. 2021Which transformer architecture fits my data? A vocabulary bottleneck in self-attentioncited by 16

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2021: 2; 2022: 4; 2023: 6; 2024: 2; 2025: 3; 2026: 1.

18 papers.

Sort:

By area:

  1. 2021StyleSwin: Transformer-based GAN for High-resolution Image Generationcited by 152
  2. 2023DiffiT: Diffusion Vision Transformers for Image Generationcited by 106
  3. 2023Understanding the Role of the Projector in Knowledge Distillationcited by 31
  4. 2023Approximation Rate of the Transformer Architecture for Sequence Modelingcited by 28
  5. 2022Self-Promoted Supervision for Few-Shot Transformercited by 22
  6. 2021ELSA: Enhanced Local Self-Attention for Vision Transformercited by 18
  7. 2023Functional Interpolation for Relative Positions Improves Long Context Transformerscited by 18
  8. 2023A Brief Survey on the Approximation Theory for Sequence Modellingcited by 15
  9. 2025Sample Complexity and Representation Ability of Test-time Scaling Paradigmscited by 12
  10. 2023Learning a Fourier Transform for Linear Relative Positional Encodings in Transformerscited by 10
  11. 2022CodedVTR: Codebook-based Sparse Voxel Transformer with Geometric Guidancecited by 7
  12. 2022Towards Flexible Inductive Bias via Progressive Reparameterization Schedulingcited by 4
6 more
  1. 2022Comparison Analysis of Traditional Machine Learning and Deep Learning Techniques for Data and Image Classificationcited by 3
  2. 2024A Hyper-Transformer model for Controllable Pareto Front Learning with Split Feasibility Constraintscited by 3
  3. 2024Artificial Human Intelligence: The role of Humans in the Development of Next Generation AIcited by 1
  4. 2025Numerical Investigation of Sequence Modeling Theory using Controllable Memory Functionscited by 1
  5. 2025Dextr: Zero-Shot Neural Architecture Search with Singular Value Decomposition and Extrinsic Curvaturecited by 1
  6. 2026On the Expressive Power of Transformers for Maxout Networks and Continuous Piecewise Linear Functionscited by 1