mlsanityData to 7 Oct 2026

On the Relationship between Self-Attention and Convolutional Layers

arXiv 1911.03584 ↗First posted 8 Nov 2019cs.LG cs.CL cs.CVcited by 299

Not on the map. Why?

Abstract (arXiv)

Recent trends of incorporating attention mechanisms in vision have led researchers to reconsider the supremacy of convolutional layers as a primary building block. Beyond helping CNNs to handle long-range dependencies, Ramachandran et al. (2019) showed that attention can completely replace convolution and achieve state-of-the-art performance on vision tasks. This raises the question: do learned attention layers operate similarly to convolutional layers? This work provides evidence that attention layers can perform convolution and, indeed, they often learn to do so in practice. Specifically, we prove that a multi-head self-attention layer with sufficient number of heads is at least as expressive as any convolutional layer. Our numerical experiments then show that self-attention layers attend to pixel-grid patterns similarly to CNN layers, corroborating our analysis. Our code is publicly available.

Key facts. Cited by 299 papers in the map (data through 7 Oct 2026). First posted on arXiv 8 Nov 2019.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 81; 2022: 75; 2023: 55; 2024: 37; 2025: 32; 2026: 19.

299 papers.

Sort:

By area:

  1. 2021SwinIR: Image Restoration Using Swin Transformercited by 1,869
  2. 2021Is Space-Time Attention All You Need for Video Understanding?cited by 1,609
  3. 2021MLP-Mixer: An all-MLP Architecture for Visioncited by 1,441
  4. 2021Frozen in Time: A Joint Video and Image Encoder for End-to-End Retrievalcited by 1,175
  5. 2021Perceiver: General Perception with Iterative Attentioncited by 1,089
  6. 2021Transformers in Vision: A Surveycited by 1,009
  7. 2021Scaling Vision Transformerscited by 993
  8. 2023Scaling Vision Transformers to 22 Billion Parameterscited by 690
  9. 2021Do Vision Transformers See Like Convolutional Neural Networks?cited by 671
  10. 2021ConViT: Improving Vision Transformers with Soft Convolutional Inductive Biasescited by 484
  11. 2021LeViT: a Vision Transformer in ConvNet's Clothing for Faster Inferencecited by 475
  12. 2022DeiT III: Revenge of the ViTcited by 417
38 more
  1. 2021Attention is Not All You Need: Pure Attention Loses Rank Doubly Exponentially with Depthcited by 411
  2. 2021Conditional Positional Encodings for Vision Transformerscited by 402
  3. 2021Early Convolutions Help Transformers See Bettercited by 395
  4. 2021Deep ViT Features as Dense Visual Descriptorscited by 327
  5. 2022How Do Vision Transformers Work?cited by 305
  6. 2021Understanding Robustness of Transformers for Image Classificationcited by 250
  7. 2021Localizing Objects with Self-Supervised Transformers and no Labelscited by 227
  8. 2021Scaling Local Self-Attention for Parameter Efficient Visual Backbonescited by 220
  9. 2022UniFormer: Unifying Convolution and Self-attention for Visual Recognitioncited by 206
  10. 2022Patches Are All You Need?cited by 201
  11. 2021UniT: Multimodal Multitask Learning with a Unified Transformercited by 195
  12. 2022Representing Long-Range Context for Graph Neural Networks with Global Attentioncited by 193
  13. 2022Towards Multi-spatiotemporal-scale Generalized PDE Modelingcited by 190
  14. 2022Pure Transformers are Powerful Graph Learnerscited by 188
  15. 2022UniFormer: Unified Transformer for Efficient Spatiotemporal Representation Learningcited by 182
  16. 2021Chasing Sparsity in Vision Transformers: An End-to-End Explorationcited by 181
  17. 2021On the Robustness of Vision Transformers to Adversarial Examplescited by 167
  18. 2021ASFormer: Transformer for Action Segmentationcited by 166
  19. 2022Scaling Up Your Kernels to 31x31: Revisiting Large Kernel Design in CNNscited by 155
  20. 2021StyleSwin: Transformer-based GAN for High-resolution Image Generationcited by 152
  21. 2021TokenLearner: What Can 8 Learned Tokens Do for Images and Videos?cited by 127
  22. 2021CycleMLP: A MLP-like Architecture for Dense Predictioncited by 126
  23. 2021Generative Adversarial Transformerscited by 120
  24. 2021A Survey of Visual Transformerscited by 117
  25. 2022What to Hide from Your Students: Attention-Guided Masked Image Modelingcited by 117
  26. 2021Relaxed Transformer Decoders for Direct Action Proposal Generationcited by 116
  27. 2021Global Self-Attention as a Replacement for Graph Convolutioncited by 113
  28. 2021Are Convolutional Neural Networks or Transformers more like human vision?cited by 109
  29. 2021LambdaNetworks: Modeling Long-Range Interactions Without Attentioncited by 107
  30. 2022Unveiling Transformers with LEGO: a synthetic reasoning taskcited by 105
  31. 2023OctFormer: Octree-based Transformers for 3D Point Cloudscited by 105
  32. 2021Involution: Inverting the Inherence of Convolution for Visual Recognitioncited by 99
  33. 2021Nested Hierarchical Transformer: Towards Accurate, Data-Efficient and Interpretable Visual Understandingcited by 97
  34. 2022Vision Transformers provably learn spatial structurecited by 92
  35. 2021An Image is Worth 16x16 Words, What is a Video Worth?cited by 87
  36. 2021Scalable Vision Transformers with Hierarchical Poolingcited by 84
  37. 2022Fast Vision Transformers with HiLo Attentioncited by 78
  38. 2023A Theoretical Understanding of Shallow Vision Transformers: Learning, Generalization, and Sample Complexitycited by 77

50 most cited of 299 All 299 (JSON)