mlsanityData to 14 Oct 2026

Stand-Alone Self-Attention in Vision Models

arXiv 1906.05909 ↗First posted 13 Jun 2019cs.CVcited by 542

Not on the map. Why?

Abstract (arXiv)

Convolutions are a fundamental building block of modern computer vision systems. Recent approaches have argued for going beyond convolutions in order to capture long-range dependencies. These efforts focus on augmenting convolutional models with content-based interactions, such as self-attention and non-local means, to achieve gains on a number of vision tasks. The natural question that arises is whether attention can be a stand-alone primitive for vision models instead of serving as just an augmentation on top of convolutions. In developing and testing a pure self-attention vision model, we verify that self-attention can indeed be an effective stand-alone layer. A simple procedure of replacing all instances of spatial convolutions with a form of self-attention applied to ResNet model produces a fully self-attentional model that outperforms the baseline on ImageNet classification with 12% fewer FLOPS and 29% fewer parameters. On COCO object detection, a pure self-attention model matches the mAP of a baseline RetinaNet while having 39% fewer FLOPS and 34% fewer parameters. Detailed ablation studies demonstrate that self-attention is especially impactful when used in later layers. These results establish that stand-alone self-attention is an important addition to the vision practitioner's toolbox.

Key facts. Cited by 542 papers in the map (data through 14 Oct 2026). First posted on arXiv 13 Jun 2019.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2019: 1; 2020: 0; 2021: 221; 2022: 143; 2023: 78; 2024: 57; 2025: 36; 2026: 6.

542 papers.

Sort:

By area:

  1. 2021Swin Transformer: Hierarchical Vision Transformer using Shifted Windowscited by 12,139
  2. 2022A ConvNet for the 2020scited by 3,579
  3. 2021Pyramid Vision Transformer: A Versatile Backbone for Dense Prediction without Convolutionscited by 1,962
  4. 2021SwinIR: Image Restoration Using Swin Transformercited by 1,887
  5. 2021Vision Transformers for Dense Predictioncited by 1,803
  6. 2021Is Space-Time Attention All You Need for Video Understanding?cited by 1,620
  7. 2021ViViT: A Video Vision Transformercited by 1,620
  8. 2021An Empirical Study of Training Self-Supervised Vision Transformerscited by 1,616
  9. 2021MLP-Mixer: An all-MLP Architecture for Visioncited by 1,446
  10. 2021Frozen in Time: A Joint Video and Image Encoder for End-to-End Retrievalcited by 1,177
  11. 2021Perceiver: General Perception with Iterative Attentioncited by 1,104
  12. 2021Tokens-to-Token ViT: Training Vision Transformers from Scratch on ImageNetcited by 1,101
38 more
  1. 2021Transformers in Vision: A Surveycited by 1,013
  2. 2021UNETR: Transformers for 3D Medical Image Segmentationcited by 959
  3. 2021Multiscale Vision Transformerscited by 897
  4. 2021CrossViT: Cross-Attention Multi-Scale Vision Transformer for Image Classificationcited by 806
  5. 2021Going deeper with Image Transformerscited by 764
  6. 2021Do Vision Transformers See Like Convolutional Neural Networks?cited by 675
  7. 2021MobileViT: Light-weight, General-purpose, and Mobile-friendly Vision Transformercited by 642
  8. 2021CoAtNet: Marrying Convolution and Attention for All Data Sizescited by 569
  9. 2021Twins: Revisiting the Design of Spatial Attention in Vision Transformerscited by 565
  10. 2021CSWin Transformer: A General Vision Transformer Backbone with Cross-Shaped Windowscited by 527
  11. 2021MetaFormer Is Actually What You Need for Visioncited by 524
  12. 2021ConViT: Improving Vision Transformers with Soft Convolutional Inductive Biasescited by 484
  13. 2021Intriguing Properties of Vision Transformerscited by 446
  14. 2021VATT: Transformers for Multimodal Self-Supervised Learning from Raw Video, Audio and Textcited by 429
  15. 2021Attention is Not All You Need: Pure Attention Loses Rank Doubly Exponentially with Depthcited by 419
  16. 2021Conditional Positional Encodings for Vision Transformerscited by 406
  17. 2021Early Convolutions Help Transformers See Bettercited by 396
  18. 2021Bottleneck Transformers for Visual Recognitioncited by 364
  19. 2021Rethinking Spatial Dimensions of Vision Transformerscited by 361
  20. 2021CMT: Convolutional Neural Networks Meet Vision Transformerscited by 310
  21. 2021DeepViT: Towards Deeper Vision Transformercited by 289
  22. 2021Focal Self-attention for Local-Global Interactions in Vision Transformerscited by 288
  23. 2022Wayformer: Motion Forecasting via Simple & Efficient Attention Networkscited by 280
  24. 2021Escaping the Big Data Paradigm with Compact Transformerscited by 264
  25. 2021N"UWA: Visual Synthesis Pre-training for Neural visUal World creAtioncited by 253
  26. 2021Understanding Robustness of Transformers for Image Classificationcited by 249
  27. 2021Incorporating Convolution Designs into Visual Transformerscited by 248
  28. 2022Transformers in Medical Imaging: A Surveycited by 245
  29. 2022Neighborhood Attention Transformercited by 239
  30. 2021Attention Mechanisms in Computer Vision: A Surveycited by 232
  31. 2021Localizing Objects with Self-Supervised Transformers and no Labelscited by 230
  32. 2021Learning to Estimate Hidden Motions with Global Motion Aggregationcited by 224
  33. 2021Co-Scale Conv-Attentional Image Transformerscited by 224
  34. 2021Scaling Local Self-Attention for Parameter Efficient Visual Backbonescited by 220
  35. 2021Group-Free 3D Object Detection via Transformerscited by 215
  36. 2022Visual Attention Networkcited by 210
  37. 2021Rethinking and Improving Relative Position Encoding for Vision Transformercited by 207
  38. 2022UniFormer: Unifying Convolution and Self-attention for Visual Recognitioncited by 206

50 most cited of 542 All 542 (JSON)