mlsanityData to 7 Oct 2026

Exploring Self-attention for Image Recognition

arXiv 2004.13621 ↗First posted 28 Apr 2020cs.CVcited by 313

Not on the map. Why?

Abstract (arXiv)

Recent work has shown that self-attention can serve as a basic building block for image recognition models. We explore variations of self-attention and assess their effectiveness for image recognition. We consider two forms of self-attention. One is pairwise self-attention, which generalizes standard dot-product attention and is fundamentally a set operator. The other is patchwise self-attention, which is strictly more powerful than convolution. Our pairwise self-attention networks match or outperform their convolutional counterparts, and the patchwise models substantially outperform the convolutional baselines. We also conduct experiments that probe the robustness of learned representations and conclude that self-attention networks may have significant benefits in terms of robustness and generalization.

Key facts. Cited by 313 papers in the map (data through 7 Oct 2026). First posted on arXiv 28 Apr 2020.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 119; 2022: 76; 2023: 58; 2024: 42; 2025: 17; 2026: 1.

313 papers.

Sort:

By area:

  1. 2021Swin Transformer: Hierarchical Vision Transformer using Shifted Windowscited by 12,082
  2. 2021Emerging Properties in Self-Supervised Vision Transformerscited by 6,893
  3. 2021Pyramid Vision Transformer: A Versatile Backbone for Dense Prediction without Convolutionscited by 1,960
  4. 2021Vision Transformers for Dense Predictioncited by 1,789
  5. 2021An Empirical Study of Training Self-Supervised Vision Transformerscited by 1,610
  6. 2021Is Space-Time Attention All You Need for Video Understanding?cited by 1,609
  7. 2021Tokens-to-Token ViT: Training Vision Transformers from Scratch on ImageNetcited by 1,096
  8. 2021Perceiver: General Perception with Iterative Attentioncited by 1,089
  9. 2021Transformers in Vision: A Surveycited by 1,009
  10. 2021Multiscale Vision Transformerscited by 896
  11. 2021CrossViT: Cross-Attention Multi-Scale Vision Transformer for Image Classificationcited by 806
  12. 2021Going deeper with Image Transformerscited by 759
38 more
  1. 2021LeViT: a Vision Transformer in ConvNet's Clothing for Faster Inferencecited by 475
  2. 2021VATT: Transformers for Multimodal Self-Supervised Learning from Raw Video, Audio and Textcited by 427
  3. 2021Early Convolutions Help Transformers See Bettercited by 395
  4. 2022Point Transformer V2: Grouped Vector Attention and Partition-based Poolingcited by 384
  5. 2021Bottleneck Transformers for Visual Recognitioncited by 364
  6. 2021ResMLP: Feedforward networks for image classification with data-efficient trainingcited by 338
  7. 2021XCiT: Cross-Covariance Image Transformerscited by 335
  8. 2021DeepViT: Towards Deeper Vision Transformercited by 288
  9. 2021Understanding Robustness of Transformers for Image Classificationcited by 250
  10. 2021Incorporating Convolution Designs into Visual Transformerscited by 248
  11. 2021Attention Mechanisms in Computer Vision: A Surveycited by 232
  12. 2021Weakly-supervised Video Anomaly Detection with Robust Temporal Feature Magnitude Learningcited by 226
  13. 2021Co-Scale Conv-Attentional Image Transformerscited by 224
  14. 2021Scaling Local Self-Attention for Parameter Efficient Visual Backbonescited by 220
  15. 2021UniT: Multimodal Multitask Learning with a Unified Transformercited by 195
  16. 2021Embracing Single Stride 3D Object Detector with Sparse Transformercited by 193
  17. 2021Are Transformers More Robust Than CNNs?cited by 187
  18. 2021AutoFormer: Searching Transformers for Visual Recognitioncited by 186
  19. 2022Stratified Transformer for 3D Point Cloud Segmentationcited by 166
  20. 2024PointMamba: A Simple State Space Model for Point Cloud Analysiscited by 166
  21. 2021VOLO: Vision Outlooker for Visual Recognitioncited by 164
  22. 2021Visformer: The Vision-friendly Transformercited by 137
  23. 2021SOFT: Softmax-free Transformer with Linear Complexitycited by 128
  24. 2021TokenLearner: What Can 8 Learned Tokens Do for Images and Videos?cited by 127
  25. 2021RegionViT: Regional-to-Local Attention for Vision Transformerscited by 126
  26. 2021ForgeryNet: A Versatile Benchmark for Comprehensive Forgery Analysiscited by 124
  27. 2023Spherical Transformer for LiDAR-based 3D Recognitioncited by 124
  28. 2023Transformer-Based Visual Segmentation: A Surveycited by 124
  29. 2021TS-CAM: Token Semantic Coupled Attention Map for Weakly Supervised Object Localizationcited by 122
  30. 2021Generative Adversarial Transformerscited by 120
  31. 2021A Survey of Visual Transformerscited by 117
  32. 2022ReSTR: Convolution-free Referring Image Segmentation Using Transformerscited by 112
  33. 2022Global Tracking Transformerscited by 110
  34. 2021Beyond Self-attention: External Attention using Two Linear Layers for Visual Taskscited by 109
  35. 2021LambdaNetworks: Modeling Long-Range Interactions Without Attentioncited by 107
  36. 2021SSTVOS: Sparse Spatiotemporal Transformers for Video Object Segmentationcited by 106
  37. 2022Vision Transformer for NeRF-Based View Synthesis from a Single Input Imagecited by 104
  38. 2021Variational Relational Point Completion Networkcited by 103

50 most cited of 313 All 313 (JSON)