mlsanityData to 14 Oct 2026

Attention Mechanisms in Computer Vision: A Survey

arXiv 2111.07624 ↗First posted 15 Nov 2021cs.CVcited by 232

Not on the map. Why?

Abstract (arXiv)

Humans can naturally and effectively find salient regions in complex scenes. Motivated by this observation, attention mechanisms were introduced into computer vision with the aim of imitating this aspect of the human visual system. Such an attention mechanism can be regarded as a dynamic weight adjustment process based on features of the input image. Attention mechanisms have achieved great success in many visual tasks, including image classification, object detection, semantic segmentation, video understanding, image generation, 3D vision, multi-modal tasks and self-supervised learning. In this survey, we provide a comprehensive review of various attention mechanisms in computer vision and categorize them according to approach, such as channel attention, spatial attention, temporal attention and branch attention; a related repository https://github.com/MenghaoGuo/Awesome-Vision-Attentions is dedicated to collecting related work. We also suggest future directions for attention mechanism research.

Key facts. Cited by 232 papers in the map (data through 14 Oct 2026). Builds on 146 of its 198 references that are in the map. First posted on arXiv 15 Nov 2021.

Builds on

146 of 198 references.

References · 146

  1. 2017Attention Is All You Needcited by 60,361
  2. 2015Deep Residual Learning for Image Recognitioncited by 52,096
  3. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,399
  4. 2014Adam: A Method for Stochastic Optimizationcited by 37,240
  5. 2020Language Models are Few-Shot Learnerscited by 32,419
  6. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 30,186
  7. 2021Swin Transformer: Hierarchical Vision Transformer using Shifted Windowscited by 12,139
  8. 2020End-to-End Object Detection with Transformerscited by 7,414
  9. 2021Masked Autoencoders Are Scalable Vision Learnerscited by 7,403
  10. 2015Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shiftcited by 7,005
  11. 2021Emerging Properties in Self-Supervised Vision Transformerscited by 6,968
  12. 2016Layer Normalizationcited by 5,669
38 more
  1. 2017Squeeze-and-Excitation Networkscited by 4,789
  2. 2014Neural Machine Translation by Jointly Learning to Align and Translatecited by 4,548
  3. 2014Going Deeper with Convolutionscited by 4,241
  4. 2016Gaussian Error Linear Units (GELUs)cited by 3,502
  5. 2020Deformable DETR: Deformable Transformers for End-to-End Object Detectioncited by 3,422
  6. 2021SegFormer: Simple and Efficient Design for Semantic Segmentation with Transformerscited by 2,940
  7. 2019XLNet: Generalized Autoregressive Pretraining for Language Understandingcited by 2,811
  8. 2016Aggregated Residual Transformations for Deep Neural Networkscited by 2,546
  9. 2016Pyramid Scene Parsing Networkcited by 2,479
  10. 2018CBAM: Convolutional Block Attention Modulecited by 2,477
  11. 2021BEiT: BERT Pre-Training of Image Transformerscited by 2,375
  12. 2017Non-local Neural Networkscited by 2,293
  13. 2021Pyramid Vision Transformer: A Versatile Backbone for Dense Prediction without Convolutionscited by 1,962
  14. 2019Transformer-XL: Attentive Language Models Beyond a Fixed-Length Contextcited by 1,850
  15. 2021An Empirical Study of Training Self-Supervised Vision Transformerscited by 1,616
  16. 2020Rethinking Attention with Performerscited by 1,601
  17. 2018Group Normalizationcited by 1,524
  18. 2017Deformable Convolutional Networkscited by 1,488
  19. 2020Point Transformercited by 1,470
  20. 2021MLP-Mixer: An all-MLP Architecture for Visioncited by 1,446
  21. 2018Attention U-Net: Learning Where to Look for the Pancreascited by 1,435
  22. 2015Spatial Transformer Networkscited by 1,429
  23. 2015Show, Attend and Tell: Neural Image Caption Generation with Visual Attentioncited by 1,423
  24. 2020Sharpness-Aware Minimization for Efficiently Improving Generalizationcited by 1,417
  25. 2020Rethinking Semantic Segmentation from a Sequence-to-Sequence Perspective with Transformerscited by 1,369
  26. 2018Image Super-Resolution Using Very Deep Residual Channel Attention Networkscited by 1,264
  27. 2018Self-Attention with Relative Position Representationscited by 1,120
  28. 2021Tokens-to-Token ViT: Training Vision Transformers from Scratch on ImageNetcited by 1,101
  29. 2021Transformers in Vision: A Surveycited by 1,013
  30. 2018Dual Attention Network for Scene Segmentationcited by 1,003
  31. 2019VL-BERT: Pre-training of Generic Visual-Linguistic Representationscited by 954
  32. 2021CvT: Introducing Convolutions to Vision Transformerscited by 947
  33. 2017Revisiting Unreasonable Effectiveness of Data in Deep Learning Eracited by 919
  34. 2020Pre-Trained Image Processing Transformercited by 893
  35. 2018Self-Attention Generative Adversarial Networkscited by 809
  36. 2021Going deeper with Image Transformerscited by 764
  37. 2020PCT: Point cloud transformercited by 732
  38. 2014Spatial Pyramid Pooling in Deep Convolutional Networks for Visual Recognitioncited by 720

50 most cited of 146 All 146 (JSON)

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2021: 1; 2022: 47; 2023: 65; 2024: 60; 2025: 40; 2026: 19.

232 papers.

Sort:

By area:

  1. 2023MDTv2: Masked Diffusion Transformer is a Strong Image Synthesizercited by 262
  2. 2022SegNeXt: Rethinking Convolutional Attention Design for Semantic Segmentationcited by 252
  3. 2022Transformers in Medical Imaging: A Surveycited by 245
  4. 2022Visual Attention Networkcited by 210
  5. 2023Large-scale Multi-Modal Pre-trained Models: A Comprehensive Surveycited by 153
  6. 2023Spike-driven Transformercited by 146
  7. 2022Towards An End-to-End Framework for Flow-Guided Video Inpaintingcited by 113
  8. 2023Swin3D: A Pretrained Transformer Backbone for 3D Indoor Scene Understandingcited by 113
  9. 2022Attention Spiking Neural Networkscited by 107
  10. 2023Large Selective Kernel Network for Remote Sensing Object Detectioncited by 98
  11. 2023Collaborative Perception in Autonomous Driving: Methods, Datasets and Challengescited by 87
  12. 2022BrainGB: A Benchmark for Brain Network Analysis with Graph Neural Networkscited by 75
38 more
  1. 2024Sora as a World Model? A Complete Survey on Text-to-Video Generationcited by 53
  2. 2023Speck: A Smart event-based Vision Sensor with a low latency 327K Neuron Convolutional Neuronal Network Processing Pipelinecited by 52
  3. 2024LSKNet: A Foundation Lightweight Backbone for Remote Sensingcited by 44
  4. 2023Change Detection Methods for Remote Sensing in the Last Decade: A Comprehensive Reviewcited by 41
  5. 2022Visual Attention Methods in Deep Learning: An In-Depth Surveycited by 37
  6. 2022Context-Enhanced Stereo Transformercited by 37
  7. 2022Multi-Task Learning with Multi-Query Transformer for Dense Predictioncited by 34
  8. 2023YOLO-MS: Rethinking Multi-Scale Representation Learning for Real-time Object Detectioncited by 33
  9. 2022HiFuse: Hierarchical Multi-Scale Feature Fusion Network for Medical Image Classificationcited by 30
  10. 2022DAE-Former: Dual Attention-guided Efficient Transformer for Medical Image Segmentationcited by 27
  11. 20223D Vision with Transformers: A Surveycited by 21
  12. 2023Solving Attention Kernel Regression Problem via Pre-conditionercited by 20
  13. 2024AUCSeg: AUC-oriented Pixel-level Long-tail Semantic Segmentationcited by 18
  14. 2024Diffusion-Based Visual Art Creation: A Survey and New Perspectivescited by 17
  15. 2022A survey on attention mechanisms for medical applications: are we moving towards better algorithms?cited by 15
  16. 2022DPA-1: Pretraining of Attention-based Deep Potential Model for Molecular Simulationcited by 15
  17. 2023Towards Artificial General Intelligence (AGI) in the Internet of Things (IoT): Opportunities and Challengescited by 14
  18. 2024Cross-modulated Attention Transformer for RGBT Trackingcited by 14
  19. 2024FactorizePhys: Matrix Factorization for Multidimensional Attention in Remote Physiological Sensingcited by 14
  20. 2022Quantum Neural Network Classifiers: A Tutorialcited by 13
  21. 2023Inherent Redundancy in Spiking Neural Networkscited by 13
  22. 2023Low-Resolution Self-Attention for Semantic Segmentationcited by 12
  23. 2022OccamNets: Mitigating Dataset Bias by Favoring Simpler Hypothesescited by 11
  24. 2022SEMICON: A Learning-to-hash Solution for Large-scale Fine-grained Image Retrievalcited by 11
  25. 2023EEG motor imagery decoding: A framework for comparative analysis with channel attention mechanismscited by 11
  26. 2024Graph Representation Learning via Causal Diffusion for Out-of-Distribution Recommendationcited by 11
  27. 2023Disambiguated Attention Embedding for Multi-Instance Partial-Label Learningcited by 10
  28. 2024FSTA-SNN:Frequency-based Spatial-Temporal Attention Module for Spiking Neural Networkscited by 10
  29. 2023Attention Mechanisms in Medical Image Segmentation: A Surveycited by 9
  30. 2023Understanding Self-attention Mechanism via Dynamical System Perspectivecited by 9
  31. 2024EyeTrans: Merging Human and Machine Attention for Neural Code Summarizationcited by 9
  32. 2024Theoretically Achieving Continuous Representation of Oriented Bounding Boxescited by 9
  33. 2022Visual Attention-based Self-supervised Absolute Depth Estimation using Geometric Priors in Autonomous Drivingcited by 8
  34. 2023Conversational AI-Powered Design: ChatGPT as Designer, User, and Productcited by 8
  35. 2023Channel prior convolutional attention for medical image segmentationcited by 8
  36. 2022Large-Kernel Attention for 3D Medical Image Segmentationcited by 7
  37. 2022ESKNet-An enhanced adaptive selection kernel convolution for breast tumors segmentationcited by 7
  38. 2023Better "CMOS" Produces Clearer Images: Learning Space-Variant Blur Estimation for Blind Image Super-Resolutioncited by 7

50 most cited of 232 All 232 (JSON)