mlsanityData to 7 Oct 2026

A General Survey on Attention Mechanisms in Deep Learning

arXiv 2203.14263 ↗First posted 27 Mar 2022cs.LGcited by 64

Not on the map. Why?

Abstract (arXiv)

Attention is an important mechanism that can be employed for a variety of deep learning models across many different domains and tasks. This survey provides an overview of the most important attention mechanisms proposed in the literature. The various attention mechanisms are explained by means of a framework consisting of a general attention model, uniform notation, and a comprehensive taxonomy of attention mechanisms. Furthermore, the various measures for evaluating attention models are reviewed, and methods to characterize the structure of attention models based on the proposed framework are discussed. Last, future work in the field of attention models is considered.

Key facts. Cited by 64 papers in the map (data through 7 Oct 2026). Builds on 77 of its 127 references that are in the map. First posted on arXiv 27 Mar 2022.

Builds on

77 of 127 references.

References · 77

  1. 2017Attention Is All You Needcited by 59,932
  2. 2017Graph Attention Networkscited by 8,854
  3. 2016Layer Normalizationcited by 5,579
  4. 2014Neural Machine Translation by Jointly Learning to Align and Translatecited by 4,530
  5. 2019Transformer-XL: Attentive Language Models Beyond a Fixed-Length Contextcited by 1,830
  6. 2020Reformer: The Efficient Transformercited by 1,658
  7. 2021TransUNet: Transformers Make Strong Encoders for Medical Image Segmentationcited by 1,552
  8. 2020Linformer: Self-Attention with Linear Complexitycited by 1,462
  9. 2015Show, Attend and Tell: Neural Image Caption Generation with Visual Attentioncited by 1,421
  10. 2018Attention U-Net: Learning Where to Look for the Pancreascited by 1,420
  11. 2017Bottom-Up and Top-Down Attention for Image Captioning and Visual Question Answeringcited by 1,310
  12. 2014On the Properties of Neural Machine Translation: Encoder-Decoder Approachescited by 1,303
38 more
  1. 2015Effective Approaches to Attention-based Neural Machine Translationcited by 1,024
  2. 2019BERT4Rec: Sequential Recommendation with Bidirectional Encoder Representations from Transformercited by 952
  3. 2019Attention is not Explanationcited by 916
  4. 2018Self-Attention Generative Adversarial Networkscited by 809
  5. 2020Efficient Transformers: A Surveycited by 784
  6. 2018Image Transformercited by 770
  7. 2017Dynamic Routing Between Capsulescited by 689
  8. 2014Neural Turing Machinescited by 630
  9. 2019Attention is not not Explanationcited by 629
  10. 2015Attention-Based Models for Speech Recognitioncited by 400
  11. 2014Recurrent Models of Visual Attentioncited by 344
  12. 2019Meshed-Memory Transformer for Image Captioningcited by 343
  13. 2017A Structured Self-attentive Sentence Embeddingcited by 318
  14. 2018Deep Audio-Visual Speech Recognitioncited by 318
  15. 2020Exploring Self-attention for Image Recognitioncited by 313
  16. 2016Hierarchical Question-Image Co-Attention for Visual Question Answeringcited by 278
  17. 2016Bidirectional Attention Flow for Machine Comprehensioncited by 278
  18. 2015Stacked Attention Networks for Image Question Answeringcited by 273
  19. 2016Joint CTC-Attention based End-to-End Speech Recognition using Multi-task Learningcited by 254
  20. 2017On the Automatic Generation of Medical Imaging Reportscited by 215
  21. 2018End-to-End Dense Video Captioning with Masked Transformercited by 214
  22. 2020Synthesizer: Rethinking Self-Attention in Transformer Modelscited by 204
  23. 2018QANet: Combining Local Convolution with Global Self-Attention for Reading Comprehensioncited by 164
  24. 2015End-to-End Attention-based Large Vocabulary Speech Recognitioncited by 148
  25. 2020Lite Transformer with Long-Short Range Attentioncited by 127
  26. 2019An Attentive Survey of Attention Modelscited by 123
  27. 2016Human Attention in Visual Question Answering: Do Humans and Deep Networks Look at the Same Regions?cited by 111
  28. 2020GCAN: Graph-aware Co-Attention Networks for Explainable Fake News Detection on Social Mediacited by 108
  29. 2017Attend and Diagnose: Clinical Time Series Analysis using Attention Modelscited by 105
  30. 2018Attention-based Graph Neural Network for Semi-supervised Learningcited by 100
  31. 2019Knowledge-Enriched Transformer for Emotion Detection in Textual Conversationscited by 92
  32. 2017Interactive Attention Networks for Aspect-Level Sentiment Classificationcited by 91
  33. 2019Multi-scale self-guided attention for medical image segmentationcited by 84
  34. 2019Multi-Agent Game Abstraction via Graph Attention Neural Networkcited by 81
  35. 2017DiSAN: Directional Self-Attention Network for RNN/CNN-Free Language Understandingcited by 71
  36. 2018Summarizing Videos with Attentioncited by 69
  37. 2018Unsupervised Attention-guided Image to Image Translationcited by 58
  38. 2017Massive Exploration of Neural Machine Translation Architecturescited by 55

50 most cited of 77 All 77 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2022: 4; 2023: 11; 2024: 15; 2025: 22; 2026: 12.

64 papers.

Sort:

By area:

  1. 2024Attention Heads of Large Language Models: A Surveycited by 65
  2. 2025VScan: Rethinking Visual Token Reduction for Efficient Large Vision-Language Modelscited by 42
  3. 2022Adaptable Butterfly Accelerator for Attention-based NNs via Hardware and Algorithm Co-designcited by 29
  4. 2023A Machine Learning-oriented Survey on Tiny Machine Learningcited by 26
  5. 2024Geometry of Lightning Self-Attention: Identifiability and Dimensioncited by 16
  6. 2023FRAMU: Attention-based Machine Unlearning using Federated Reinforcement Learningcited by 12
  7. 2026ApET: Approximation-Error Guided Token Compression for Efficient VLMscited by 11
  8. 2025ProxyAttn: Guided Sparse Attention via Representative Headscited by 9
  9. 2025Not All Features Deserve Attention: Graph-Guided Dependency Learning for Tabular Data Generation with Language Modelscited by 7
  10. 2023Track Mix Generation on Music Streaming Services using Transformerscited by 6
  11. 2024Attention Beats Linear for Fast Implicit Neural Representation Generationcited by 6
  12. 2026LLMind: Bio-inspired Training-free Adaptive Visual Representations for Vision-Language Modelscited by 6
38 more
  1. 2023A Survey of Deep Learning: From Activations to Transformerscited by 5
  2. 2024A Survey of Transformer Enabled Time Series Synthesiscited by 5
  3. 2024RNNs, CNNs and Transformers in Human Action Recognition: A Survey and a Hybrid Modelcited by 5
  4. 2022Attention and Self-Attention in Random Forestscited by 4
  5. 2023Breast Cancer Classification with Enhanced Interpretability: DALAResNet50 and DT Grad-CAMcited by 4
  6. 2024Enhancing Efficiency in Vision Transformer Networks: Design Techniques and Insightscited by 4
  7. 2025Demographic Attributes Prediction from Speech Using WavLM Embeddingscited by 4
  8. 2025AttZoom: Attention Zoom for Better Visual Featurescited by 4
  9. 2025AutoPrune: Each Complexity Deserves a Pruning Policycited by 4
  10. 2024Applications of interpretable deep learning in neuroimaging: a comprehensive reviewcited by 3
  11. 2024What comes after transformers? -- A selective survey connecting ideas in deep learningcited by 3
  12. 2023Brain Tumor Segmentation Based on Deep Learning, Attention Mechanisms, and Energy-Based Uncertainty Predictioncited by 2
  13. 2025Risk-Aware Reinforcement Learning for Autonomous Driving: Improving Safety When Driving through Intersectioncited by 2
  14. 2025GeloVec: Higher Dimensional Geometric Smoothing for Coherent Visual Feature Extraction in Image Segmentationcited by 2
  15. 2025KairosAD: A SAM-Based Model for Industrial Anomaly Detection on Embedded Devicescited by 2
  16. 2023WATUNet: A Deep Neural Network for Segmentation of Volumetric Sweep Imaging Ultrasoundcited by 1
  17. 2024Revisiting Generative Adversarial Networks for Binary Semantic Segmentation on Imbalanced Datasetscited by 1
  18. 2024Towards Fault Tolerance in Multi-Agent Reinforcement Learningcited by 1
  19. 2025Alternatives To Next Token Prediction In Text Generation -- A Surveycited by 1
  20. 2026Decoupled Similarity for Task-Aware Token Pruning in Large Vision-Language Modelscited by 1
  21. 2026Attention-based graph neural networks: a surveycited by 1
  22. 2022LARF: Two-level Attention-based Random Forests with a Mixture of Contamination Modelscited by 0
  23. 2022Encoder-Decoder Model for Suffix Prediction in Predictive Monitoringcited by 0
  24. 2023GIPA: A General Information Propagation Algorithm for Graph Learningcited by 0
  25. 2023Neural Attention Forests: Transformer-Based Forest Improvementcited by 0
  26. 2023Assessment of few-hits machine learning classification algorithms for low energy physics in liquid argon detectorscited by 0
  27. 2023Empowering Precision Medicine: AI-Driven Schizophrenia Diagnosis via EEG Signals: A Comprehensive Review from 2002-2023cited by 0
  28. 2024Machine Learning and Data Analysis Using Posets: A Surveycited by 0
  29. 2024Developing An Attention-Based Ensemble Learning Framework for Financial Portfolio Optimisationcited by 0
  30. 2024A Multi-Modal Explainability Approach for Human-Aware Robots in Multi-Party Conversationcited by 0
  31. 2024BiLSTM and Attention-Based Modulation Classification of Realistic Wireless Signalscited by 0
  32. 2024RotCAtt-TransUNet++: Novel Deep Neural Network for Sophisticated Cardiac Segmentationcited by 0
  33. 2025AoI-Sensitive Data Forwarding with Distributed Beamforming in UAV-Assisted IoTcited by 0
  34. 2025A HEART for the environment: Transformer-Based Spatiotemporal Modeling for Air Quality Predictioncited by 0
  35. 2025Bringing together invertible UNets with invertible attention modules for memory-efficient diffusion modelscited by 0
  36. 2025Wavelet Probabilistic Recurrent Convolutional Network for Multivariate Time Series Classificationcited by 0
  37. 2025MAGNet: A Multi-Scale Attention-Guided Graph Fusion Network for DRC Violation Detectioncited by 0
  38. 2025Ensemble-Based Survival Models with the Self-Attended Beran Estimator Predictionscited by 0

50 most cited of 64 All 64 (JSON)