mlsanityData to 14 Oct 2026

Neural Attention Models in Deep Learning: Survey and Taxonomy

arXiv 2112.05909 ↗First posted 11 Dec 2021cs.LG cs.AIcited by 8

Not on the map. Why?

Abstract (arXiv)

Attention is a state of arousal capable of dealing with limited processing bottlenecks in human beings by focusing selectively on one piece of information while ignoring other perceptible information. For decades, concepts and functions of attention have been studied in philosophy, psychology, neuroscience, and computing. Currently, this property has been widely explored in deep neural networks. Many different neural attention models are now available and have been a very active research area over the past six years. From the theoretical standpoint of attention, this survey provides a critical analysis of major neural attention models. Here we propose a taxonomy that corroborates with theoretical aspects that predate Deep Learning. Our taxonomy provides an organizational structure that asks new questions and structures the understanding of existing attentional mechanisms. In particular, 17 criteria derived from psychology and neuroscience classic studies are formulated for qualitative comparison and critical analysis on the 51 main models found on a set of more than 650 papers analyzed. Also, we highlight several theoretical issues that have not yet been explored, including discussions about biological plausibility, highlight current research trends, and provide insights for the future.

Key facts. Cited by 8 papers in the map (data through 14 Oct 2026). Builds on 58 of its 108 references that are in the map. First posted on arXiv 11 Dec 2021.

Builds on

58 of 108 references.

References · 58

  1. 2017Attention Is All You Needcited by 60,361
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,399
  3. 2020Language Models are Few-Shot Learnerscited by 32,419
  4. 2017Graph Attention Networkscited by 8,923
  5. 2016Layer Normalizationcited by 5,669
  6. 2017Squeeze-and-Excitation Networkscited by 4,789
  7. 2014Neural Machine Translation by Jointly Learning to Align and Translatecited by 4,548
  8. 2015Spatial Transformer Networkscited by 1,429
  9. 2015Show, Attend and Tell: Neural Image Caption Generation with Visual Attentioncited by 1,423
  10. 2016Paying More Attention to Attention: Improving the Performance of Convolutional Neural Networks via Attention Transfercited by 1,077
  11. 2015Teaching Machines to Read and Comprehendcited by 1,049
  12. 2015Effective Approaches to Attention-based Neural Machine Translationcited by 1,026
38 more
  1. 2018Dual Attention Network for Scene Segmentationcited by 1,003
  2. 2015Pointer Networkscited by 895
  3. 2019Heterogeneous Graph Attention Networkcited by 734
  4. 2014Neural Turing Machinescited by 644
  5. 2016Adaptive Computation Time for Recurrent Neural Networkscited by 506
  6. 2016Tutorial on Variational Autoencoderscited by 409
  7. 2014Memory Networkscited by 363
  8. 2014Recurrent Models of Visual Attentioncited by 349
  9. 2016Top-down Neural Attention by Excitation Backpropcited by 321
  10. 2019Attentive Neural Processescited by 283
  11. 2018Bilinear Attention Networkscited by 280
  12. 2016Hierarchical Question-Image Co-Attention for Visual Question Answeringcited by 278
  13. 2016Bidirectional Attention Flow for Machine Comprehensioncited by 278
  14. 2018Attention Gated Networks: Learning to Leverage Salient Regions in Medical Imagescited by 272
  15. 2015Highway Networkscited by 260
  16. 2016Knowing When to Look: Adaptive Attention via A Visual Sentinel for Image Captioningcited by 232
  17. 2015DRAW: A Recurrent Neural Network For Image Generationcited by 191
  18. 2018Compositional Attention Networks for Machine Reasoningcited by 183
  19. 2018A2-Nets: Double Attention Networkscited by 136
  20. 2019An Attentive Survey of Attention Modelscited by 124
  21. 2015Ask Me Anything: Dynamic Memory Networks for Natural Language Processingcited by 117
  22. 2020IMRAM: Iterative Matching with Recurrent Attention Memory for Cross-Modal Image-Text Retrievalcited by 115
  23. 2016Dual Attention Networks for Multimodal Reasoning and Matchingcited by 114
  24. 2019Attention in Natural Language Processingcited by 101
  25. 2017Structured Attention Networkscited by 95
  26. 2016A Convolutional Attention Network for Extreme Summarization of Source Codecited by 91
  27. 2017DiSAN: Directional Self-Attention Network for RNN/CNN-Free Language Understandingcited by 71
  28. 2019Hyperbolic Graph Attention Networkcited by 70
  29. 2015Reasoning about Entailment with Neural Attentioncited by 69
  30. 2016Dynamic Memory Networks for Visual and Textual Question Answeringcited by 62
  31. 2016Dynamic Coattention Networks For Question Answeringcited by 49
  32. 2020Learning to Combine Top-Down and Bottom-Up Signals in Recurrent Neural Networks with Attention over Modulescited by 38
  33. 2020Modality Shifting Attention Network for Multi-modal Video Question Answeringcited by 38
  34. 2018Latent Alignment and Variational Attentioncited by 36
  35. 2018Sparse Attentive Backtracking: Temporal CreditAssignment Through Remindingcited by 34
  36. 2016Strategic Attentive Writer for Learning Macro-Actionscited by 30
  37. 2017Visual Reference Resolution using Attention Memory for Visual Dialogcited by 28
  38. 2017Few-shot Autoregressive Density Estimation: Towards Learning to Learn Distributionscited by 24

50 most cited of 58 All 58 (JSON)

Built on by

Cited by year

Citing papers per year: 2022: 5; 2023: 1; 2024: 1; 2025: 1.

8 papers.

  1. 2024Attention Heads of Large Language Models: A Surveycited by 67
  2. 2022Attention-based Random Forest and Contamination Modelcited by 10
  3. 2022AGBoost: Attention-based Modification of Gradient Boosting Machinecited by 5
  4. 2022Attention and Self-Attention in Random Forestscited by 4
  5. 2023Attention Schema in Neural Agentscited by 4
  6. 2022Improved Anomaly Detection by Using the Attention-Based Isolation Forestcited by 2
  7. 2022LARF: Two-level Attention-based Random Forests with a Mixture of Contamination Modelscited by 0
  8. 2025Ensemble-Based Survival Models with the Self-Attended Beran Estimator Predictionscited by 0