mlsanityData to 7 Oct 2026

An Attentive Survey of Attention Models

arXiv 1904.02874 ↗First posted 5 Apr 2019cs.LG stat.MLcited by 123

Not on the map. Why?

Abstract (arXiv)

Attention Model has now become an important concept in neural networks that has been researched within diverse application domains. This survey provides a structured and comprehensive overview of the developments in modeling attention. In particular, we propose a taxonomy which groups existing techniques into coherent categories. We review salient neural architectures in which attention has been incorporated, and discuss applications in which modeling attention has shown a significant impact. We also describe how attention has been used to improve the interpretability of neural networks. Finally, we discuss some future research directions in attention. We hope this survey will provide a succinct introduction to attention models and guide practitioners while developing approaches for their applications.

Key facts. Cited by 123 papers in the map (data through 7 Oct 2026). First posted on arXiv 5 Apr 2019.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 40; 2022: 32; 2023: 14; 2024: 21; 2025: 13; 2026: 3.

123 papers.

Sort:

By area:

  1. 2021Transformers in Vision: A Surveycited by 1,009
  2. 2022Transformers in Time Series: A Surveycited by 489
  3. 2021Dynamic Neural Networks: A Surveycited by 405
  4. 2022Transformers in Medical Imaging: A Surveycited by 245
  5. 2021Attention Mechanisms in Computer Vision: A Surveycited by 232
  6. 2021Efficient Deep Learning: A Survey on Making Deep Learning Models Smaller, Faster, and Bettercited by 153
  7. 2024Mobile Edge Intelligence for Large Language Models: A Contemporary Surveycited by 125
  8. 2021Recent advances and clinical applications of deep learning in medical image analysiscited by 104
  9. 2021AI in Finance: Challenges, Techniques and Opportunitiescited by 81
  10. 2024Attention Heads of Large Language Models: A Surveycited by 65
  11. 2022A General Survey on Attention Mechanisms in Deep Learningcited by 64
  12. 2021Attention, please! A survey of Neural Attention Models in Deep Learningcited by 50
38 more
  1. 2022Context-Enhanced Stereo Transformercited by 37
  2. 2022Attention Mechanism in Neural Networks: Where it Comes and Where it Goescited by 29
  3. 2021Attentive Neural Controlled Differential Equations for Time-series Classification and Forecastingcited by 28
  4. 2021Exploring Transformers in Natural Language Generation: GPT, BERT, and XLNetcited by 27
  5. 2023Object Detection with Transformers: A Reviewcited by 22
  6. 2024InstInfer: In-Storage Attention Offloading for Cost-Effective Long-Context LLM Inferencecited by 19
  7. 2021Deep Learning for Android Malware Defenses: a Systematic Literature Reviewcited by 18
  8. 2021Real-time Instance Segmentation of Surgical Instruments using Attention and Multi-scale Feature Fusioncited by 17
  9. 2023Label-Efficient Learning in Agriculture: A Comprehensive Reviewcited by 16
  10. 2021ACE-NODE: Attentive Co-Evolving Neural Ordinary Differential Equationscited by 15
  11. 2022TiSAT: Time Series Anomaly Transformercited by 15
  12. 2022A survey on attention mechanisms for medical applications: are we moving towards better algorithms?cited by 15
  13. 2023Interpretation of Time-Series Deep Models: A Surveycited by 15
  14. 2022AOE-Net: Entities Interactions Modeling with Adaptive Attention Mechanism for Temporal Action Proposals Generationcited by 14
  15. 2022ProstAttention-Net: A deep attention model for prostate cancer segmentation by aggressiveness in MRI scanscited by 14
  16. 2023Exploiting Unlabelled Photos for Stronger Fine-Grained SBIRcited by 14
  17. 2024Deep Learning for Satellite Image Time Series Analysis: A Reviewcited by 13
  18. 2021AEI: Actors-Environment Interaction with Adaptive Attention for Temporal Action Proposals Generationcited by 11
  19. 2022LightX3ECG: A Lightweight and eXplainable Deep Learning System for 3-lead Electrocardiogram Classificationcited by 11
  20. 2022Generalization in Neural Networks: A Broad Surveycited by 11
  21. 2022Attention-based Random Forest and Contamination Modelcited by 10
  22. 2021A Hierarchical Conditional Random Field-based Attention Mechanism Approach for Gastric Histopathology Image Classificationcited by 9
  23. 2021pix2rule: End-to-end Neuro-symbolic Rule Learningcited by 9
  24. 2024Attention versus Contrastive Learning of Tabular Data -- A Data-centric Benchmarkingcited by 9
  25. 2025Gated KalmaNet: A Fading Memory Layer Through Test-Time Ridge Regressioncited by 9
  26. 2021GraphTheta: A Distributed Graph Neural Network Learning System With Flexible Training Strategycited by 8
  27. 2021Neural Attention Models in Deep Learning: Survey and Taxonomycited by 8
  28. 2022Attention-embedded Quadratic Network (Qttention) for Effective and Interpretable Bearing Fault Diagnosiscited by 8
  29. 2024Automated Program Repair: Emerging trends pose and expose problems for benchmarkscited by 7
  30. 2024Is Mamba Compatible with Trajectory Optimization in Offline Reinforcement Learning?cited by 7
  31. 2024QUITO: Accelerating Long-Context Reasoning through Query-Guided Context Compressioncited by 7
  32. 2025Not All Features Deserve Attention: Graph-Guided Dependency Learning for Tabular Data Generation with Language Modelscited by 7
  33. 2021Attention Models for Point Clouds in Deep Learning: A Surveycited by 6
  34. 2021Set-to-Sequence Methods in Machine Learning: a Reviewcited by 6
  35. 2021Review of end-to-end speech synthesis technology based on deep learningcited by 6
  36. 2021MEDUSA: Multi-scale Encoder-Decoder Self-Attention Deep Neural Network Architecture for Medical Image Analysiscited by 6
  37. 2021Multi-Attention Multiple Instance Learningcited by 6
  38. 2021Depth as Attention for Face Representation Learningcited by 5

50 most cited of 123 All 123 (JSON)