mlsanityData to 14 Oct 2026

Attention Mechanism in Neural Networks: Where it Comes and Where it Goes

arXiv 2204.13154 ↗First posted 27 Apr 2022cs.LGcited by 29

Not on the map. Why?

Abstract (arXiv)

A long time ago in the machine learning literature, the idea of incorporating a mechanism inspired by the human visual system into neural networks was introduced. This idea is named the attention mechanism, and it has gone through a long development period. Today, many works have been devoted to this idea in a variety of tasks. Remarkable performance has recently been demonstrated. The goal of this paper is to provide an overview from the early work on searching for ways to implement attention idea with neural networks until the recent trends. This review emphasizes the important milestones during this progress regarding different tasks. By this way, this study aims to provide a road map for researchers to explore the current development and get inspired for novel approaches beyond the attention.

Key facts. Cited by 29 papers in the map (data through 14 Oct 2026). Builds on 111 of its 195 references that are in the map. First posted on arXiv 27 Apr 2022.

Builds on

111 of 195 references.

References · 111

  1. 2017Attention Is All You Needcited by 60,361
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,399
  3. 2020Language Models are Few-Shot Learnerscited by 32,419
  4. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 30,186
  5. 2020End-to-End Object Detection with Transformerscited by 7,414
  6. 2022Generative Adversarial Networkscited by 5,627
  7. 2014Neural Machine Translation by Jointly Learning to Align and Translatecited by 4,548
  8. 2019DistilBERT, a distilled version of BERT: smaller, faster, cheaper and lightercited by 4,307
  9. 2014Learning Phrase Representations using RNN Encoder-Decoder for Statistical Machine Translationcited by 4,057
  10. 2020Deformable DETR: Deformable Transformers for End-to-End Object Detectioncited by 3,422
  11. 2014Sequence to Sequence Learning with Neural Networkscited by 2,986
  12. 2019ALBERT: A Lite BERT for Self-supervised Learning of Language Representationscited by 2,790
38 more
  1. 2020DeBERTa: Decoding-enhanced BERT with Disentangled Attentioncited by 2,181
  2. 2020Transformers are RNNs: Fast Autoregressive Transformers with Linear Attentioncited by 2,022
  3. 2019BioBERT: a pre-trained biomedical language representation model for biomedical text miningcited by 1,947
  4. 2019Transformer-XL: Attentive Language Models Beyond a Fixed-Length Contextcited by 1,850
  5. 2020Big Bird: Transformers for Longer Sequencescited by 1,684
  6. 2020Reformer: The Efficient Transformercited by 1,668
  7. 2020Rethinking Attention with Performerscited by 1,601
  8. 2019Generating Long Sequences with Sparse Transformerscited by 1,546
  9. 2020MiniLM: Deep Self-Attention Distillation for Task-Agnostic Compression of Pre-Trained Transformerscited by 1,463
  10. 2015Show, Attend and Tell: Neural Image Caption Generation with Visual Attentioncited by 1,423
  11. 2019TinyBERT: Distilling BERT for Natural Language Understandingcited by 1,167
  12. 2018Self-Attention with Relative Position Representationscited by 1,120
  13. 2021Perceiver: General Perception with Iterative Attentioncited by 1,104
  14. 2015Effective Approaches to Attention-based Neural Machine Translationcited by 1,026
  15. 2019What Does BERT Look At? An Analysis of BERT's Attentioncited by 1,003
  16. 2014Show and Tell: A Neural Image Caption Generatorcited by 879
  17. 2019Analyzing Multi-Head Self-Attention: Specialized Heads Do the Heavy Lifting, the Rest Can Be Prunedcited by 856
  18. 2019SpanBERT: Improving Pre-training by Representing and Predicting Spanscited by 834
  19. 2018Self-Attention Generative Adversarial Networkscited by 809
  20. 2018Image Transformercited by 772
  21. 2018Universal Transformerscited by 647
  22. 2015Listen, Attend and Spellcited by 543
  23. 2019Stand-Alone Self-Attention in Vision Modelscited by 542
  24. 2019Graph Transformer Networkscited by 523
  25. 2019Patient Knowledge Distillation for BERT Model Compressioncited by 508
  26. 2020Efficient Content-Based Sparse Attention with Routing Transformerscited by 471
  27. 2015Attention-Based Models for Speech Recognitioncited by 401
  28. 2018Efficient Attention: Attention with Linear Complexitiescited by 368
  29. 2017A Deep Reinforced Model for Abstractive Summarizationcited by 365
  30. 2019Q-BERT: Hessian Based Ultra Low Precision Quantization of BERTcited by 359
  31. 2021Random Feature Attentioncited by 354
  32. 2014Recurrent Models of Visual Attentioncited by 349
  33. 2021XCiT: Cross-Covariance Image Transformerscited by 338
  34. 2015Visual7W: Grounded Question Answering in Imagescited by 318
  35. 2017A Structured Self-attentive Sentence Embeddingcited by 318
  36. 2019Q8BERT: Quantized 8Bit BERTcited by 300
  37. 2019On the Relationship between Self-Attention and Convolutional Layerscited by 300
  38. 2016Hierarchical Question-Image Co-Attention for Visual Question Answeringcited by 278

50 most cited of 111 All 111 (JSON)

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2022: 2; 2023: 5; 2024: 8; 2025: 9; 2026: 5.

29 papers.

Sort:

By area:

  1. 2023A Comprehensive Survey on Applications of Transformers for Deep Learning Taskscited by 71
  2. 2024Graph Transformers: A Surveycited by 32
  3. 2024A Review of Graph Neural Networks in Epidemic Modelingcited by 31
  4. 2025RF-DETR Object Detection vs YOLOv12 : A Study of Transformer-based and CNN-based Architectures for Single-Class and Multi-Class Greenfruit Detection in Complex Orchard Environments Under Label Ambiguitycited by 12
  5. 2024Deep Analysis of Time Series Data for Smart Grid Startup Strategies: A Transformer-LSTM-PSO Model Approachcited by 11
  6. 2025A Survey of Link Prediction in Temporal Networkscited by 6
  7. 2025QLLM: Do We Really Need a Mixing Network for Credit Assignment in Multi-Agent Reinforcement Learning?cited by 6
  8. 2023From paintbrush to pixel: A review of deep neural networks in AI-generated artcited by 5
  9. 2023Cross-Domain Car Detection Model with Integrated Convolutional Block Attention Mechanismcited by 5
  10. 2022Attention and Self-Attention in Random Forestscited by 4
  11. 2025BrainNet-MoE: Brain-Inspired Mixture-of-Experts Learning for Neurological Disease Identificationcited by 3
  12. 2025Integrating Biological and Machine Intelligence: Attention Mechanisms in Brain-Computer Interfacescited by 2
17 more
  1. 2023Universal Deoxidation of Semiconductor Substrates Assisted by Machine-Learning and Real-Time-Feedback-Controlcited by 1
  2. 2024Utilizing Neural Transducers for Two-Stage Text-to-Speech via Semantic Token Predictioncited by 1
  3. 2024Explainable Artificial Intelligence Techniques for Irregular Temporal Classification of Multidrug Resistance Acquisition in Intensive Care Unit Patientscited by 1
  4. 2024Reorganizing attention-space geometry with expressive attentioncited by 1
  5. 2025Why Attention Fails: A Taxonomy of Faults in Attention-Based Neural Networkscited by 1
  6. 2026Nonparametric Teaching of Attention Learnerscited by 1
  7. 2022LARF: Two-level Attention-based Random Forests with a Mixture of Contamination Modelscited by 0
  8. 2023The Potential of Wearable Sensors for Assessing Patient Acuity in Intensive Care Unit (ICU)cited by 0
  9. 2024A dynamic vision sensor object recognition model based on trainable event-driven convolution and spiking attention mechanismcited by 0
  10. 2024SegINR: Segment-wise Implicit Neural Representation for Sequence Alignment in Neural Text-to-Speechcited by 0
  11. 2025MMformer with Adaptive Transferable Attention: Advancing Multivariate Time Series Forecasting for Environmental Applicationscited by 0
  12. 2025Ensemble-Based Survival Models with the Self-Attended Beran Estimator Predictionscited by 0
  13. 2025RIS-Enabled Smart Wireless Environments: Fundamentals and Distributed Optimizationcited by 0
  14. 2026YOLOBirDrone: Dataset for Bird vs Drone Detection and Classification and a YOLO based enhanced learning architecturecited by 0
  15. 2026IMSAHLO: Integrating Multi-Scale Attention and Hybrid Loss Optimization Framework for Robust Neuronal Brain Cell Segmentationcited by 0
  16. 2026QAROO: AI-Driven Online Task Offloading for Energy-Efficient and Sustainable MEC Networkscited by 0
  17. 2026Large Models for Battery Prognostics and Health Management: A Review and Future Roadmapcited by 0