mlsanityData to 7 Oct 2026

Attention in Natural Language Processing

arXiv 1902.02181 ↗First posted 4 Feb 2019cs.CL cs.AI cs.LGcited by 100

Not on the map. Why?

Abstract (arXiv)

Attention is an increasingly popular mechanism used in a wide range of neural architectures. The mechanism itself has been realized in a variety of formats. However, because of the fast-paced advances in this domain, a systematic overview of attention is still missing. In this article, we define a unified model for attention architectures in natural language processing, with a focus on those designed to work with vector representations of the textual data. We propose a taxonomy of attention models according to four dimensions: the representation of the input, the compatibility function, the distribution function, and the multiplicity of the input and/or output. We present the examples of how prior information can be exploited in attention models and discuss ongoing research efforts and open challenges in the area, providing the first extensive categorization of the vast body of literature in this exciting domain.

Key facts. Cited by 100 papers in the map (data through 7 Oct 2026). First posted on arXiv 4 Feb 2019.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 26; 2022: 26; 2023: 17; 2024: 16; 2025: 11; 2026: 4.

100 papers.

Sort:

By area:

  1. 2022Transformers in Time Series: A Surveycited by 489
  2. 2021A Survey of Visual Transformerscited by 117
  3. 2021A Practical Survey on Faster and Lighter Transformerscited by 73
  4. 2021Attention, please! A survey of Neural Attention Models in Deep Learningcited by 50
  5. 2023Change Detection Methods for Remote Sensing in the Last Decade: A Comprehensive Reviewcited by 41
  6. 2023Look Before You Leap: An Exploratory Study of Uncertainty Measurement for Large Language Modelscited by 40
  7. 2021Order in the Court: Explainable AI Methods Prone to Disagreementcited by 36
  8. 2023Interpreting Pretrained Language Models via Concept Bottleneckscited by 33
  9. 2022Behavioral Intention Prediction in Driving Scenes: A Surveycited by 30
  10. 2022Attention Mechanism in Neural Networks: Where it Comes and Where it Goescited by 29
  11. 2021Attentive Neural Controlled Differential Equations for Time-series Classification and Forecastingcited by 28
  12. 2021Exploring Transformers in Natural Language Generation: GPT, BERT, and XLNetcited by 27
38 more
  1. 2021Accounting for Agreement Phenomena in Sentence Comprehension with Transformer Language Models: Effects of Similarity-based Interference on Surprisal and Attentioncited by 26
  2. 2022Interpretable by Design: Learning Predictors by Composing Interpretable Queriescited by 26
  3. 2023A Machine Learning-oriented Survey on Tiny Machine Learningcited by 26
  4. 2022SEAT: Stable and Explainable Attentioncited by 25
  5. 2021Counterfactual Explanations for Models of Codecited by 24
  6. 2021Global-Local Attention for Emotion Recognitioncited by 20
  7. 2022Survey on Automated Short Answer Grading with Deep Learning: from Word Embeddings to Transformerscited by 20
  8. 2022Challenges in Applying Explainability Methods to Improve the Fairness of NLP Modelscited by 20
  9. 2023Multimodal Data Integration for Oncology in the Era of Deep Neural Networks: A Reviewcited by 20
  10. 2024Do Machines and Humans Focus on Similar Code? Exploring Explainability of Large Language Models in Code Summarizationcited by 16
  11. 2024Selective Prompt Anchoring for Code Generationcited by 16
  12. 2021ACE-NODE: Attentive Co-Evolving Neural Ordinary Differential Equationscited by 15
  13. 2022DPA-1: Pretraining of Attention-based Deep Potential Model for Molecular Simulationcited by 15
  14. 2022AOE-Net: Entities Interactions Modeling with Adaptive Attention Mechanism for Temporal Action Proposals Generationcited by 14
  15. 2023GBG++: A Fast and Stable Granular Ball Generation Method for Classificationcited by 13
  16. 2023Do Large Language Models Pay Similar Attention Like Human Programmers When Generating Code?cited by 13
  17. 2021Misinformation Detection on YouTube Using Video Captionscited by 12
  18. 2023Sparsity-Guided Holistic Explanation for LLMs with Interpretable Inference-Time Interventioncited by 12
  19. 2021AEI: Actors-Environment Interaction with Adaptive Attention for Temporal Action Proposals Generationcited by 11
  20. 2022Graph Attention Transformer Network for Multi-Label Image Classificationcited by 11
  21. 2024Attention Meets Post-hoc Interpretability: A Mathematical Perspectivecited by 11
  22. 2021Word-level Human Interpretable Scoring Mechanism for Novel Text Detection Using Tsetlin Machinescited by 10
  23. 2022Cross-Attention Transformer for Video Interpolationcited by 10
  24. 2023SketchXAI: A First Look at Explainability for Human Sketchescited by 10
  25. 2021Neural Attention Models in Deep Learning: Survey and Taxonomycited by 8
  26. 2023Enabling Resource-efficient AIoT System with Cross-level Optimization: A surveycited by 8
  27. 2023Heavy-Tailed Regularization of Weight Matrices in Deep Neural Networkscited by 7
  28. 2021Attention Models for Point Clouds in Deep Learning: A Surveycited by 6
  29. 2021Multi-Task Attentive Residual Networks for Argument Miningcited by 6
  30. 2021Multimodal Continuous Visual Attention Mechanismscited by 6
  31. 2023AMD-HookNet for Glacier Front Segmentationcited by 6
  32. 2025Rethinking Explainability in the Era of Multimodal AIcited by 6
  33. 2021Thank you for Attention: A survey on Attention-based Artificial Neural Networks for Automatic Speech Recognitioncited by 5
  34. 2022Natural Language Inference with Self-Attention for Veracity Assessment of Pandemic Claimscited by 5
  35. 2022A Survey of Text Representation Methods and Their Genealogycited by 5
  36. 2024Symmetric Dot-Product Attention for Efficient Training of BERT Language Modelscited by 5
  37. 2021An ensemble deep learning technique for detecting suicidal ideation from posts in social media platformscited by 4
  38. 2022Multi-branch Cascaded Swin Transformers with Attention to k-space Sampling Pattern for Accelerated MRI Reconstructioncited by 4

50 most cited of 100 All 100 (JSON)