# An Introductory Survey on Attention Mechanisms in NLP Problems

arXiv: https://arxiv.org/abs/1811.05544 · HTML: https://mlsanity.com/paper/1811.05544/

## Key facts

- Cited by 32 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 12 Nov 2018.

## Built on by (most cited first, 30 of 32)

- [Transformers in Vision: A Survey](https://mlsanity.com/paper/2101.01169/) (2021)
- [Explainability for Large Language Models: A Survey](https://mlsanity.com/paper/2309.01029/) (2023)
- [A General Survey on Attention Mechanisms in Deep Learning](https://mlsanity.com/paper/2203.14263/) (2022)
- [Generating images from caption and vice versa via CLIP-Guided Generative Latent Space Search](https://mlsanity.com/paper/2102.01645/) (2021)
- [Attention, please! A survey of Neural Attention Models in Deep Learning](https://mlsanity.com/paper/2103.16775/) (2021)
- [Visual Attention Methods in Deep Learning: An In-Depth Survey](https://mlsanity.com/paper/2204.07756/) (2022)
- [Machine Learning for Autonomous Vehicle's Trajectory Prediction: A comprehensive survey, Challenges, and Future Research Directions](https://mlsanity.com/paper/2307.07527/) (2023)
- [Building Blocks for a Complex-Valued Transformer Architecture](https://mlsanity.com/paper/2306.09827/) (2023)
- [Recommender systems based on graph embedding techniques: A comprehensive review](https://mlsanity.com/paper/2109.09587/) (2021)
- [AMD-HookNet for Glacier Front Segmentation](https://mlsanity.com/paper/2302.02744/) (2023)
- [Set-to-Sequence Methods in Machine Learning: a Review](https://mlsanity.com/paper/2103.09656/) (2021)
- [Detecting Political Biases of Named Entities and Hashtags on Twitter](https://mlsanity.com/paper/2209.08110/) (2022)
- [Towards Trustable Language Models: Investigating Information Quality of Large Language Models](https://mlsanity.com/paper/2401.13086/) (2024)
- [Vision Transformers For Weeds and Crops Classification Of High Resolution UAV Images](https://mlsanity.com/paper/2109.02716/) (2021)
- [TSEM: Temporally Weighted Spatiotemporal Explainable Neural Network for Multivariate Time Series](https://mlsanity.com/paper/2205.13012/) (2022)
- [Attention in Diffusion Model: A Survey](https://mlsanity.com/paper/2504.03738/) (2025)
- [Literature review on vulnerability detection using NLP technology](https://mlsanity.com/paper/2104.11230/) (2021)
- [Textual Data Distributions: Kullback Leibler Textual Distributions Contrasts on GPT-2 Generated Texts, with Supervised, Unsupervised Learning on Vaccine & Market Topics & Sentiment](https://mlsanity.com/paper/2107.02025/) (2021)
- [Deep Sequence Modeling: Development and Applications in Asset Pricing](https://mlsanity.com/paper/2108.08999/) (2021)
- [Attention Based Communication and Control for Multi-UAV Path Planning](https://mlsanity.com/paper/2112.12584/) (2021)
- [EGFI: Drug-Drug Interaction Extraction and Generation with Fusion of Enriched Entity and Sentence Information](https://mlsanity.com/paper/2101.09914/) (2021)
- [Automatic Issue Classifier: A Transfer Learning Framework for Classifying Issue Reports](https://mlsanity.com/paper/2202.06149/) (2022)
- [Deep Learning and Computer Vision Techniques for Microcirculation Analysis: A Review](https://mlsanity.com/paper/2205.05493/) (2022)
- [Tri-Attention: Explicit Context-Aware Attention Mechanism for Natural Language Processing](https://mlsanity.com/paper/2211.02899/) (2022)
- [TransCORALNet: A Two-Stream Transformer CORAL Networks for Supply Chain Credit Assessment Cold Start](https://mlsanity.com/paper/2311.18749/) (2023)
- [Cross-Attention is Half Explanation in Speech-to-Text Models](https://mlsanity.com/paper/2509.18010/) (2025)
- [TaleFrame: An Interactive Story Generation System with Fine-Grained Control and Large Language Models](https://mlsanity.com/paper/2512.02402/) (2025)
- [Transformer-Encoder-GRU (T-E-GRU) for Chinese Sentiment Analysis on Chinese Comment Text](https://mlsanity.com/paper/2108.00400/) (2021)
- [Attention-Driven Lightweight Model for Pigmented Skin Lesion Detection](https://mlsanity.com/paper/2308.02119/) (2023)
- [MaintainoMATE: A GitHub App for Intelligent Automation of Maintenance Activities](https://mlsanity.com/paper/2308.16464/) (2023)

## Abstract (arXiv)

First derived from human intuition, later adapted to machine translation for automatic token alignment, attention mechanism, a simple method that can be used for encoding sequence data based on the importance score each element is assigned, has been widely applied to and attained significant improvement in various tasks in natural language processing, including sentiment classification, text summarization, question answering, dependency parsing, etc. In this paper, we survey through recent works and conduct an introductory summary of the attention mechanism in different NLP problems, aiming to provide our readers with basic knowledge on this widely used method, discuss its different variants for different tasks, explore its association with other techniques in machine learning, and examine methods for evaluating its performance.