mlsanityData to 14 Oct 2026

An Introductory Survey on Attention Mechanisms in NLP Problems

arXiv 1811.05544 ↗First posted 12 Nov 2018cs.CL cs.LG stat.MLcited by 32

Not on the map. Why?

Abstract (arXiv)

First derived from human intuition, later adapted to machine translation for automatic token alignment, attention mechanism, a simple method that can be used for encoding sequence data based on the importance score each element is assigned, has been widely applied to and attained significant improvement in various tasks in natural language processing, including sentiment classification, text summarization, question answering, dependency parsing, etc. In this paper, we survey through recent works and conduct an introductory summary of the attention mechanism in different NLP problems, aiming to provide our readers with basic knowledge on this widely used method, discuss its different variants for different tasks, explore its association with other techniques in machine learning, and examine methods for evaluating its performance.

Key facts. Cited by 32 papers in the map (data through 14 Oct 2026). First posted on arXiv 12 Nov 2018.

Builds on

No reference list yet.

Built on by

Cited by year

Citing papers per year: 2021: 12; 2022: 7; 2023: 7; 2024: 3; 2025: 3.

32 papers.

Sort:

By area:

  1. 2021Transformers in Vision: A Surveycited by 1,013
  2. 2023Explainability for Large Language Models: A Surveycited by 333
  3. 2022A General Survey on Attention Mechanisms in Deep Learningcited by 64
  4. 2021Generating images from caption and vice versa via CLIP-Guided Generative Latent Space Searchcited by 57
  5. 2021Attention, please! A survey of Neural Attention Models in Deep Learningcited by 51
  6. 2022Visual Attention Methods in Deep Learning: An In-Depth Surveycited by 37
  7. 2023Machine Learning for Autonomous Vehicle's Trajectory Prediction: A comprehensive survey, Challenges, and Future Research Directionscited by 23
  8. 2023Building Blocks for a Complex-Valued Transformer Architecturecited by 16
  9. 2021Recommender systems based on graph embedding techniques: A comprehensive reviewcited by 8
  10. 2023AMD-HookNet for Glacier Front Segmentationcited by 7
  11. 2021Set-to-Sequence Methods in Machine Learning: a Reviewcited by 6
  12. 2022Detecting Political Biases of Named Entities and Hashtags on Twittercited by 6
20 more
  1. 2024Towards Trustable Language Models: Investigating Information Quality of Large Language Modelscited by 6
  2. 2021Vision Transformers For Weeds and Crops Classification Of High Resolution UAV Imagescited by 5
  3. 2022TSEM: Temporally Weighted Spatiotemporal Explainable Neural Network for Multivariate Time Seriescited by 4
  4. 2025Attention in Diffusion Model: A Surveycited by 4
  5. 2021Literature review on vulnerability detection using NLP technologycited by 2
  6. 2021Textual Data Distributions: Kullback Leibler Textual Distributions Contrasts on GPT-2 Generated Texts, with Supervised, Unsupervised Learning on Vaccine & Market Topics & Sentimentcited by 2
  7. 2021Deep Sequence Modeling: Development and Applications in Asset Pricingcited by 2
  8. 2021Attention Based Communication and Control for Multi-UAV Path Planningcited by 2
  9. 2021EGFI: Drug-Drug Interaction Extraction and Generation with Fusion of Enriched Entity and Sentence Informationcited by 1
  10. 2022Automatic Issue Classifier: A Transfer Learning Framework for Classifying Issue Reportscited by 1
  11. 2022Deep Learning and Computer Vision Techniques for Microcirculation Analysis: A Reviewcited by 1
  12. 2022Tri-Attention: Explicit Context-Aware Attention Mechanism for Natural Language Processingcited by 1
  13. 2023TransCORALNet: A Two-Stream Transformer CORAL Networks for Supply Chain Credit Assessment Cold Startcited by 1
  14. 2025Cross-Attention is Half Explanation in Speech-to-Text Modelscited by 1
  15. 2025TaleFrame: An Interactive Story Generation System with Fine-Grained Control and Large Language Modelscited by 1
  16. 2021Transformer-Encoder-GRU (T-E-GRU) for Chinese Sentiment Analysis on Chinese Comment Textcited by 0
  17. 2023Attention-Driven Lightweight Model for Pigmented Skin Lesion Detectioncited by 0
  18. 2023MaintainoMATE: A GitHub App for Intelligent Automation of Maintenance Activitiescited by 0
  19. 2024Shayona@SMM4H23: COVID-19 Self diagnosis classification using BERT and LightGBM modelscited by 0
  20. 2024Characterized Diffusion Networks for Enhanced Autonomous Driving Trajectory Predictioncited by 0