mlsanityData to 7 Oct 2026

Attention Heads of Large Language Models: A Survey

arXiv 2409.03752 ↗First posted 5 Sep 2024cs.CLcited by 65

Not on the map. Why?

Abstract (arXiv)

Since the advent of ChatGPT, Large Language Models (LLMs) have excelled in various tasks but remain as black-box systems. Understanding the reasoning bottlenecks of LLMs has become a critical challenge, as these limitations are deeply tied to their internal architecture. Among these, attention heads have emerged as a focal point for investigating the underlying mechanics of LLMs. In this survey, we aim to demystify the internal reasoning processes of LLMs by systematically exploring the roles and mechanisms of attention heads. We first introduce a novel four-stage framework inspired by the human thought process: Knowledge Recalling, In-Context Identification, Latent Reasoning, and Expression Preparation. Using this framework, we comprehensively review existing research to identify and categorize the functions of specific attention heads. Additionally, we analyze the experimental methodologies used to discover these special heads, dividing them into two categories: Modeling-Free and Modeling-Required methods. We further summarize relevant evaluation methods and benchmarks. Finally, we discuss the limitations of current research and propose several potential future directions.

Key facts. Cited by 65 papers in the map (data through 7 Oct 2026). Builds on 99 of its 149 references that are in the map. First posted on arXiv 5 Sep 2024.

Builds on

99 of 149 references.

References · 99

  1. 2017Attention Is All You Needcited by 59,932
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  3. 2022Chain-of-Thought Prompting Elicits Reasoning in Large Language Modelscited by 12,302
  4. 2021Training Verifiers to Solve Math Word Problemscited by 9,743
  5. 2020Measuring Massive Multitask Language Understandingcited by 7,768
  6. 2020Scaling Laws for Neural Language Modelscited by 6,411
  7. 2021Measuring Mathematical Problem Solving With the MATH Datasetcited by 5,203
  8. 2021RoFormer: Enhanced Transformer with Rotary Position Embeddingcited by 4,581
  9. 2021On the Opportunities and Risks of Foundation Modelscited by 3,961
  10. 2021Switch Transformers: Scaling to Trillion Parameter Models with Simple and Efficient Sparsitycited by 3,014
  11. 2021TruthfulQA: Measuring How Models Mimic Human Falsehoodscited by 2,919
  12. 2017TriviaQA: A Large Scale Distantly Supervised Challenge Dataset for Reading Comprehensioncited by 2,895
38 more
  1. 2015Character-level Convolutional Networks for Text Classificationcited by 2,293
  2. 2020GLU Variants Improve Transformercited by 1,724
  3. 2018Don't Give Me the Details, Just the Summary! Topic-Aware Convolutional Neural Networks for Extreme Summarizationcited by 1,308
  4. 2020Transformer Feed-Forward Layers Are Key-Value Memoriescited by 1,307
  5. 2023Inference-Time Intervention: Eliciting Truthful Answers from a Language Modelcited by 1,259
  6. 2022Interpretability in the Wild: a Circuit for Indirect Object Identification in GPT-2 smallcited by 1,130
  7. 2022In-context Learning and Induction Headscited by 930
  8. 2020On Layer Normalization in the Transformer Architecturecited by 906
  9. 2019Analyzing Multi-Head Self-Attention: Specialized Heads Do the Heavy Lifting, the Rest Can Be Prunedcited by 844
  10. 2023Towards Automated Circuit Discovery for Mechanistic Interpretabilitycited by 700
  11. 2023The Linear Representation Hypothesis and the Geometry of Large Language Modelscited by 679
  12. 2021Knowledge Neurons in Pretrained Transformerscited by 590
  13. 2023Dissecting Recall of Factual Associations in Auto-Regressive Language Modelscited by 512
  14. 2018Explaining Explanations: An Overview of Interpretability of Machine Learningcited by 508
  15. 2021Causal Abstractions of Neural Networkscited by 401
  16. 2017Methods for Interpreting and Understanding Deep Neural Networkscited by 342
  17. 2023Towards Best Practices of Activation Patching in Language Models: Metrics and Methodscited by 328
  18. 2019Revealing the Dark Secrets of BERTcited by 300
  19. 2023MQuAKE: Assessing Knowledge Editing in Language Models via Multi-Hop Questionscited by 292
  20. 2023Function Vectors in Large Language Modelscited by 292
  21. 2024A Survey on Mixture of Experts in Large Language Modelscited by 233
  22. 2024Retrieval Head Mechanistically Explains Long-Context Factualitycited by 219
  23. 2020Retrieve, Program, Repeat: Complex Knowledge Base Question Answering via Alternate Meta-learningcited by 216
  24. 2020Understanding the Difficulty of Training Transformerscited by 210
  25. 2023How does GPT-2 compute greater-than?: Interpreting mathematical abilities in a pre-trained language modelcited by 199
  26. 2023Finding Alignments Between Interpretable Causal Variables and Distributed Neural Representationscited by 197
  27. 2024How to use and interpret activation patchingcited by 191
  28. 2023Linearity of Relation Decoding in Transformer Language Modelscited by 190
  29. 2019BERT has a Mouth, and It Must Speak: BERT as a Markov Random Field Language Modelcited by 184
  30. 2022Language models show human-like content effects on reasoning taskscited by 171
  31. 2023Linear Representations of Sentiment in Large Language Modelscited by 157
  32. 2019Adaptively Sparse Transformerscited by 154
  33. 2023Birth of a Transformer: A Memory Viewpointcited by 154
  34. 2023What In-Context Learning "Learns" In-Context: Disentangling Task Recognition and Task Learningcited by 149
  35. 2023Does Circuit Analysis Interpretability Scale? Evidence from Multiple Choice Capabilities in Chinchillacited by 145
  36. 2024Lookback Lens: Detecting and Mitigating Contextual Hallucinations in Large Language Models Using Only Attention Mapscited by 140
  37. 2022Toward Transparent AI: A Survey on Interpreting the Inner Structures of Deep Neural Networkscited by 126
  38. 2019An Attentive Survey of Attention Modelscited by 123

50 most cited of 99 All 99 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 7; 2025: 36; 2026: 22.

65 papers.

Sort:

By area:

  1. 2025See What You Are Told: Visual Attention Sink in Large Multimodal Modelscited by 127
  2. 2024Not All Heads Matter: A Head-Level KV Cache Compression Method with Integrated Retrieval and Reasoningcited by 96
  3. 2025Your Large Vision-Language Model Only Needs A Few Attention Heads For Visual Groundingcited by 78
  4. 2024On the Role of Attention Heads in Large Language Model Safetycited by 70
  5. 2025Beyond Semantics: Rediscovering Spatial Awareness in Vision-Language Modelscited by 35
  6. 2024Unveiling Visual Perception in Language Models: An Attention Head Analysis Approachcited by 34
  7. 2025Attention Illuminates LLM Reasoning: The Preplan-and-Anchor Rhythm Enables Fine-Grained Policy Optimizationcited by 23
  8. 2025The Reasoning-Memorization Interplay in Language Models Is Mediated by a Single Directioncited by 21
  9. 2024Revealing the Barriers of Language Agents in Planningcited by 20
  10. 2025Spotlight Your Instructions: Instruction-following with Dynamic Attention Steeringcited by 18
  11. 2025Causal Head Gating: A Framework for Interpreting Roles of Attention Heads in Transformerscited by 16
  12. 2025Understanding the Skill Gap in Recurrent Language Models: The Role of the Gather-and-Aggregate Mechanismcited by 15
38 more
  1. 2025Object Detection with Multimodal Large Vision-Language Models: An In-depth Reviewcited by 14
  2. 2025Learning to Attribute with Attentioncited by 11
  3. 2025ZeroTuning: Unlocking the Initial Token's Power to Enhance Large Language Models Without Trainingcited by 11
  4. 2025MindGuard: Intrinsic Decision Inspection for Securing LLM Agents Against Metadata Poisoningcited by 10
  5. 2025Dynamic Attention-Guided Context Decoding for Mitigating Context Faithfulness Hallucinations in Large Language Modelscited by 9
  6. 2025Repetitions are not all alike: distinct mechanisms sustain repetition in language modelscited by 9
  7. 2025ProxyAttn: Guided Sparse Attention via Representative Headscited by 9
  8. 2025All You Need is One: Capsule Prompt Tuning with a Single Vectorcited by 8
  9. 2025GRAPHMOE: Amplifying Cognitive Depth of Mixture-of-Experts Network via Introducing Self-Rethinking Mechanismcited by 7
  10. 2025Interpreting Attention Heads for Image-to-Text Information Flow in Large Vision-Language Modelscited by 7
  11. 2026Causality neq Invariance: Function and Concept Vectors in LLMscited by 7
  12. 2025Unveiling Simplicities of Attention: Adaptive Long-Context Head Identificationcited by 5
  13. 2025AttentionInfluence: Adopting Attention Head Influence for Weak-to-Strong Pretraining Data Selectioncited by 5
  14. 2025Cognitive Mirrors: Exploring the Diverse Functional Roles of Attention Heads in LLM Reasoningcited by 5
  15. 2026Structural Feature Engineering for Generative Engine Optimization: How Content Structure Shapes Citation Behaviorcited by 5
  16. 2024Goal Inference from Open-Ended Dialogcited by 4
  17. 2025ADEPT: Continual Pretraining via Adaptive Expansion and Dynamic Decoupled Tuningcited by 4
  18. 2026AttenMIA: LLM Membership Inference Attack through Attention Signalscited by 4
  19. 2026Steering at the Source: Style Modulation Heads for Robust Persona Controlcited by 4
  20. 2026SafeSeek: Universal Attribution of Safety Circuits in Language Modelscited by 4
  21. 2025Compress, Gather, and Recompute: REFORMing Long-Context Processing in Transformerscited by 3
  22. 2025The Compositional Architecture of Regret in Large Language Modelscited by 3
  23. 2025Task Vectors, Learned Not Extracted: Performance Gains and Mechanistic Insightcited by 3
  24. 2025ssToken: Self-modulated and Semantic-aware Token Selection for LLM Fine-tuningcited by 3
  25. 2026Stop Tracking Me! Proactive Defense Against Attribute Inference Attack in LLMscited by 3
  26. 2026Your Vision-Language-Action Model Already Has Attention Heads For Path Deviation Detectioncited by 3
  27. 2026Attention in Space: Functional Roles of VLM Heads for Spatial Reasoningcited by 3
  28. 2026Improving Reasoning Capabilities in Small Models through Mixture-of-Layers Distillation with Stepwise Attention on Key Informationcited by 3
  29. 2026How Does Reasoning Flow? Tracing Attention-Induced Information Flow for Targeted RL in LLMscited by 3
  30. 2025Word Meanings in Transformer Language Modelscited by 2
  31. 2025DFAMS: Dynamic-flow guided Federated Alignment based Multi-prototype Searchcited by 2
  32. 2025Stream: Scaling up Mechanistic Interpretability to Long Context in LLMs via Sparse Attentioncited by 2
  33. 2025Investigating The Functional Roles of Attention Heads in Vision Language Models: Evidence for Reasoning Modulescited by 2
  34. 2026Compressed Sensing for Capability Localization in Large Language Modelscited by 2
  35. 2024Listening to the Wise Few: Query-Key Alignment Unlocks Latent Correct Answers in Large Language Modelscited by 1
  36. 2026Interpretable Motion-Attentive Maps: Spatio-Temporally Localizing Concepts in Video Diffusion Transformerscited by 1
  37. 2026Disentangling MLP Neuron Weights in Vocabulary Spacecited by 1
  38. 2026The Truth Stays in the Family: Enhancing Contextual Grounding via Inherited Truthful Heads in Model Lineagescited by 1

50 most cited of 65 All 65 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2022Towards Reasoning in Large Language Models: A Survey
  2. 2024Beyond Accuracy: Evaluating the Reasoning Behavior of Large Language Models -- A Survey
  3. 2025Advancing Reasoning in Large Language Models: Promising Methods and Approaches
  4. 2025Who Reasons in the Large Language Models?
  5. 2026Understanding Large Language Models
  6. 2025Attention-Aligned Reasoning for Large Language Models
  7. 2025Beyond Chains of Thought: Benchmarking Latent-Space Reasoning Abilities in Large Language Models
  8. 2025Reasoning Strategies in Large Language Models: Can They Follow, Prefer, and Optimize?