mlsanityData to 7 Oct 2026

Analyzing the Structure of Attention in a Transformer Language Model

arXiv 1906.04284 ↗First posted 7 Jun 2019cs.CL cs.LG stat.MLcited by 241

Not on the map. Why?

Abstract (arXiv)

The Transformer is a fully attention-based alternative to recurrent networks that has achieved state-of-the-art results across a range of NLP tasks. In this paper, we analyze the structure of attention in a Transformer language model, the GPT-2 small pretrained model. We visualize attention for individual instances and analyze the interaction between attention and syntax over a large corpus. We find that attention targets different parts of speech at different layer depths within the model, and that attention aligns with dependency relations most strongly in the middle layers. We also find that the deepest layers of the model capture the most distant relationships. Finally, we extract exemplar sentences that reveal highly specific patterns targeted by particular attention heads.

Key facts. Cited by 241 papers in the map (data through 7 Oct 2026). First posted on arXiv 7 Jun 2019.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 31; 2022: 29; 2023: 44; 2024: 51; 2025: 52; 2026: 34.

241 papers.

Sort:

By area:

  1. 2021Knowledge Neurons in Pretrained Transformerscited by 590
  2. 2021Rethinking Spatial Dimensions of Vision Transformerscited by 361
  3. 2021FNet: Mixing Tokens with Fourier Transformscited by 343
  4. 2021Thinking Like Transformerscited by 182
  5. 2022A Review on Language Models as Knowledge Basescited by 156
  6. 2025Why do LLMs attend to the first token?cited by 122
  7. 2024Model Editing Harms General Abilities of Large Language Models: Regularization to the Rescuecited by 101
  8. 2021Pose Recognition with Cascade Transformerscited by 99
  9. 2023Language Model Behavior: A Comprehensive Surveycited by 92
  10. 2023How Do Transformers Learn Topic Structure: Towards a Mechanistic Understandingcited by 85
  11. 2022Rethinking the Role of Scale for In-Context Learning: An Interpretability-based Case Study at 66 Billion Scalecited by 64
  12. 2024How Large Language Models Encode Context Knowledge? A Layer-Wise Probing Studycited by 64
38 more
  1. 2022Measuring the Mixing of Contextual Information in the Transformercited by 63
  2. 2024Unveiling and Mitigating Memorization in Text-to-image Diffusion Models through Cross Attentioncited by 57
  3. 2024Confidence Regulation Neurons in Language Modelscited by 57
  4. 2022VL-InterpreT: An Interactive Visualization Tool for Interpreting Vision-Language Transformerscited by 53
  5. 2021Attention, please! A survey of Neural Attention Models in Deep Learningcited by 50
  6. 2022What Do They Capture? -- A Structural Analysis of Pre-Trained Language Models for Source Codecited by 50
  7. 2023AttentionViz: A Global View of Transformer Attentioncited by 50
  8. 2023Towards a Mechanistic Interpretation of Multi-Step Reasoning Capabilities of Language Modelscited by 50
  9. 2024A Mechanistic Analysis of a Transformer Trained on a Symbolic Multi-Step Reasoning Taskcited by 49
  10. 2022On the Blind Spots of Model-Based Evaluation Metrics for Text Generationcited by 48
  11. 2022Investigating Explainability of Generative AI for Code through Scenario-based Designcited by 47
  12. 2024DeciMamba: Exploring the Length Extrapolation Potential of Mambacited by 45
  13. 2023Decoder-Only or Encoder-Decoder? Interpreting Language Model as a Regularized Encoder-Decodercited by 44
  14. 2023LasUIE: Unifying Information Extraction with Latent Adaptive Structure-aware Generative Language Modelcited by 43
  15. 2023The Closeness of In-Context Learning and Weight Shifting for Softmax Regressioncited by 41
  16. 2023Look Before You Leap: An Exploratory Study of Uncertainty Measurement for Large Language Modelscited by 40
  17. 2024Loss Landscape Degeneracy and Stagewise Development in Transformerscited by 39
  18. 2024Attention with Markov: A Framework for Principled Analysis of Transformers via Markov Chainscited by 39
  19. 2023Towards autonomous system: flexible modular production system enhanced with large language model agentscited by 38
  20. 2022On the Explainability of Natural Language Processing Deep Modelscited by 37
  21. 2024LazyDiT: Lazy Learning for the Acceleration of Diffusion Transformerscited by 35
  22. 2022Attention as a Guide for Simultaneous Speech Translationcited by 32
  23. 2023VISIT: Visualizing and Interpreting the Semantic Information Flow of Transformerscited by 32
  24. 2023Mitigating Over-smoothing in Transformers via Regularized Nonlocal Functionalscited by 31
  25. 2024On the Challenges and Opportunities in Generative AIcited by 31
  26. 2023Transformers are uninterpretable with myopic methods: a case study with bounded Dyck grammarscited by 30
  27. 2021Supervising Model Attention with Human Explanations for Robust Natural Language Inferencecited by 29
  28. 2021How BPE Affects Memorization in Transformerscited by 29
  29. 2023A Mathematical Abstraction for Balancing the Trade-off Between Creativity and Reality in Large Language Modelscited by 29
  30. 2021Improving Transformers with Probabilistic Attention Keyscited by 28
  31. 2022LM-Debugger: An Interactive Tool for Inspection and Intervention in Transformer-Based Language Modelscited by 28
  32. 2023Dissociating language and thought in large language modelscited by 27
  33. 2025MUDDFormer: Breaking Residual Bottlenecks in Transformers via Multiway Dynamic Dense Connectionscited by 27
  34. 2021Accounting for Agreement Phenomena in Sentence Comprehension with Transformer Language Models: Effects of Similarity-based Interference on Surprisal and Attentioncited by 26
  35. 2021FMMformer: Efficient and Flexible Transformer via Decomposed Near-field and Far-field Attentioncited by 26
  36. 2022AnyMorph: Learning Transferable Polices By Inferring Agent Morphologycited by 25
  37. 2024Language Models Need Inductive Biases to Count Inductivelycited by 25
  38. 2024Pruning for Protection: Increasing Jailbreak Resistance in Aligned LLMs Without Fine-Tuningcited by 24

50 most cited of 241 All 241 (JSON)