mlsanityData to 14 Oct 2026

What Does BERT Look At? An Analysis of BERT's Attention

arXiv 1906.04341 ↗First posted 11 Jun 2019cs.CLcited by 1,003

Not on the map. Why?

Cited by 41 in LLM reasoning, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

Large pre-trained neural networks such as BERT have had great recent success in NLP, motivating a growing body of research investigating what aspects of language they are able to learn from unlabeled data. Most recent analysis has focused on model outputs (e.g., language model surprisal) or internal vector representations (e.g., probing classifiers). Complementary to these works, we propose methods for analyzing the attention mechanisms of pre-trained models and apply them to BERT. BERT's attention heads exhibit patterns such as attending to delimiter tokens, specific positional offsets, or broadly attending over the whole sentence, with heads in the same layer often exhibiting similar behaviors. We further show that certain attention heads correspond well to linguistic notions of syntax and coreference. For example, we find heads that attend to the direct objects of verbs, determiners of nouns, objects of prepositions, and coreferent mentions with remarkably high accuracy. Lastly, we propose an attention-based probing classifier and use it to further demonstrate that substantial syntactic information is captured in BERT's attention.

Key facts. Cited by 1,003 papers in the map (data through 14 Oct 2026). First posted on arXiv 11 Jun 2019.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2021: 163; 2022: 159; 2023: 155; 2024: 168; 2025: 162; 2026: 196.

1,003 papers.

Sort:

By area:

  1. 2021Probing Classifiers: Promises, Shortcomings, and Advancescited by 876
  2. 2021GPT Understands, Toocited by 716
  3. 2021CogView: Mastering Text-to-Image Generation via Transformerscited by 694
  4. 2021Knowledge Neurons in Pretrained Transformerscited by 601
  5. 2022Transformer Feed-Forward Layers Build Predictions by Promoting Concepts in the Vocabulary Spacecited by 532
  6. 2023Model Tells You What to Discard: Adaptive KV Cache Compression for LLMscited by 474
  7. 2021Recent Advances in Natural Language Processing via Large Pre-Trained Language Models: A Surveycited by 463
  8. 2023TinyStories: How Small Can Language Models Be and Still Speak Coherent English?cited by 449
  9. 2021Causal Abstractions of Neural Networkscited by 422
  10. 2021FNet: Mixing Tokens with Fourier Transformscited by 347
  11. 2021Efficient Attentions for Long Document Summarizationcited by 345
  12. 2023Explainability for Large Language Models: A Surveycited by 333
38 more
  1. 2021Pre-Trained Models: Past, Present and Futurecited by 319
  2. 2021MERLOT: Multimodal Neural Script Knowledge Modelscited by 311
  3. 2024Massive Activations in Large Language Modelscited by 294
  4. 2024DuoAttention: Efficient Long-Context LLM Inference with Retrieval and Streaming Headscited by 288
  5. 2022Transformers Learn Shortcuts to Automatacited by 268
  6. 2024Chain of Thought Empowers Transformers to Solve Inherently Serial Problemscited by 260
  7. 2021N"UWA: Visual Synthesis Pre-training for Neural visUal World creAtioncited by 253
  8. 2022What the DAAM: Interpreting Stable Diffusion Using Cross Attentioncited by 236
  9. 2024Ada-KV: Optimizing KV Cache Eviction by Adaptive Budget Allocation for Efficient LLM Inferencecited by 235
  10. 2024Assessing the Brittleness of Safety Alignment via Pruning and Low-Rank Modificationscited by 231
  11. 2023Parameter-Efficient Fine-Tuning Methods for Pretrained Language Models: A Critical Review and Assessmentcited by 230
  12. 2024The Unreasonable Ineffectiveness of the Deeper Layerscited by 226
  13. 2022cosFormer: Rethinking Softmax in Attentioncited by 213
  14. 2023Causal Abstraction: A Theoretical Foundation for Mechanistic Interpretabilitycited by 205
  15. 2023How does GPT-2 compute greater-than?: Interpreting mathematical abilities in a pre-trained language modelcited by 202
  16. 2021AutoFormer: Searching Transformers for Visual Recognitioncited by 188
  17. 2021Thinking Like Transformerscited by 184
  18. 2021Implicit Representations of Meaning in Neural Language Modelscited by 180
  19. 2021Synthesis of Compositional Animations from Textual Descriptionscited by 178
  20. 2024Towards Measuring and Modeling "Culture" in LLMs: A Surveycited by 167
  21. 2021Condenser: a Pre-training Architecture for Dense Retrievalcited by 162
  22. 2022A Review on Language Models as Knowledge Basescited by 157
  23. 2021Post-hoc Interpretability for Neural NLP: A Surveycited by 156
  24. 2023Quantizable Transformers: Removing Outliers by Helping Attention Heads Do Nothingcited by 151
  25. 2021Inductive Biases and Variable Creation in Self-Attention Mechanismscited by 144
  26. 2024Lookback Lens: Detecting and Mitigating Contextual Hallucinations in Large Language Models Using Only Attention Mapscited by 142
  27. 2021Understanding and Overcoming the Challenges of Efficient Transformer Quantizationcited by 141
  28. 2024Towards Understanding Cross and Self-Attention in Stable Diffusion for Text-Guided Image Editingcited by 135
  29. 2023Representational Strengths and Limitations of Transformerscited by 133
  30. 2022Analyzing Transformers in Embedding Spacecited by 132
  31. 2022Towards Faithful Model Explanation in NLP: A Surveycited by 130
  32. 2022Toward Transparent AI: A Survey on Interpreting the Inner Structures of Deep Neural Networkscited by 129
  33. 2023Unlimiformer: Long-Range Transformers with Unlimited Length Inputcited by 128
  34. 2025Why do LLMs attend to the first token?cited by 127
  35. 2021COCO-LM: Correcting and Contrasting Text Sequences for Language Model Pretrainingcited by 126
  36. 2023Sudden Drops in the Loss: Syntax Acquisition, Phase Transitions, and Simplicity Bias in MLMscited by 124
  37. 2024Transformers are Multi-State RNNscited by 120
  38. 2024AttnLRP: Attention-Aware Layer-Wise Relevance Propagation for Transformerscited by 120

50 most cited of 1,003 All 1,003 (JSON)