mlsanityData to 14 Oct 2026

Revealing the Dark Secrets of BERT

arXiv 1908.08593 ↗First posted 21 Aug 2019cs.CL cs.LG stat.MLcited by 301

Not on the map. Why?

Abstract (arXiv)

BERT-based architectures currently give state-of-the-art performance on many NLP tasks, but little is known about the exact mechanisms that contribute to its success. In the current work, we focus on the interpretation of self-attention, which is one of the fundamental underlying components of BERT. Using a subset of GLUE tasks and a set of handcrafted features-of-interest, we propose the methodology and carry out a qualitative and quantitative analysis of the information encoded by the individual BERT's heads. Our findings suggest that there is a limited set of attention patterns that are repeated across different heads, indicating the overall model overparametrization. While different heads consistently use the same attention patterns, they have varying impact on performance across different tasks. We show that manually disabling attention in certain heads leads to a performance improvement over the regular fine-tuned BERT models.

Key facts. Cited by 301 papers in the map (data through 14 Oct 2026). First posted on arXiv 21 Aug 2019.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2021: 76; 2022: 59; 2023: 52; 2024: 54; 2025: 34; 2026: 26.

301 papers.

Sort:

By area:

  1. 2021Knowledge Neurons in Pretrained Transformerscited by 601
  2. 2023Model Tells You What to Discard: Adaptive KV Cache Compression for LLMscited by 474
  3. 2021Recent Advances in Natural Language Processing via Large Pre-Trained Language Models: A Surveycited by 463
  4. 2021A Survey of Transformerscited by 401
  5. 2023Explainability for Large Language Models: A Surveycited by 333
  6. 2021Pre-Trained Models: Past, Present and Futurecited by 319
  7. 2024Massive Activations in Large Language Modelscited by 294
  8. 2022What the DAAM: Interpreting Stable Diffusion Using Cross Attentioncited by 236
  9. 2023Parameter-Efficient Fine-Tuning Methods for Pretrained Language Models: A Critical Review and Assessmentcited by 230
  10. 2022cosFormer: Rethinking Softmax in Attentioncited by 213
  11. 2023Landmark Attention: Random-Access Infinite Context Length for Transformerscited by 213
  12. 2021Implicit Representations of Meaning in Neural Language Modelscited by 180
38 more
  1. 2023PMET: Precise Model Editing in a Transformercited by 173
  2. 2021MoEfication: Transformer Feed-forward Layers are Mixtures of Expertscited by 152
  3. 2023Quantizable Transformers: Removing Outliers by Helping Attention Heads Do Nothingcited by 151
  4. 2024Fine-Tuning Enhances Existing Mechanisms: A Case Study on Entity Trackingcited by 149
  5. 2021AMMUS : A Survey of Transformer-based Pretrained Models in Natural Language Processingcited by 131
  6. 2022Revealing the Dark Secrets of Masked Image Modelingcited by 117
  7. 2021BERT Busters: Outlier Dimensions that Disrupt Transformerscited by 112
  8. 2023Language Model Behavior: A Comprehensive Surveycited by 92
  9. 2022DataPerf: Benchmarks for Data-Centric AI Developmentcited by 90
  10. 2021Fast, Effective, and Self-Supervised: Transforming Masked Language Models into Universal Lexical and Sentence Encoderscited by 88
  11. 2022MiniViT: Compressing Vision Transformers with Weight Multiplexingcited by 88
  12. 2023Neurons in Large Language Models: Dead, N-gram, Positionalcited by 88
  13. 2023How Do Transformers Learn Topic Structure: Towards a Mechanistic Understandingcited by 86
  14. 2022Efficient Methods for Natural Language Processing: A Surveycited by 78
  15. 2021Paragraph-level Rationale Extraction through Regularization: A case study on European Court of Human Rights Casescited by 77
  16. 2021Identifying and Mitigating Spurious Correlations for Improving Robustness in NLP Modelscited by 76
  17. 2021Probing Across Time: What Does RoBERTa Know and When?cited by 73
  18. 2022Revisiting Over-smoothing in BERT from the Perspective of Graphcited by 72
  19. 2024Attention Heads of Large Language Models: A Surveycited by 67
  20. 2022Measuring the Mixing of Contextual Information in the Transformercited by 65
  21. 2022Probing for the Usage of Grammatical Numbercited by 59
  22. 2022Discovering Latent Concepts Learned in BERTcited by 56
  23. 2022Entropy-based Attention Regularization Frees Unintended Bias Mitigation from Listscited by 55
  24. 2022Outliers Dimensions that Disrupt Transformers Are Driven by Frequencycited by 55
  25. 2021PARP: Prune, Adjust and Re-Prune for Self-Supervised Speech Recognitioncited by 53
  26. 2024Explainable and Interpretable Multimodal Large Language Models: A Comprehensive Surveycited by 52
  27. 2021Incorporating Residual and Normalization Layers into Analysis of Masked Language Modelscited by 51
  28. 2022What Do They Capture? -- A Structural Analysis of Pre-Trained Language Models for Source Codecited by 50
  29. 2023AttentionViz: A Global View of Transformer Attentioncited by 50
  30. 2021Red Alarm for Pre-trained Models: Universal Vulnerability to Neuron-Level Backdoor Attackscited by 49
  31. 2021SparseBERT: Rethinking the Importance Analysis in Self-attentioncited by 46
  32. 2025Parameter-Efficient Fine-Tuning for Foundation Modelscited by 45
  33. 2022GlobEnc: Quantifying Global Token Attribution by Incorporating the Whole Encoder Layer in Transformerscited by 44
  34. 2024SKVQ: Sliding-window Key and Value Cache Quantization for Large Language Modelscited by 44
  35. 2023Quantifying Context Mixing in Transformerscited by 43
  36. 2023LasUIE: Unifying Information Extraction with Latent Adaptive Structure-aware Generative Language Modelcited by 43
  37. 2021A Closer Look at How Fine-tuning Changes BERTcited by 42
  38. 2021Deep Subjecthood: Higher-Order Grammatical Features in Multilingual BERTcited by 41

50 most cited of 301 All 301 (JSON)