mlsanityData to 7 Oct 2026

A Primer in BERTology: What we know about how BERT works

arXiv 2002.12327 ↗First posted 27 Feb 2020cs.CLcited by 807

Not on the map. Why?

Cited by 29 in LLM reasoning, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

Transformer-based models have pushed state of the art in many areas of NLP, but our understanding of what is behind their success is still limited. This paper is the first survey of over 150 studies of the popular BERT model. We review the current state of knowledge about how BERT works, what kind of information it learns and how it is represented, common modifications to its training objectives and architecture, the overparameterization issue and approaches to compression. We then outline directions for future research.

Key facts. Cited by 807 papers in the map (data through 7 Oct 2026). First posted on arXiv 27 Feb 2020.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 180; 2022: 153; 2023: 154; 2024: 122; 2025: 102; 2026: 96.

807 papers.

Sort:

By area:

  1. 2021Probing Classifiers: Promises, Shortcomings, and Advancescited by 831
  2. 2023The Linear Representation Hypothesis and the Geometry of Large Language Modelscited by 679
  3. 2021Recent Advances in Natural Language Processing via Large Pre-Trained Language Models: A Surveycited by 457
  4. 2021Learning How to Ask: Querying LMs with Mixtures of Soft Promptscited by 376
  5. 2021Measuring and Improving Consistency in Pretrained Language Modelscited by 361
  6. 2023Explainability for Large Language Models: A Surveycited by 331
  7. 2023Finding Neurons in a Haystack: Case Studies with Sparse Probingcited by 328
  8. 2023PandaLM: An Automatic Evaluation Benchmark for LLM Instruction Tuning Optimizationcited by 311
  9. 2023Language Models Represent Space and Timecited by 306
  10. 2021Factual Probing Is [MASK]: Learning vs. Learning to Recallcited by 291
  11. 2024Knowledge Conflicts for LLMs: A Surveycited by 274
  12. 2023A Survey on Model Compression for Large Language Modelscited by 263
38 more
  1. 2022Self-Supervised Speech Representation Learning: A Reviewcited by 258
  2. 2023Does Localization Inform Editing? Surprising Differences in Causality-Based Localization vs. Knowledge Editing in Language Modelscited by 237
  3. 2022Extracting Latent Steering Vectors from Pretrained Language Modelscited by 217
  4. 2021When Does Pretraining Help? Assessing Self-Supervised Learning for Law and the CaseHOLD Datasetcited by 199
  5. 2023How does GPT-2 compute greater-than?: Interpreting mathematical abilities in a pre-trained language modelcited by 199
  6. 2022CREPE: Can Vision-Language Foundation Models Reason Compositionally?cited by 177
  7. 2021Masked Language Modeling and the Distributional Hypothesis: Order Word Matters Pre-training for Littlecited by 176
  8. 2023Scaling Down to Scale Up: A Guide to Parameter-Efficient Fine-Tuningcited by 171
  9. 2022Quantifying Privacy Risks of Masked Language Models Using Membership Inference Attackscited by 168
  10. 2022A Review on Language Models as Knowledge Basescited by 156
  11. 2021Post-hoc Interpretability for Neural NLP: A Surveycited by 155
  12. 2023Pushing Mixture of Experts to the Limit: Extremely Parameter Efficient MoE for Instruction Tuningcited by 148
  13. 2022Probing Pre-Trained Language Models for Cross-Cultural Differences in Valuescited by 147
  14. 2021Inductive Biases and Variable Creation in Self-Attention Mechanismscited by 144
  15. 2023Representational Strengths and Limitations of Transformerscited by 132
  16. 2022Analyzing Transformers in Embedding Spacecited by 131
  17. 2023DeID-GPT: Zero-shot Medical Text De-Identification by GPT-4cited by 128
  18. 2021Acquisition of Chess Knowledge in AlphaZerocited by 120
  19. 2021BERT Busters: Outlier Dimensions that Disrupt Transformerscited by 110
  20. 2023Are Emergent Abilities in Large Language Models just In-Context Learning?cited by 109
  21. 2023The geometry of hidden representations of large transformer modelscited by 108
  22. 2021Transformer visualization via dictionary learning: contextualized embedding as a linear superposition of transformer factorscited by 106
  23. 2022Unveiling Transformers with LEGO: a synthetic reasoning taskcited by 105
  24. 2023Interpretability at Scale: Identifying Causal Mechanisms in Alpacacited by 102
  25. 2023How do Language Models Bind Entities in Context?cited by 102
  26. 2023Tracr: Compiled Transformers as a Laboratory for Interpretabilitycited by 99
  27. 2022What Artificial Neural Networks Can Tell Us About Human Language Acquisitioncited by 97
  28. 2021WRENCH: A Comprehensive Benchmark for Weak Supervisioncited by 95
  29. 2023Language Model Behavior: A Comprehensive Surveycited by 92
  30. 2021The MultiBERTs: BERT Reproductions for Robustness Analysiscited by 89
  31. 2021Fast, Effective, and Self-Supervised: Transforming Masked Language Models into Universal Lexical and Sentence Encoderscited by 88
  32. 2021Why Do Pretrained Language Models Help in Downstream Tasks? An Analysis of Head and Prompt Tuningcited by 82
  33. 2023Attending to Graph Transformerscited by 81
  34. 2021Do Language Models Have Beliefs? Methods for Detecting, Updating, and Visualizing Model Beliefscited by 79
  35. 2024Grokked Transformers are Implicit Reasoners: A Mechanistic Journey to the Edge of Generalizationcited by 78
  36. 2021Paragraph-level Rationale Extraction through Regularization: A case study on European Court of Human Rights Casescited by 77
  37. 2022Neighborhood Contrastive Learning for Scientific Document Representations with Citation Embeddingscited by 75
  38. 2022Iteratively Prompt Pre-trained Language Models for Chain of Thoughtcited by 75

50 most cited of 807 All 807 (JSON)