mlsanityData to 7 Oct 2026

An Interpretability Illusion for BERT

arXiv 2104.07143 ↗First posted 14 Apr 2021cs.CL cs.LGcited by 94

Not on the map. Why?

Abstract (arXiv)

We describe an "interpretability illusion" that arises when analyzing the BERT model. Activations of individual neurons in the network may spuriously appear to encode a single, simple concept, when in fact they are encoding something far more complex. The same effect holds for linear combinations of activations. We trace the source of this illusion to geometric properties of BERT's embedding space as well as the fact that common text corpora represent only narrow slices of possible English sentences. We provide a taxonomy of model-learned concepts and discuss methodological implications for interpretability research, especially the importance of testing hypotheses on multiple data sets.

Key facts. Cited by 94 papers in the map (data through 7 Oct 2026). Builds on 21 of its 29 references that are in the map. First posted on arXiv 14 Apr 2021.

Builds on

21 of 29 references.

References · 21

  1. 2019Sentence-BERT: Sentence Embeddings using Siamese BERT-Networkscited by 9,049
  2. 2013Intriguing properties of neural networkscited by 5,114
  3. 2018GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understandingcited by 4,854
  4. 2013Efficient Estimation of Word Representations in Vector Spacecited by 4,346
  5. 2013Visualizing and Understanding Convolutional Networkscited by 2,278
  6. 2017Interpretability Beyond Feature Attribution: Quantitative Testing with Concept Activation Vectors (TCAV)cited by 1,309
  7. 2016Man is to Computer Programmer as Woman is to Homemaker? Debiasing Word Embeddingscited by 1,304
  8. 2015Aligning Books and Movies: Towards Story-like Visual Explanations by Watching Movies and Reading Bookscited by 1,041
  9. 2017Network Dissection: Quantifying Interpretability of Deep Visual Representationscited by 703
  10. 2019What do you learn from context? Probing for sentence structure in contextualized word representationscited by 436
  11. 2015Visualizing and Understanding Neural Models in NLPcited by 199
  12. 2020Causal Mediation Analysis for Interpreting Neural NLP: The Case of Gender Biascited by 183
9 more
  1. 2016Synthesizing the preferred inputs for neurons in neural networks via deep generator networkscited by 181
  2. 2014Object Detectors Emerge in Deep Scene CNNscited by 178
  3. 2020Unsupervised Domain Clusters in Pretrained Language Modelscited by 160
  4. 2019Black is to Criminal as Caucasian is to Police: Detecting and Removing Multiclass Bias in Word Embeddingscited by 146
  5. 2018What Is One Grain of Sand in the Desert? Analyzing Individual Neurons in Deep NLP Modelscited by 124
  6. 2020Analyzing Individual Neurons in Pre-trained Language Modelscited by 97
  7. 2019Gender-preserving Debiasing for Pre-trained Word Embeddingscited by 61
  8. 2019Discovery of Natural Language Concepts in Individual Units of CNNscited by 21
  9. 2018Interpretable Textual Neuron Representations for NLPcited by 16

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 1; 2022: 8; 2023: 23; 2024: 30; 2025: 19; 2026: 13.

94 papers.

Sort:

By area:

  1. 2022Interpretability in the Wild: a Circuit for Indirect Object Identification in GPT-2 smallcited by 1,130
  2. 2023Towards Automated Circuit Discovery for Mechanistic Interpretabilitycited by 700
  3. 2024Scaling and evaluating sparse autoencoderscited by 530
  4. 2024Mechanistic Interpretability for AI Safety -- A Reviewcited by 415
  5. 2023Finding Neurons in a Haystack: Case Studies with Sparse Probingcited by 328
  6. 2023Does Localization Inform Editing? Surprising Differences in Causality-Based Localization vs. Knowledge Editing in Language Modelscited by 237
  7. 2023Causal Abstraction: A Theoretical Foundation for Mechanistic Interpretabilitycited by 194
  8. 2024Transcoders Find Interpretable LLM Feature Circuitscited by 186
  9. 2023Localizing Model Behavior with Path Patchingcited by 182
  10. 2024Improving Dictionary Learning with Gated Sparse Autoencoderscited by 171
  11. 2023Getting aligned on representational alignmentcited by 142
  12. 2022Natural Language Descriptions of Deep Visual Featurescited by 136
38 more
  1. 2022Toward Transparent AI: A Survey on Interpreting the Inner Structures of Deep Neural Networkscited by 126
  2. 2023AI Transparency in the Age of LLMs: A Human-Centered Research Roadmapcited by 119
  3. 2023Mechanistically analyzing the effects of fine-tuning on procedurally defined taskscited by 110
  4. 2025Are Sparse Autoencoders Useful? A Case Study in Sparse Probingcited by 109
  5. 2024In-Context Language Learning: Architectures and Algorithmscited by 86
  6. 2024A Primer on the Inner Workings of Transformer-based Language Modelscited by 83
  7. 2024Towards Principled Evaluations of Sparse Autoencoders for Interpretability and Controlcited by 83
  8. 2024RAVEL: Evaluating Interpretability Methods on Disentangling Language Model Representationscited by 77
  9. 2023Is This the Subspace You Are Looking for? An Interpretability Illusion for Subspace Activation Patchingcited by 75
  10. 2023Exposing Attention Glitches with Flip-Flop Language Modelingcited by 74
  11. 2023Eight Things to Know about Large Language Modelscited by 73
  12. 2023Copy Suppression: Comprehensively Understanding an Attention Headcited by 73
  13. 2025Large Language Models Share Representations of Latent Grammatical Concepts Across Typologically Diverse Languagescited by 57
  14. 2022Overlooked factors in concept-based explanations: Dataset choice, concept learnability, and human capabilitycited by 46
  15. 2024Interpreting and Improving Large Language Models in Arithmetic Calculationcited by 43
  16. 2023Learning Transformer Programscited by 40
  17. 2025Weight-sparse transformers have interpretable circuitscited by 40
  18. 2025Enhancing Automated Interpretability with Output-Centric Feature Descriptionscited by 39
  19. 2024The Missing Curve Detectors of InceptionV1: Applying Sparse Autoencoders to InceptionV1 Early Visioncited by 37
  20. 2024Evaluating Open-Source Sparse Autoencoders on Disentangling Factual Knowledge in GPT-2 Smallcited by 34
  21. 2022Interpreting Neural Networks through the Polytope Lenscited by 33
  22. 2024Iteration Head: A Mechanistic Study of Chain-of-Thoughtcited by 31
  23. 2023Transformers are uninterpretable with myopic methods: a case study with bounded Dyck grammarscited by 30
  24. 2024Multilinear Mixture of Experts: Scalable Expert Specialization through Factorizationcited by 28
  25. 2023Neuron to Graph: Interpreting Language Model Neurons at Scalecited by 26
  26. 2024Mechanistic?cited by 25
  27. 2023A Glitch in the Matrix? Locating and Detecting Language Model Grounding with Fakepediacited by 24
  28. 2023Interpretability Illusions in the Generalization of Simplified Modelscited by 24
  29. 2024An X-Ray Is Worth 15 Features: Sparse Autoencoders for Interpretable Radiology Report Generationcited by 23
  30. 2023Grokking Group Multiplication with Cosetscited by 22
  31. 2024Learned feature representations are biased by complexity, learning order, position, and morecited by 22
  32. 2026Manifold Steering Reveals the Shared Geometry of Neural Network Representation and Behaviorcited by 22
  33. 2024Decomposing and Editing Predictions by Modeling Model Computationcited by 19
  34. 2025Sparse Autoencoder Features for Classifications and Transferabilitycited by 19
  35. 2024Inferring Functionality of Attention Heads from their Parameterscited by 17
  36. 2024Large Language Model Bias Mitigation from the Perspective of Knowledge Editingcited by 16
  37. 2024Attention layers provably solve single-location regressioncited by 15
  38. 2024Identifying and Mitigating Social Bias Knowledge in Language Modelscited by 13

50 most cited of 94 All 94 (JSON)