mlsanityData to 7 Oct 2026

A Reply to Makelov et al. (2023)'s "Interpretability Illusion" Arguments

arXiv 2401.12631 ↗First posted 23 Jan 2024cs.LG cs.AI cs.CLcited by 15

Not on the map. Why?

Abstract (arXiv)

We respond to the recent paper by Makelov et al. (2023), which reviews subspace interchange intervention methods like distributed alignment search (DAS; Geiger et al. 2023) and claims that these methods potentially cause "interpretability illusions". We first review Makelov et al. (2023)'s technical notion of what an "interpretability illusion" is, and then we show that even intuitive and desirable explanations can qualify as illusions in this sense. As a result, their method of discovering "illusions" can reject explanations they consider "non-illusory". We then argue that the illusions Makelov et al. (2023) see in practice are artifacts of their training and evaluation paradigms. We close by emphasizing that, though we disagree with their core characterization, Makelov et al. (2023)'s examples and discussion have undoubtedly pushed the field of interpretability forward.

Key facts. Cited by 15 papers in the map (data through 7 Oct 2026). Builds on 14 of its 24 references that are in the map. First posted on arXiv 23 Jan 2024.

Builds on

14 of 24 references.

References · 14

  1. 2022Interpretability in the Wild: a Circuit for Indirect Object Identification in GPT-2 smallcited by 1,130
  2. 2023The Linear Representation Hypothesis and the Geometry of Large Language Modelscited by 679
  3. 2020Null It Out: Guarding Protected Attributes by Iterative Nullspace Projectioncited by 442
  4. 2021Causal Abstractions of Neural Networkscited by 401
  5. 2023Emergent Linear Representations in World Models of Self-Supervised Sequence Modelscited by 313
  6. 2023Finding Alignments Between Interpretable Causal Variables and Distributed Neural Representationscited by 197
  7. 2020CausaLM: Causal Model Explanation Through Counterfactual Language Modelscited by 117
  8. 2021Inducing Causal Structure for Interpretable Neural Networkscited by 107
  9. 2023Interpretability at Scale: Identifying Causal Mechanisms in Alpacacited by 102
  10. 2020Are Neural Nets Modular? Inspecting Functional Modularity Through Differentiable Weight Maskscited by 97
  11. 2023Is This the Subspace You Are Looking for? An Interpretability Illusion for Subspace Activation Patchingcited by 75
  12. 2022CEBaB: Estimating the Causal Effects of Real-World Concepts on NLP Model Behaviorcited by 54
2 more
  1. 2023Rigorously Assessing Natural Language Explanations of Neuronscited by 44
  2. 2021Sparse Interventions in Language Models with Differentiable Maskingcited by 34

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 5; 2025: 5; 2026: 5.

15 papers.

Sort:

By area:

  1. 2024ReFT: Representation Finetuning for Language Modelscited by 186
  2. 2024A Primer on the Inner Workings of Transformer-based Language Modelscited by 83
  3. 2025The Non-Linear Representation Dilemma: Is Causal Abstraction Enough for Mechanistic Interpretability?cited by 34
  4. 2025Everything, Everywhere, All at Once: Is Mechanistic Interpretability Identifiable?cited by 32
  5. 2024Mechanistic?cited by 25
  6. 2025Mechanistic Interpretability of Emotion Inference in Large Language Modelscited by 24
  7. 2024Optimal ablation for interpretabilitycited by 21
  8. 2025How does Chain of Thought Think? Mechanistic Interpretability of Chain-of-Thought Reasoning with Sparse Autoencodingcited by 19
  9. 2024Beyond the Doors of Perception: Vision Transformers Represent Relations Between Objectscited by 12
  10. 2025The Ramon Llull's Thinking Machine for Automated Ideationcited by 2
  11. 2026Perturbation: A simple and efficient adversarial tracer for representation learning in language modelscited by 1
  12. 2026Fine-Grained Analysis of Shared Syntactic Mechanisms in Language Modelscited by 0
3 more
  1. 2026Inherited Heads: Audio language models track speakers with their text backbone's attention, and an attention-mass ranking retrieves a different setcited by 0
  2. 2026A Survey on the Linear Representation Hypothesiscited by 0
  3. 2026Right Answer, Wrong Mechanism: Detecting Pernicious Divergence in Causal Interventionscited by 0