mlsanityData to 14 Oct 2026

Everything, Everywhere, All at Once: Is Mechanistic Interpretability Identifiable?

arXiv 2502.20914 ↗First posted 28 Feb 2025cs.LG cs.AI cs.CLcited by 34

On the map: Mechanistic interpretability → Circuits and mechanistic analysis

Abstract (arXiv)

As AI systems are used in high-stakes applications, ensuring interpretability is crucial. Mechanistic Interpretability (MI) aims to reverse-engineer neural networks by extracting human-understandable algorithms to explain their behavior. This work examines a key question: for a given behavior, and under MI's criteria, does a unique explanation exist? Drawing on identifiability in statistics, where parameters are uniquely inferred under specific assumptions, we explore the identifiability of MI explanations. We identify two main MI strategies: (1) "where-then-what," which isolates a circuit replicating model behavior before interpreting it, and (2) "what-then-where," which starts with candidate algorithms and searches for neural activation subspaces implementing them, using causal alignment. We test both strategies on Boolean functions and small multi-layer perceptrons, fully enumerating candidate explanations. Our experiments reveal systematic non-identifiability: multiple circuits can replicate behavior, a circuit can have multiple interpretations, several algorithms can align with the network, and one algorithm can align with different subspaces. Is uniqueness necessary? A pragmatic approach may require only predictive and manipulability standards. If uniqueness is essential for understanding, stricter criteria may be needed. We also reference the inner interpretability framework, which validates explanations through multiple criteria. This work contributes to defining explanation standards in AI.

Key facts. Cited by 34 papers in the map (data through 14 Oct 2026). Placed in Circuits and mechanistic analysis, one of 13 directions in Mechanistic interpretability. Builds on 33 of its 72 references that are in the map. First posted on arXiv 28 Feb 2025.

Builds on

33 of 72 references.

Closest ancestors · 7

Same direction, newest first

  1. 2024The Computational Complexity of Circuit Discovery for Inner Interpretability10 Oct
  2. 2024Transcoders Find Interpretable LLM Feature Circuits17 Jun
  3. 2024Mechanistic Interpretability for AI Safety -- A Review22 Apr
  4. 2024PURE: Turning Polysemantic Neurons Into Pure Features by Identifying Relevant Circuits9 Apr
  5. 2023Is This the Subspace You Are Looking for? An Interpretability Illusion for Subspace Activation Patching28 Nov
  6. 2023Towards Automated Circuit Discovery for Mechanistic Interpretability28 Apr
  7. 2022Interpretability in the Wild: a Circuit for Indirect Object Identification in GPT-2 small1 Nov

Other references · 26

  1. 2022Locating and Editing Factual Associations in GPTcited by 2,691
  2. 2015Learning Deep Features for Discriminative Localizationcited by 2,436
  3. 2013Deep Inside Convolutional Networks: Visualising Image Classification Models and Saliency Mapscited by 2,371
  4. 2013Visualizing and Understanding Convolutional Networkscited by 2,290
  5. 2018The Lottery Ticket Hypothesis: Finding Sparse, Trainable Neural Networkscited by 2,262
  6. 2023Dissecting Recall of Factual Associations in Auto-Regressive Language Modelscited by 528
  7. 2019The State of Sparsity in Deep Neural Networkscited by 476
  8. 2024Sparse Feature Circuits: Discovering and Editing Interpretable Causal Graphs in Language Modelscited by 435
  9. 2023Finding Alignments Between Interpretable Causal Variables and Distributed Neural Representationscited by 209
  10. 2024Have Faith in Faithfulness: Going Beyond Circuit Overlap When Finding Model Mechanismscited by 149
  11. 2017Causal Consistency of Structural Equation Modelscited by 118
  12. 2024AtP*: An efficient and scalable method for localizing LLM behaviour to componentscited by 111
14 more
  1. 2021Inducing Causal Structure for Interpretable Neural Networkscited by 110
  2. 2018Abstracting Causal Modelscited by 107
  3. 2021Neuron-level Interpretation of Deep NLP Models: A Surveycited by 81
  4. 2021Neural Response Interpretation through the Lens of Critical Pathwayscited by 32
  5. 2023Causal Representation Learning Made Identifiable by Grouping of Observational Variablescited by 32
  6. 2022Predicting is not Understanding: Recognizing and Addressing Underspecification in Machine Learningcited by 26
  7. 2023Analyzing Vision Transformers for Image Classification in Class Embedding Spacecited by 26
  8. 2023A Glitch in the Matrix? Locating and Detecting Language Model Grounding with Fakepediacited by 24
  9. 2024A Survey of Lottery Ticket Hypothesiscited by 24
  10. 2024A Reply to Makelov et al. (2023)'s "Interpretability Illusion" Argumentscited by 17
  11. 2024The Cognitive Revolution in Interpretability: From Explaining Behavior to Interpreting Representations and Algorithmscited by 13
  12. 2021On the Convergence of Shallow Neural Network Training with Randomly Masked Neuronscited by 11
  13. 2024Position: An Inner Interpretability Framework for AI Inspired by Lessons from Cognitive Neurosciencecited by 11
  14. 2021Laughing Heads: Can Transformers Detect What Makes a Sentence Funny?cited by 9

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 11; 2026: 23.

34 papers.

Sort:

By area:

  1. 2025The Non-Linear Representation Dilemma: Is Causal Abstraction Enough for Mechanistic Interpretability?cited by 37
  2. 2025Mechanistic Interpretability as Statistical Estimation: A Variance Analysiscited by 15
  3. 2025Position: Mechanistic Interpretability Should Prioritize Feature Consistency in SAEscited by 14
  4. 2025The Dead Salmons of AI Interpretabilitycited by 12
  5. 2026Formal Mechanistic Interpretability: Automated Circuit Discovery with Provable Guaranteescited by 11
  6. 2025Addressing divergent representations from causal interventions on neural networkscited by 7
  7. 2025Mechanistic Interpretability Needs Philosophycited by 6
  8. 2025Unboxing the Black Box: Mechanistic Interpretability for Algorithmic Understanding of Neural Networkscited by 6
  9. 2025Finding Manifolds With Bilinear Autoencoderscited by 5
  10. 2025Interpretability as Alignment: Making Internal Understanding a Design Principlecited by 3
  11. 2025Distributional Semantics Tracing: A Framework for Explaining Hallucinations in Large Language Modelscited by 3
  12. 2026Diagnosing Generalization Failures from Representational Geometry Markerscited by 2
22 more
  1. 2026Make Mechanistic Interpretability Auditable: A Call to Develop Guidelines via Continuous Collaborative Reviewingcited by 2
  2. 2026Circuit Claims Depend on What Is Extracted and How It Is Comparedcited by 2
  3. 2026Explanation Multiplicity: Circuit-Level Interpretability Evidence Does Not Survive Defensible Analytic Variationcited by 2
  4. 2025EvoXplain: When Machine Learning Models Agree on Predictions but Disagree on Why -- Measuring Mechanistic Multiplicity Across Training Runscited by 1
  5. 2026Transformers converge to invariant algorithmic corescited by 1
  6. 2026Perturbation: A simple and efficient adversarial tracer for representation learning in language modelscited by 1
  7. 2026A Unifying Framework for Unsupervised Concept Extractioncited by 1
  8. 2026Bucketing the Good Apples: A Method for Diagnosing and Improving Causal Abstractioncited by 1
  9. 2026Bilinear autoencoders find interpretable manifoldscited by 1
  10. 2026Judge Circuits Explain Format-Induced Inconsistency in LLM-as-a-Judgecited by 1
  11. 2026Emergent, not Immanent: A Baradian Reading of Explainable AIcited by 0
  12. 2026Behavior Learning (BL): Learning Hierarchical Optimization Structures from Datacited by 0
  13. 2026Transformers Can Learn Rules They've Never Seen: Proof of Computation Beyond Interpolationcited by 0
  14. 2026Probing for Reading Timescited by 0
  15. 2026When Are Two Networks the Same? Tensor Similarity for Mechanistic Interpretabilitycited by 0
  16. 2026Tracing Computation Density in LLMscited by 0
  17. 2026The Role of Rigor in Artificial Intelligencecited by 0
  18. 2026Shortcut Before Circuit: Document Statistics Time In-Context Conflict Resolutioncited by 0
  19. 2026Mechanistic Interpretability Reveals Shared Causal Subspaces in Brain-to-Speech Decoderscited by 0
  20. 2026The Identifiability and Observability of Deep Normalized Attentioncited by 0
  21. 2026FlyHack: Stress-Testing Mechanistic Claims in a Connectome-Constrained Visual Modelcited by 0
  22. 2026Causal Effect Identifiability in Causal Abstractions: A Strict Hierarchy of Criteriacited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2026Mechanistic Interpretability for Neural Networks: Circuits, Sparse Features and Symbolic Reasoning
  2. 2026Tracking Equivalent Mechanistic Interpretations Across Neural Networks
  3. 2024Challenges in Mechanistically Interpreting Model Representations
  4. 2024Validating Mechanistic Interpretations: An Axiomatic Approach
  5. 2025A Mathematical Philosophy of Explanations in Mechanistic Interpretability -- The Strange Science Part I.i
  6. 2026Rethinking Circuit Evaluation: Do Circuits Explain Model Errors?
  7. 2025Open Problems in Mechanistic Interpretability
  8. 2025Combining Causal Models for More Accurate Abstractions of Neural Networks