mlsanityData to 14 Oct 2026

Validating Mechanistic Interpretations: An Axiomatic Approach

arXiv 2407.13594 ↗First posted 18 Jul 2024cs.LGcited by 4

On the map: Mechanistic interpretability → Circuits and mechanistic analysis

Abstract (arXiv)

Mechanistic interpretability aims to reverse engineer the computation performed by a neural network in terms of its internal components. Although there is a growing body of research on mechanistic interpretation of neural networks, the notion of a mechanistic interpretation itself is often ad-hoc. Inspired by the notion of abstract interpretation from the program analysis literature that aims to develop approximate semantics for programs, we give a set of axioms that formally characterize a mechanistic interpretation as a description that approximately captures the semantics of the neural network under analysis in a compositional manner. We demonstrate the applicability of these axioms for validating mechanistic interpretations on an existing, well-known interpretability study as well as on a new case study involving a Transformer-based model trained to solve the well-known 2-SAT problem.

Key facts. Cited by 4 papers in the map (data through 14 Oct 2026). Placed in Circuits and mechanistic analysis, one of 13 directions in Mechanistic interpretability. Builds on 55 of its 72 references that are in the map. First posted on arXiv 18 Jul 2024.

Builds on

55 of 72 references.

Closest ancestors · 5

Same direction, newest first

  1. 2023Towards Automated Circuit Discovery for Mechanistic Interpretability28 Apr
  2. 2023A Toy Model of Universality: Reverse Engineering How Networks Learn Group Operations6 Feb
  3. 2023Progress measures for grokking via mechanistic interpretability12 Jan
  4. 2023Causal Abstraction: A Theoretical Foundation for Mechanistic Interpretability11 Jan
  5. 2022Interpretability in the Wild: a Circuit for Indirect Object Identification in GPT-2 small1 Nov

Other references · 50

  1. 2017Attention Is All You Needcited by 60,361
  2. 2016Grad-CAM: Visual Explanations from Deep Networks via Gradient-based Localizationcited by 6,733
  3. 2013Deep Inside Convolutional Networks: Visualising Image Classification Models and Saliency Mapscited by 2,371
  4. 2023Sparse Autoencoders Find Highly Interpretable Features in Language Modelscited by 1,420
  5. 2016Understanding intermediate layers using linear classifier probescited by 1,353
  6. 2023Representation Engineering: A Top-Down Approach to AI Transparencycited by 1,336
  7. 2017Interpretability Beyond Feature Attribution: Quantitative Testing with Concept Activation Vectors (TCAV)cited by 1,319
  8. 2017SmoothGrad: removing noise by adding noisecited by 1,109
  9. 2019BERT Rediscovers the Classical NLP Pipelinecited by 1,059
  10. 2022In-context Learning and Induction Headscited by 964
  11. 2019Attention is not Explanationcited by 929
  12. 2021Probing Classifiers: Promises, Shortcomings, and Advancescited by 876
38 more
  1. 2018RISE: Randomized Input Sampling for Explanation of Black-box Modelscited by 838
  2. 2020Quantifying Attention Flow in Transformerscited by 775
  3. 2022Discovering Latent Knowledge in Language Models Without Supervisioncited by 767
  4. 2019Designing and Interpreting Probes with Control Taskscited by 640
  5. 2020Transformer Interpretability Beyond Attention Visualizationcited by 627
  6. 2023Eliciting Latent Predictions from Transformers with the Tuned Lenscited by 623
  7. 2017Interpretable Explanations of Black Boxes by Meaningful Perturbationcited by 612
  8. 2019Score-CAM: Score-Weighted Visual Explanations for Convolutional Neural Networkscited by 582
  9. 2023Faith and Fate: Limits of Transformers on Compositionalitycited by 503
  10. 2022Emergent World Representations: Exploring a Sequence Model Trained on a Synthetic Taskcited by 431
  11. 2021Causal Abstractions of Neural Networkscited by 422
  12. 2023Finding Neurons in a Haystack: Case Studies with Sparse Probingcited by 334
  13. 2023Emergent Linear Representations in World Models of Self-Supervised Sequence Modelscited by 319
  14. 2022Transformers Learn Shortcuts to Automatacited by 268
  15. 2019Analyzing the Structure of Attention in a Transformer Language Modelcited by 243
  16. 2019On Completeness-aware Concept-Based Explanations in Deep Neural Networkscited by 239
  17. 2024How to use and interpret activation patchingcited by 214
  18. 2023Finding Alignments Between Interpretable Causal Variables and Distributed Neural Representationscited by 209
  19. 2020Causal Mediation Analysis for Interpreting Neural NLP: The Case of Gender Biascited by 191
  20. 2021Thinking Like Transformerscited by 184
  21. 2023The Clock and the Pizza: Two Stories in Mechanistic Explanation of Neural Networkscited by 156
  22. 2023Does Circuit Analysis Interpretability Scale? Evidence from Multiple Choice Capabilities in Chinchillacited by 149
  23. 2022Toward Transparent AI: A Survey on Interpreting the Inner Structures of Deep Neural Networkscited by 129
  24. 2020The elephant in the interpretability room: Why use attention as explanation when we have saliency methods?cited by 122
  25. 2019Learning to Deceive with Attention-Based Explanationscited by 111
  26. 2021Inducing Causal Structure for Interpretable Neural Networkscited by 110
  27. 2022Unveiling Transformers with LEGO: a synthetic reasoning taskcited by 106
  28. 2023Tracr: Compiled Transformers as a Laboratory for Interpretabilitycited by 104
  29. 2023Interpretability at Scale: Identifying Causal Mechanisms in Alpacacited by 103
  30. 2022Concept Activation Regions: A Generalized Framework For Concept-Based Explanationscited by 57
  31. 2020How Can Self-Attention Networks Recognize Dyck-n Languages?cited by 51
  32. 2023Understanding Addition in Transformerscited by 34
  33. 2018Influence-Directed Explanations for Deep Convolutional Networkscited by 31
  34. 2022Causal Proxy Models for Concept-Based Model Explanationscited by 29
  35. 2022Human-Centered Concept Explanations for Neural Networkscited by 23
  36. 2024Editing Conceptual Knowledge for Large Language Modelscited by 21
  37. 2020Influence Patterns for Explaining Information Flow in BERTcited by 14
  38. 2023Uncovering Intermediate Variables in Transformers using Circuit Probingcited by 12

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 1; 2026: 3.

4 papers.

  1. 2026Formal Mechanistic Interpretability: Automated Circuit Discovery with Provable Guaranteescited by 11
  2. 2025Towards Combinatorial Interpretability of Neural Computationcited by 9
  3. 2026Agent Security is a Systems Problemcited by 5
  4. 2026From Circuit Evidence to Mechanistic Theory: An Inductive Logic Approachcited by 1

Similar, not cited

Close in wording, no citation either way

  1. 2026Tracking Equivalent Mechanistic Interpretations Across Neural Networks
  2. 2026Mechanistic Interpretability for Neural Networks: Circuits, Sparse Features and Symbolic Reasoning
  3. 2025Mechanistic Interpretability Needs Philosophy
  4. 2025A Mathematical Philosophy of Explanations in Mechanistic Interpretability -- The Strange Science Part I.i
  5. 2025Unboxing the Black Box: Mechanistic Interpretability for Algorithmic Understanding of Neural Networks
  6. 2025Everything, Everywhere, All at Once: Is Mechanistic Interpretability Identifiable?
  7. 2025Open Problems in Mechanistic Interpretability
  8. 2025Combining Causal Models for More Accurate Abstractions of Neural Networks