# A Mechanistic Explanatory Strategy for XAI

arXiv: https://arxiv.org/abs/2411.01332 · HTML: https://mlsanity.com/paper/2411.01332/

## Key facts

- Cited by 3 papers in the map (data through 14 Oct 2026).
- Builds on 17 of its 66 references that are in the map.
- First posted on arXiv 2 Nov 2024.

## Builds on (most cited first, 17 of 17)

- ["Why Should I Trust You?": Explaining the Predictions of Any Classifier](https://mlsanity.com/paper/1602.04938/) (2016)
- [Sparse Autoencoders Find Highly Interpretable Features in Language Models](https://mlsanity.com/paper/2309.08600/) (2023)
- [Explanation in Artificial Intelligence: Insights from the Social Sciences](https://mlsanity.com/paper/1706.07269/) (2017)
- [In-context Learning and Induction Heads](https://mlsanity.com/paper/2209.11895/) (2022)
- [Towards Automated Circuit Discovery for Mechanistic Interpretability](https://mlsanity.com/paper/2304.14997/) (2023)
- [The Mythos of Model Interpretability](https://mlsanity.com/paper/1606.03490/) (2016)
- [Methods for Interpreting and Understanding Deep Neural Networks](https://mlsanity.com/paper/1706.07979/) (2017)
- [Understanding Neural Networks Through Deep Visualization](https://mlsanity.com/paper/1506.06579/) (2015)
- [Explaining Explanations in AI](https://mlsanity.com/paper/1811.01439/) (2018)
- [Explainable AI: Beware of Inmates Running the Asylum Or: How I Learnt to Stop Worrying and Love the Social and Behavioural Sciences](https://mlsanity.com/paper/1712.00547/) (2017)
- [What Does Explainable AI Really Mean? A New Conceptualization of Perspectives](https://mlsanity.com/paper/1710.00794/) (2017)
- [The Pragmatic Turn in Explainable Artificial Intelligence (XAI)](https://mlsanity.com/paper/2002.09595/) (2020)
- [Interpretable to Whom? A Role-based Model for Analyzing Interpretable Machine Learning Systems](https://mlsanity.com/paper/1806.07552/) (2018)
- [FIND: A Function Description Benchmark for Evaluating Interpretability Methods](https://mlsanity.com/paper/2309.03886/) (2023)
- [Reasons, Values, Stakeholders: A Philosophical Framework for Explainable Artificial Intelligence](https://mlsanity.com/paper/2103.00752/) (2021)
- [Examining the causal structures of deep neural networks using information theory](https://mlsanity.com/paper/2010.13871/) (2020)
- [Explainable and Human-Grounded AI for Decision Support Systems: The Theory of Epistemic Quasi-Partnerships](https://mlsanity.com/paper/2409.14839/) (2024)

## Built on by (most cited first, 3 of 3)

- [Taming Knowledge Conflicts in Language Models](https://mlsanity.com/paper/2503.10996/) (2025)
- [Mechanistic Interpretability Needs Philosophy](https://mlsanity.com/paper/2506.18852/) (2025)
- [Unboxing the Black Box: Mechanistic Interpretability for Algorithmic Understanding of Neural Networks](https://mlsanity.com/paper/2511.19265/) (2025)

## Abstract (arXiv)

Despite significant advancements in XAI, scholars note a persistent lack of solid conceptual foundations and integration with broader scientific discourse on explanation. In response, emerging research draws on explanatory strategies from various sciences and the philosophy of science literature to fill these gaps. This paper outlines a mechanistic strategy for explaining the functional organization of deep learning systems, situating recent developments in explainable AI within a broader philosophical context. According to the mechanistic approach, the explanation of opaque AI systems involves identifying mechanisms that drive decision making. For deep neural networks, this means discerning functionally relevant components, such as neurons, layers, circuits, or activation patterns, and understanding their roles through decomposition, localization, and recomposition. Proof-of-principle case studies from image recognition and language modeling align these theoretical approaches with mechanistic interpretability research from OpenAI and Anthropic. The findings suggest that pursuing mechanistic explanations can uncover elements that traditional explainability techniques may overlook, ultimately contributing to more thoroughly explainable AI