# Unboxing the Black Box: Mechanistic Interpretability for Algorithmic Understanding of Neural Networks arXiv: https://arxiv.org/abs/2511.19265 ยท HTML: https://mlsanity.com/paper/2511.19265/ ## Key facts - Cited by 6 papers in the map (data through 14 Oct 2026). - Placed in Circuits and mechanistic analysis, one of 13 directions in Mechanistic interpretability. - Builds on 113 of its 163 references that are in the map. - First posted on arXiv 24 Nov 2025. ## Placed in - Mechanistic interpretability / [Circuits and mechanistic analysis](https://mlsanity.com/mechinterp/circuits-and-mechanistic-analysis/) ## Builds on (most cited first, 30 of 113) - [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018) - [A Unified Approach to Interpreting Model Predictions](https://mlsanity.com/paper/1705.07874/) (2017) - ["Why Should I Trust You?": Explaining the Predictions of Any Classifier](https://mlsanity.com/paper/1602.04938/) (2016) - [BLIP: Bootstrapping Language-Image Pre-training for Unified Vision-Language Understanding and Generation](https://mlsanity.com/paper/2201.12086/) (2022) - [Locating and Editing Factual Associations in GPT](https://mlsanity.com/paper/2202.05262/) (2022) - [Deep contextualized word representations](https://mlsanity.com/paper/1802.05365/) (2018) - [Explainable Artificial Intelligence (XAI): Concepts, Taxonomies, Opportunities and Challenges toward Responsible AI](https://mlsanity.com/paper/1910.10045/) (2019) - [Sparse Autoencoders Find Highly Interpretable Features in Language Models](https://mlsanity.com/paper/2309.08600/) (2023) - [Understanding intermediate layers using linear classifier probes](https://mlsanity.com/paper/1610.01644/) (2016) - [Transformer Feed-Forward Layers Are Key-Value Memories](https://mlsanity.com/paper/2012.14913/) (2020) - [Interpretability in the Wild: a Circuit for Indirect Object Identification in GPT-2 small](https://mlsanity.com/paper/2211.00593/) (2022) - [Toy Models of Superposition](https://mlsanity.com/paper/2209.10652/) (2022) - [In-context Learning and Induction Heads](https://mlsanity.com/paper/2209.11895/) (2022) - [Adversarial Examples Are Not Bugs, They Are Features](https://mlsanity.com/paper/1905.02175/) (2019) - [Probing Classifiers: Promises, Shortcomings, and Advances](https://mlsanity.com/paper/2102.12452/) (2021) - [Discovering Latent Knowledge in Language Models Without Supervision](https://mlsanity.com/paper/2212.03827/) (2022) - [Extracting Training Data from Diffusion Models](https://mlsanity.com/paper/2301.13188/) (2023) - [Towards Automated Circuit Discovery for Mechanistic Interpretability](https://mlsanity.com/paper/2304.14997/) (2023) - [The Linear Representation Hypothesis and the Geometry of Large Language Models](https://mlsanity.com/paper/2311.03658/) (2023) - [The Geometry of Truth: Emergent Linear Structure in Large Language Model Representations of True/False Datasets](https://mlsanity.com/paper/2310.06824/) (2023) - [Designing and Interpreting Probes with Control Tasks](https://mlsanity.com/paper/1909.03368/) (2019) - [Eliciting Latent Predictions from Transformers with the Tuned Lens](https://mlsanity.com/paper/2303.08112/) (2023) - [Transformer Feed-Forward Layers Build Predictions by Promoting Concepts in the Vocabulary Space](https://mlsanity.com/paper/2203.14680/) (2022) - [Dissecting Recall of Factual Associations in Auto-Regressive Language Models](https://mlsanity.com/paper/2304.14767/) (2023) - [What you can cram into a single vector: Probing sentence embeddings for linguistic properties](https://mlsanity.com/paper/1805.01070/) (2018) - [What do you learn from context? Probing for sentence structure in contextualized word representations](https://mlsanity.com/paper/1905.06316/) (2019) - [Sparse Feature Circuits: Discovering and Editing Interpretable Causal Graphs in Language Models](https://mlsanity.com/paper/2403.19647/) (2024) - [Mechanistic Interpretability for AI Safety -- A Review](https://mlsanity.com/paper/2404.14082/) (2024) - [A Survey of Transformers](https://mlsanity.com/paper/2106.04554/) (2021) - [Finding Neurons in a Haystack: Case Studies with Sparse Probing](https://mlsanity.com/paper/2305.01610/) (2023) ## Built on by (most cited first, 6 of 6) - [Mechanistic Interpretability of Brain-to-Speech Models Across Speech Modes](https://mlsanity.com/paper/2602.01247/) (2026) - [WASD: Locating Critical Neurons as Sufficient Conditions for Explaining and Controlling LLM Behavior](https://mlsanity.com/paper/2603.18474/) (2026) - [Mechanistic Interpretability for Neural Networks: Circuits, Sparse Features and Symbolic Reasoning](https://mlsanity.com/paper/2607.07316/) (2026) - [Interpretative Interfaces: Designing for AI-Mediated Reading Practices and the Knowledge Commons](https://mlsanity.com/paper/2603.15863/) (2026) - [GeoLAN: Geometric Learning of Latent Explanatory Directions in Large Language Models](https://mlsanity.com/paper/2603.19460/) (2026) - [Mechanistic Interpretability Reveals Shared Causal Subspaces in Brain-to-Speech Decoders](https://mlsanity.com/paper/2609.31992/) (2026) ## Abstract (arXiv) The black box nature of deep neural networks poses a significant challenge for the deployment of transparent and trustworthy artificial intelligence (AI) systems. With the growing presence of AI in society, it becomes increasingly important to develop methods that can explain and interpret the decisions made by these systems. To address this, mechanistic interpretability (MI) emerged as a promising and distinctive research program within the broader field of explainable artificial intelligence (XAI). MI is the process of studying the inner computations of neural networks and translating them into human-understandable algorithms. It encompasses reverse engineering techniques aimed at uncovering the computational algorithms implemented by neural networks. In this article, we propose a unified taxonomy of MI approaches and provide a detailed analysis of key techniques, illustrated with concrete examples and pseudo-code. We contextualize MI within the broader interpretability landscape, comparing its goals, methods, and insights to other strands of XAI. Additionally, we trace the development of MI as a research area, highlighting its conceptual roots and the accelerating pace of recent work. We argue that MI holds significant potential to support a more scientific understanding of machine learning systems -- treating models not only as tools for solving tasks, but also as systems to be studied and understood. We hope to invite new researchers into the field of mechanistic interpretability.