# A Comprehensive Survey on Self-Interpretable Neural Networks

arXiv: https://arxiv.org/abs/2501.15638 · HTML: https://mlsanity.com/paper/2501.15638/

## Key facts

- Cited by 18 papers in the map (data through 14 Oct 2026).
- Builds on 247 of its 341 references that are in the map.
- First posted on arXiv 26 Jan 2025.

## Builds on (most cited first, 30 of 247)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Chain-of-Thought Prompting Elicits Reasoning in Large Language Models](https://mlsanity.com/paper/2201.11903/) (2022)
- [A Unified Approach to Interpreting Model Predictions](https://mlsanity.com/paper/1705.07874/) (2017)
- ["Why Should I Trust You?": Explaining the Predictions of Any Classifier](https://mlsanity.com/paper/1602.04938/) (2016)
- [Neural Machine Translation by Jointly Learning to Align and Translate](https://mlsanity.com/paper/1409.0473/) (2014)
- [Axiomatic Attribution for Deep Networks](https://mlsanity.com/paper/1703.01365/) (2017)
- [Tree of Thoughts: Deliberate Problem Solving with Large Language Models](https://mlsanity.com/paper/2305.10601/) (2023)
- [Categorical Reparameterization with Gumbel-Softmax](https://mlsanity.com/paper/1611.01144/) (2016)
- [A Survey on Large Language Model based Autonomous Agents](https://mlsanity.com/paper/2308.11432/) (2023)
- [Explainable Artificial Intelligence (XAI): Concepts, Taxonomies, Opportunities and Challenges toward Responsible AI](https://mlsanity.com/paper/1910.10045/) (2019)
- [Learning Important Features Through Propagating Activation Differences](https://mlsanity.com/paper/1704.02685/) (2017)
- [Sparse Autoencoders Find Highly Interpretable Features in Language Models](https://mlsanity.com/paper/2309.08600/) (2023)
- [Towards A Rigorous Science of Interpretable Machine Learning](https://mlsanity.com/paper/1702.08608/) (2017)
- [Explanation in Artificial Intelligence: Insights from the Social Sciences](https://mlsanity.com/paper/1706.07269/) (2017)
- [Concept Bottleneck Models](https://mlsanity.com/paper/2007.04612/) (2020)
- [Deep Variational Information Bottleneck](https://mlsanity.com/paper/1612.00410/) (2016)
- [This Looks Like That: Deep Learning for Interpretable Image Recognition](https://mlsanity.com/paper/1806.10574/) (2018)
- [The Mythos of Model Interpretability](https://mlsanity.com/paper/1606.03490/) (2016)
- [KAN: Kolmogorov-Arnold Networks](https://mlsanity.com/paper/2404.19756/) (2024)
- [TabNet: Attentive Interpretable Tabular Learning](https://mlsanity.com/paper/1908.07442/) (2019)
- [Attention is not not Explanation](https://mlsanity.com/paper/1908.04626/) (2019)
- [Zero-Shot Learning -- A Comprehensive Evaluation of the Good, the Bad and the Ugly](https://mlsanity.com/paper/1707.00600/) (2017)
- [Towards Robust Interpretability with Self-Explaining Neural Networks](https://mlsanity.com/paper/1806.07538/) (2018)
- [e-SNLI: Natural Language Inference with Natural Language Explanations](https://mlsanity.com/paper/1812.01193/) (2018)
- [Interpretation of Neural Networks is Fragile](https://mlsanity.com/paper/1710.10547/) (2017)
- [Fooling LIME and SHAP: Adversarial Attacks on Post hoc Explanation Methods](https://mlsanity.com/paper/1911.02508/) (2019)
- [Is Attention Interpretable?](https://mlsanity.com/paper/1906.03731/) (2019)
- [Explainability in Graph Neural Networks: A Taxonomic Survey](https://mlsanity.com/paper/2012.15445/) (2020)
- [Explain Yourself! Leveraging Language Models for Commonsense Reasoning](https://mlsanity.com/paper/1906.02361/) (2019)
- [Interpretable Machine Learning: Fundamental Principles and 10 Grand Challenges](https://mlsanity.com/paper/2103.11251/) (2021)

## Built on by (most cited first, 18 of 18)

- [SciHorizon: Benchmarking AI-for-Science Readiness from Scientific Data to Large Language Models](https://mlsanity.com/paper/2503.13503/) (2025)
- [If Concept Bottlenecks are the Question, are Foundation Models the Answer?](https://mlsanity.com/paper/2504.19774/) (2025)
- [Explain Yourself, Briefly! Self-Explaining Neural Networks with Concise Sufficient Reasons](https://mlsanity.com/paper/2502.03391/) (2025)
- [Efficient Skill Discovery via Regret-Aware Optimization](https://mlsanity.com/paper/2506.21044/) (2025)
- [Concepts Worth Having: Refining VLM-Guided Concept Bottleneck Models with Minimal Annotations](https://mlsanity.com/paper/2605.16405/) (2026)
- [Can Neural Networks Learn by Experimenting on Themselves? Self-Interventional Learning from Functional Consequences to Predictive Self-Knowledge](https://mlsanity.com/paper/2608.14894/) (2026)
- [Explanations Go Linear: Post-hoc Explainability for Tabular Data with Interpretable Meta-Encoding](https://mlsanity.com/paper/2504.20667/) (2025)
- [Learning Interpretable Rules from Neural Networks: Neurosymbolic AI for Radar Hand Gesture Recognition](https://mlsanity.com/paper/2506.22443/) (2025)
- [Discovering Decoupled Functional Modules in Large Language Models](https://mlsanity.com/paper/2603.17823/) (2026)
- [Transferring Visual Explainability of Self-Explaining Models to Prediction-Only Models without Additional Training](https://mlsanity.com/paper/2507.04380/) (2025)
- [Enhancing Conversational Recommender Systems with Tree-Structured Knowledge and Pretrained Language Models](https://mlsanity.com/paper/2511.12579/) (2025)
- [Multi-task GINN-LP for Multi-target Symbolic Regression](https://mlsanity.com/paper/2511.13463/) (2025)
- [Minimal Sufficient Representations for Self-interpretable Deep Neural Networks](https://mlsanity.com/paper/2603.24041/) (2026)
- [Subgraph Concept Networks: Concept Levels in Graph Classification](https://mlsanity.com/paper/2604.18868/) (2026)
- [Concept Graph Convolutions: Message Passing in the Concept Space](https://mlsanity.com/paper/2604.20082/) (2026)
- [Why Self-Inconsistency Arises in GNN Explanations and How to Exploit It](https://mlsanity.com/paper/2605.07527/) (2026)
- [Traffic-CBM: A Structurally Interpretable Multimodal Framework for Encrypted Traffic Classification](https://mlsanity.com/paper/2606.29909/) (2026)
- [LiD-GLM: Lipschitz-constrained Deep Generalized Linear Models](https://mlsanity.com/paper/2608.16340/) (2026)

## Abstract (arXiv)

Neural networks have achieved remarkable success across various fields. However, the lack of interpretability limits their practical use, particularly in critical decision-making scenarios. Post-hoc interpretability, which provides explanations for pre-trained models, is often at risk of robustness and fidelity. This has inspired a rising interest in self-interpretable neural networks, which inherently reveal the prediction rationale through the model structures. Although there exist surveys on post-hoc interpretability, a comprehensive and systematic survey of self-interpretable neural networks is still missing. To address this gap, we first collect and review existing works on self-interpretable neural networks and provide a structured summary of their methodologies from five key perspectives: attribution-based, function-based, concept-based, prototype-based, and rule-based self-interpretation. We also present concrete, visualized examples of model explanations and discuss their applicability across diverse scenarios, including image, text, graph data, and deep reinforcement learning. Additionally, we summarize existing evaluation metrics for self-interpretability and identify open challenges in this field, offering insights for future research. To support ongoing developments, we present a publicly accessible resource to track advancements in this domain: https://github.com/yangji721/Awesome-Self-Interpretable-Neural-Network.