# Self-explaining AI as an alternative to interpretable AI

arXiv: https://arxiv.org/abs/2002.05149 · HTML: https://mlsanity.com/paper/2002.05149/

## Key facts

- Cited by 18 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 12 Feb 2020.

## Built on by (most cited first, 18 of 18)

- [Toward Transparent AI: A Survey on Interpreting the Inner Structures of Deep Neural Networks](https://mlsanity.com/paper/2207.13243/) (2022)
- [VCNet: A self-explaining model for realistic counterfactual generation](https://mlsanity.com/paper/2212.10847/) (2022)
- [Keep CALM and Improve Visual Feature Attribution](https://mlsanity.com/paper/2106.07861/) (2021)
- [Interpretability Needs a New Paradigm](https://mlsanity.com/paper/2405.05386/) (2024)
- [Survey of Consciousness Theory from Computational Perspective](https://mlsanity.com/paper/2309.10063/) (2023)
- [Geometric Remove-and-Retrain (GOAR): Coordinate-Invariant eXplainable AI Assessment](https://mlsanity.com/paper/2407.12401/) (2024)
- [Combining Cognitive and Generative AI for Self-explanation in Interactive AI Agents](https://mlsanity.com/paper/2407.18335/) (2024)
- [Approximation of group explainers with coalition structure using Monte Carlo sampling on the product space of coalitions and features](https://mlsanity.com/paper/2303.10216/) (2023)
- [A Voting Approach for Explainable Classification with Rule Learning](https://mlsanity.com/paper/2311.07323/) (2023)
- [Uncertainty-Aware Training for Cardiac Resynchronisation Therapy Response Prediction](https://mlsanity.com/paper/2109.10641/) (2021)
- [xLP: Explainable Link Prediction for Master Data Management](https://mlsanity.com/paper/2403.09806/) (2024)
- [Graph-based Neural Modules to Inspect Attention-based Architectures: A Position Paper](https://mlsanity.com/paper/2210.07117/) (2022)
- [Providing Error Detection for Deep Learning Image Classifiers Using Self-Explainability](https://mlsanity.com/paper/2210.08210/) (2022)
- [Inductive Models for Artificial Intelligence Systems are Insufficient without Good Explanations](https://mlsanity.com/paper/2401.09011/) (2024)
- [MedicalPatchNet: A Patch-Based Self-Explainable AI Architecture for Chest X-ray Classification](https://mlsanity.com/paper/2509.07477/) (2025)
- [SX-GeoTree: Self-eXplaining Geospatial Regression Tree Incorporating the Spatial Similarity of Feature Attributions](https://mlsanity.com/paper/2511.19845/) (2025)
- [UNBOX: Unveiling Black-box visual models with Natural-language](https://mlsanity.com/paper/2603.08639/) (2026)
- [Comparing Post-Hoc Explainable AI Methods for Interpreting Black-Box EEG Models in Depression Detection](https://mlsanity.com/paper/2605.28977/) (2026)

## Abstract (arXiv)

The ability to explain decisions made by AI systems is highly sought after, especially in domains where human lives are at stake such as medicine or autonomous vehicles. While it is often possible to approximate the input-output relations of deep neural networks with a few human-understandable rules, the discovery of the double descent phenomena suggests that such approximations do not accurately capture the mechanism by which deep neural networks work. Double descent indicates that deep neural networks typically operate by smoothly interpolating between data points rather than by extracting a few high level rules. As a result, neural networks trained on complex real world data are inherently hard to interpret and prone to failure if asked to extrapolate. To show how we might be able to trust AI despite these problems we introduce the concept of self-explaining AI. Self-explaining AIs are capable of providing a human-understandable explanation of each decision along with confidence levels for both the decision and explanation. For this approach to work, it is important that the explanation actually be related to the decision, ideally capturing the mechanism used to arrive at the explanation. Finally, we argue it is important that deep learning based systems include a "warning light" based on techniques from applicability domain analysis to warn the user if a model is asked to extrapolate outside its training distribution. For a video presentation of this talk see https://www.youtube.com/watch?v=Py7PVdcu7WY& .