# Towards Modeling Uncertainties of Self-explaining Neural Networks via Conformal Prediction

arXiv: https://arxiv.org/abs/2401.01549 · HTML: https://mlsanity.com/paper/2401.01549/

## Key facts

- Cited by 10 papers in the map (data through 14 Oct 2026).
- Builds on 29 of its 48 references that are in the map.
- First posted on arXiv 3 Jan 2024.

## Builds on (most cited first, 29 of 29)

- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [A Unified Approach to Interpreting Model Predictions](https://mlsanity.com/paper/1705.07874/) (2017)
- ["Why Should I Trust You?": Explaining the Predictions of Any Classifier](https://mlsanity.com/paper/1602.04938/) (2016)
- [Towards Evaluating the Robustness of Neural Networks](https://mlsanity.com/paper/1608.04644/) (2016)
- [Concept Bottleneck Models](https://mlsanity.com/paper/2007.04612/) (2020)
- [A Gentle Introduction to Conformal Prediction and Distribution-Free Uncertainty Quantification](https://mlsanity.com/paper/2107.07511/) (2021)
- [Conformal Prediction Under Covariate Shift](https://mlsanity.com/paper/1904.06019/) (2019)
- [Towards Robust Interpretability with Self-Explaining Neural Networks](https://mlsanity.com/paper/1806.07538/) (2018)
- [Fooling LIME and SHAP: Adversarial Attacks on Post hoc Explanation Methods](https://mlsanity.com/paper/1911.02508/) (2019)
- [Adaptive Conformal Inference Under Distribution Shift](https://mlsanity.com/paper/2106.00170/) (2021)
- [On Completeness-aware Concept-Based Explanations in Deep Neural Networks](https://mlsanity.com/paper/1910.07969/) (2019)
- [Deep Learning for Case-Based Reasoning through Prototypes: A Neural Network that Explains Its Predictions](https://mlsanity.com/paper/1710.04806/) (2017)
- [Reliable Post hoc Explanations: Modeling Uncertainty in Explainability](https://mlsanity.com/paper/2008.05030/) (2020)
- [MetaPoison: Practical General-purpose Clean-label Data Poisoning](https://mlsanity.com/paper/2004.00225/) (2020)
- [Which Explanation Should I Choose? A Function Approximation Perspective to Characterizing Post Hoc Explanations](https://mlsanity.com/paper/2206.01254/) (2022)
- [Sequential Predictive Conformal Inference for Time Series](https://mlsanity.com/paper/2212.03463/) (2022)
- [Exact and Robust Conformal Inference Methods for Predictive Machine Learning With Dependent Data](https://mlsanity.com/paper/1802.06300/) (2018)
- [Relative Attributing Propagation: Interpreting the Comparative Contributions of Individual Units in Deep Neural Networks](https://mlsanity.com/paper/1904.00605/) (2019)
- [SelfExplain: A Self-Explaining Architecture for Neural Text Classifiers](https://mlsanity.com/paper/2103.12279/) (2021)
- [Stop Explaining Black Box Machine Learning Models for High Stakes Decisions and Use Interpretable Models Instead](https://mlsanity.com/paper/1811.10154/) (2018)
- [Predictive Inference with Feature Conformal Prediction](https://mlsanity.com/paper/2210.00173/) (2022)
- [Human-Centered Concept Explanations for Neural Networks](https://mlsanity.com/paper/2202.12451/) (2022)
- [Understanding and Enhancing Robustness of Concept-based Models](https://mlsanity.com/paper/2211.16080/) (2022)
- [Approximating Full Conformal Prediction at Scale via Influence Functions](https://mlsanity.com/paper/2202.01315/) (2022)
- [Exact Optimization of Conformal Predictors via Incremental and Decremental Learning](https://mlsanity.com/paper/2102.03236/) (2021)
- [Instance-wise or Class-wise? A Tale of Neighbor Shapley for Concept-based Explanation](https://mlsanity.com/paper/2109.01369/) (2021)
- [What Do Deep Nets Learn? Class-wise Patterns Revealed in the Input Space](https://mlsanity.com/paper/2101.06898/) (2021)
- [Conformal Frequency Estimation with Sketched Data](https://mlsanity.com/paper/2204.04270/) (2022)
- [Learning to Select Prototypical Parts for Interpretable Sequential Data Modeling](https://mlsanity.com/paper/2212.03396/) (2022)

## Built on by (most cited first, 10 of 10)

- [A Comprehensive Survey on Self-Interpretable Neural Networks](https://mlsanity.com/paper/2501.15638/) (2025)
- [A Comprehensive Survey on the Risks and Limitations of Concept-based Models](https://mlsanity.com/paper/2506.04237/) (2025)
- [Towards Unveiling Predictive Uncertainty Vulnerabilities in the Context of the Right to Be Forgotten](https://mlsanity.com/paper/2508.07458/) (2025)
- [Quantifying Uncertainty in Natural Language Explanations of Large Language Models for Question Answering](https://mlsanity.com/paper/2509.15403/) (2025)
- [Membership Inference Attacks with False Discovery Rate Control](https://mlsanity.com/paper/2508.07066/) (2025)
- [Concerning Uncertainty -- A Systematic Survey of Uncertainty-Aware XAI](https://mlsanity.com/paper/2603.26838/) (2026)
- [Uncertainty-aware Language Guidance for Concept Bottleneck Models](https://mlsanity.com/paper/2602.23495/) (2026)
- [Smoothing-Based Conformal Prediction for Balancing Efficiency and Interpretability](https://mlsanity.com/paper/2509.22529/) (2025)
- [Empirical Characterization of Rationale Stability Under Controlled Perturbations for Explainable Pattern Recognition](https://mlsanity.com/paper/2604.04456/) (2026)
- [Quantifying and Understanding Uncertainty in Large Reasoning Models](https://mlsanity.com/paper/2604.13395/) (2026)

## Abstract (arXiv)

Despite the recent progress in deep neural networks (DNNs), it remains challenging to explain the predictions made by DNNs. Existing explanation methods for DNNs mainly focus on post-hoc explanations where another explanatory model is employed to provide explanations. The fact that post-hoc methods can fail to reveal the actual original reasoning process of DNNs raises the need to build DNNs with built-in interpretability. Motivated by this, many self-explaining neural networks have been proposed to generate not only accurate predictions but also clear and intuitive insights into why a particular decision was made. However, existing self-explaining networks are limited in providing distribution-free uncertainty quantification for the two simultaneously generated prediction outcomes (i.e., a sample's final prediction and its corresponding explanations for interpreting that prediction). Importantly, they also fail to establish a connection between the confidence values assigned to the generated explanations in the interpretation layer and those allocated to the final predictions in the ultimate prediction layer. To tackle the aforementioned challenges, in this paper, we design a novel uncertainty modeling framework for self-explaining networks, which not only demonstrates strong distribution-free uncertainty modeling performance for the generated explanations in the interpretation layer but also excels in producing efficient and effective prediction sets for the final predictions based on the informative high-level basis explanations. We perform the theoretical analysis for the proposed framework. Extensive experimental evaluation demonstrates the effectiveness of the proposed uncertainty framework.