# Explaining Predictive Uncertainty by Exposing Second-Order Effects

arXiv: https://arxiv.org/abs/2401.17441 · HTML: https://mlsanity.com/paper/2401.17441/

## Key facts

- Cited by 10 papers in the map (data through 14 Oct 2026).
- Builds on 33 of its 60 references that are in the map.
- First posted on arXiv 30 Jan 2024.

## Builds on (most cited first, 30 of 33)

- [Very Deep Convolutional Networks for Large-Scale Image Recognition](https://mlsanity.com/paper/1409.1556/) (2014)
- [Grad-CAM: Visual Explanations from Deep Networks via Gradient-based Localization](https://mlsanity.com/paper/1610.02391/) (2016)
- [A Unified Approach to Interpreting Model Predictions](https://mlsanity.com/paper/1705.07874/) (2017)
- ["Why Should I Trust You?": Explaining the Predictions of Any Classifier](https://mlsanity.com/paper/1602.04938/) (2016)
- [On Calibration of Modern Neural Networks](https://mlsanity.com/paper/1706.04599/) (2017)
- [Dropout as a Bayesian Approximation: Representing Model Uncertainty in Deep Learning](https://mlsanity.com/paper/1506.02142/) (2015)
- [Simple and Scalable Predictive Uncertainty Estimation using Deep Ensembles](https://mlsanity.com/paper/1612.01474/) (2016)
- [Axiomatic Attribution for Deep Networks](https://mlsanity.com/paper/1703.01365/) (2017)
- [Learning Important Features Through Propagating Activation Differences](https://mlsanity.com/paper/1704.02685/) (2017)
- [Can You Trust Your Model's Uncertainty? Evaluating Predictive Uncertainty Under Dataset Shift](https://mlsanity.com/paper/1906.02530/) (2019)
- [Averaging Weights Leads to Wider Optima and Better Generalization](https://mlsanity.com/paper/1803.05407/) (2018)
- [A Simple Baseline for Bayesian Uncertainty in Deep Learning](https://mlsanity.com/paper/1902.02476/) (2019)
- [Predictive Uncertainty Estimation via Prior Networks](https://mlsanity.com/paper/1802.10501/) (2018)
- [Evaluating the visualization of what a Deep Neural Network has learned](https://mlsanity.com/paper/1509.06321/) (2015)
- [Deep Evidential Regression](https://mlsanity.com/paper/1910.02600/) (2019)
- [Confidence Calibration and Predictive Uncertainty Estimation for Deep Medical Image Segmentation](https://mlsanity.com/paper/1911.13273/) (2019)
- [The Shapley Taylor Interaction Index](https://mlsanity.com/paper/1902.05622/) (2019)
- [Higher-Order Explanations of Graph Neural Networks via Relevant Walks](https://mlsanity.com/paper/2006.03589/) (2020)
- [Forecasting day-ahead electricity prices: A review of state-of-the-art algorithms, best practices and an open-access benchmark](https://mlsanity.com/paper/2008.08004/) (2020)
- [Explaining Explanations: Axiomatic Feature Interactions for Deep Networks](https://mlsanity.com/paper/2002.04138/) (2020)
- [Getting a CLUE: A Method for Explaining Uncertainty Estimates](https://mlsanity.com/paper/2006.06848/) (2020)
- [Uncertainty-Aware Reinforcement Learning for Collision Avoidance](https://mlsanity.com/paper/1702.01182/) (2017)
- [Bayesian Uncertainty Estimation for Batch Normalized Deep Networks](https://mlsanity.com/paper/1802.06455/) (2018)
- [Uncertainty Quantification with Statistical Guarantees in End-to-End Autonomous Driving Control](https://mlsanity.com/paper/1909.09884/) (2019)
- [Explaining Predictive Uncertainty with Information Theoretic Shapley Values](https://mlsanity.com/paper/2306.05724/) (2023)
- [From Clustering to Cluster Explanations via Neural Networks](https://mlsanity.com/paper/1906.07633/) (2019)
- [Building and Interpreting Deep Similarity Models](https://mlsanity.com/paper/2003.05431/) (2020)
- [Toward Explainable AI for Regression Models](https://mlsanity.com/paper/2112.11407/) (2021)
- [Explainable AI for Time Series via Virtual Inspection Layers](https://mlsanity.com/paper/2303.06365/) (2023)
- [Monte Carlo DropBlock for Modelling Uncertainty in Object Detection](https://mlsanity.com/paper/2108.03614/) (2021)

## Built on by (most cited first, 10 of 10)

- [Unveil Sources of Uncertainty: Feature Contribution to Conformal Prediction Intervals](https://mlsanity.com/paper/2505.13118/) (2025)
- [Concerning Uncertainty -- A Systematic Survey of Uncertainty-Aware XAI](https://mlsanity.com/paper/2603.26838/) (2026)
- [Normalized Relevance Measure as a Unifying Framework to Explain Neural Network Latent Structures](https://mlsanity.com/paper/2606.00557/) (2026)
- [CIRCUS: Circuit Consensus under Uncertainty via Stability Ensembles](https://mlsanity.com/paper/2603.00523/) (2026)
- [Conceptualizing Uncertainty: A Concept-based Approach to Explaining Uncertainty](https://mlsanity.com/paper/2503.03443/) (2025)
- [Fast and Accurate Explanations of Distance-Based Classifiers by Uncovering Latent Explanatory Structures](https://mlsanity.com/paper/2508.03913/) (2025)
- [No Single Metric Tells the Whole Story: A Multi-Dimensional Evaluation Framework for Uncertainty Attributions](https://mlsanity.com/paper/2603.24524/) (2026)
- [Uncertainty Gating for Cost-Aware Explainable Artificial Intelligence](https://mlsanity.com/paper/2603.29915/) (2026)
- [Uncertainty-Aware End-to-End AI Weather Forecasting: Disentangling Observation and Model Contributions](https://mlsanity.com/paper/2608.30795/) (2026)
- [Learning Adaptive Deliberation Control for Reliable Recognition with Multi-Agent Language Models](https://mlsanity.com/paper/2610.17840/) (2026)

## Abstract (arXiv)

Explainable AI has brought transparency into complex ML blackboxes, enabling, in particular, to identify which features these models use for their predictions. So far, the question of explaining predictive uncertainty, i.e. why a model 'doubts', has been scarcely studied. Our investigation reveals that predictive uncertainty is dominated by second-order effects, involving single features or product interactions between them. We contribute a new method for explaining predictive uncertainty based on these second-order effects. Computationally, our method reduces to a simple covariance computation over a collection of first-order explanations. Our method is generally applicable, allowing for turning common attribution techniques (LRP, Gradient x Input, etc.) into powerful second-order uncertainty explainers, which we call CovLRP, CovGI, etc. The accuracy of the explanations our method produces is demonstrated through systematic quantitative evaluations, and the overall usefulness of our method is demonstrated via two practical showcases.