# Can You Trust This Prediction? Auditing Pointwise Reliability After Learning

arXiv: https://arxiv.org/abs/1901.00403 · HTML: https://mlsanity.com/paper/1901.00403/

## Key facts

- Cited by 28 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 2 Jan 2019.

## Built on by (most cited first, 28 of 28)

- [Machine Unlearning of Features and Labels](https://mlsanity.com/paper/2108.11577/) (2021)
- [Metrics reloaded: Recommendations for image analysis validation](https://mlsanity.com/paper/2206.01653/) (2022)
- [If Influence Functions are the Answer, Then What is the Question?](https://mlsanity.com/paper/2209.05364/) (2022)
- [Understanding Influence Functions and Datamodels via Harmonic Analysis](https://mlsanity.com/paper/2210.01072/) (2022)
- [JAWS: Auditing Predictive Uncertainty Under Covariate Shift](https://mlsanity.com/paper/2207.10716/) (2022)
- [How Many and Which Training Points Would Need to be Removed to Flip this Prediction?](https://mlsanity.com/paper/2302.02169/) (2023)
- [A Structured Review of Literature on Uncertainty in Machine Learning & Deep Learning](https://mlsanity.com/paper/2406.00332/) (2024)
- [Error Discovery by Clustering Influence Embeddings](https://mlsanity.com/paper/2312.04712/) (2023)
- [Responsible and Regulatory Conform Machine Learning for Medicine: A Survey of Challenges and Solutions](https://mlsanity.com/paper/2107.09546/) (2021)
- [Algorithmic encoding of protected characteristics in image-based models for disease detection](https://mlsanity.com/paper/2110.14755/) (2021)
- [Deeper Understanding of Black-box Predictions via Generalized Influence Functions](https://mlsanity.com/paper/2312.05586/) (2023)
- [Test for non-negligible adverse shifts](https://mlsanity.com/paper/2107.02990/) (2021)
- [Quality Assurance Challenges for Machine Learning Software Applications During Software Development Life Cycle Phases](https://mlsanity.com/paper/2105.01195/) (2021)
- [Rescaled Influence Functions: Accurate Data Attribution in High Dimension](https://mlsanity.com/paper/2506.06656/) (2025)
- [Influence Based Defense Against Data Poisoning Attacks in Online Learning](https://mlsanity.com/paper/2104.13230/) (2021)
- [Personalized and Reliable Decision Sets: Enhancing Interpretability in Clinical Decision Support Systems](https://mlsanity.com/paper/2107.07483/) (2021)
- [CheXbreak: Misclassification Identification for Deep Learning Models Interpreting Chest X-rays](https://mlsanity.com/paper/2103.09957/) (2021)
- [Detecting and Mitigating Test-time Failure Risks via Model-agnostic Uncertainty Learning](https://mlsanity.com/paper/2109.04432/) (2021)
- [A New Approach for Interpretability and Reliability in Clinical Risk Prediction: Acute Coronary Syndrome Scenario](https://mlsanity.com/paper/2110.08331/) (2021)
- [Measuring Stochastic Data Complexity with Boltzmann Influence Functions](https://mlsanity.com/paper/2406.02745/) (2024)
- [Approximating Full Conformal Prediction for Neural Network Regression with Gauss-Newton Influence](https://mlsanity.com/paper/2507.20272/) (2025)
- [Classification, Slippage, Failure and Discovery](https://mlsanity.com/paper/2104.03886/) (2021)
- [On the Importance of Regularisation & Auxiliary Information in OOD Detection](https://mlsanity.com/paper/2107.07564/) (2021)
- [PREVIS -- A Combined Machine Learning and Visual Interpolation Approach for Interactive Reverse Engineering in Assembly Quality Control](https://mlsanity.com/paper/2201.10257/) (2022)
- [A Meta-heuristic Approach to Estimate and Explain Classifier Uncertainty](https://mlsanity.com/paper/2304.10284/) (2023)
- [Site-specific Deterministic Temperature and Humidity Forecasts with Explainable and Reliable Machine Learning](https://mlsanity.com/paper/2404.03310/) (2024)
- [Global-to-Local Support Spectrums for Language Model Explainability](https://mlsanity.com/paper/2408.05976/) (2024)
- [SG-OIF: A Stability-Guided Online Influence Framework for Reliable Vision Data](https://mlsanity.com/paper/2511.19466/) (2025)

## Abstract (arXiv)

To use machine learning in high stakes applications (e.g. medicine), we need tools for building confidence in the system and evaluating whether it is reliable. Methods to improve model reliability often require new learning algorithms (e.g. using Bayesian inference to obtain uncertainty estimates). An alternative is to audit a model after it is trained. In this paper, we describe resampling uncertainty estimation (RUE), an algorithm to audit the pointwise reliability of predictions. Intuitively, RUE estimates the amount that a prediction would change if the model had been fit on different training data. The algorithm uses the gradient and Hessian of the model's loss function to create an ensemble of predictions. Experimentally, we show that RUE more effectively detects inaccurate predictions than existing tools for auditing reliability subsequent to training. We also show that RUE can create predictive distributions that are competitive with state-of-the-art methods like Monte Carlo dropout, probabilistic backpropagation, and deep ensembles, but does not depend on specific algorithms at train-time like these methods do.