# Quantifying the Prediction Uncertainty of Machine Learning Models for Individual Data

arXiv: https://arxiv.org/abs/2412.07520 · HTML: https://mlsanity.com/paper/2412.07520/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 19 of its 130 references that are in the map.
- First posted on arXiv 10 Dec 2024.

## Builds on (most cited first, 19 of 19)

- [Explaining and Harnessing Adversarial Examples](https://mlsanity.com/paper/1412.6572/) (2014)
- [Towards Deep Learning Models Resistant to Adversarial Attacks](https://mlsanity.com/paper/1706.06083/) (2017)
- [Intriguing properties of neural networks](https://mlsanity.com/paper/1312.6199/) (2013)
- [Fashion-MNIST: a Novel Image Dataset for Benchmarking Machine Learning Algorithms](https://mlsanity.com/paper/1708.07747/) (2017)
- [A Baseline for Detecting Misclassified and Out-of-Distribution Examples in Neural Networks](https://mlsanity.com/paper/1610.02136/) (2016)
- [LSUN: Construction of a Large-scale Image Dataset using Deep Learning with Humans in the Loop](https://mlsanity.com/paper/1506.03365/) (2015)
- [Obfuscated Gradients Give a False Sense of Security: Circumventing Defenses to Adversarial Examples](https://mlsanity.com/paper/1802.00420/) (2018)
- [Surprises in High-Dimensional Ridgeless Least Squares Interpolation](https://mlsanity.com/paper/1903.08560/) (2019)
- [On Evaluating Adversarial Robustness](https://mlsanity.com/paper/1902.06705/) (2019)
- [TurkerGaze: Crowdsourcing Saliency with Webcam based Eye Tracking](https://mlsanity.com/paper/1504.06755/) (2015)
- [Benign overfitting in ridge regression](https://mlsanity.com/paper/2009.14286/) (2020)
- [Optimal Regularization Can Mitigate Double Descent](https://mlsanity.com/paper/2003.01897/) (2020)
- [Active Learning for Convolutional Neural Networks: A Core-Set Approach](https://mlsanity.com/paper/1708.00489/) (2017)
- [DeepAL: Deep Active Learning in Python](https://mlsanity.com/paper/2111.15258/) (2021)
- [Deep pNML: Predictive Normalized Maximum Likelihood for Deep Neural Networks](https://mlsanity.com/paper/1904.12286/) (2019)
- [Hyperparameter-Free Out-of-Distribution Detection Using Softmax of Scaled Cosine Similarity](https://mlsanity.com/paper/1905.10628/) (2019)
- [Revisiting minimum description length complexity in overparameterized models](https://mlsanity.com/paper/2006.10189/) (2020)
- [The Generalization Error of the Minimum-norm Solutions for Over-parameterized Neural Networks](https://mlsanity.com/paper/1912.06987/) (2019)
- [Increasing Depth Leads to U-Shaped Test Risk in Over-parameterized Convolutional Networks](https://mlsanity.com/paper/2010.09610/) (2020)

## Abstract (arXiv)

Machine learning models have exhibited exceptional results in various domains. The most prevalent approach for learning is the empirical risk minimizer (ERM), which adapts the model's weights to reduce the loss on a training set and subsequently leverages these weights to predict the label for new test data. Nonetheless, ERM makes the assumption that the test distribution is similar to the training distribution, which may not always hold in real-world situations. In contrast, the predictive normalized maximum likelihood (pNML) was proposed as a min-max solution for the individual setting where no assumptions are made on the distribution of the tested input. This study investigates pNML's learnability for linear regression and neural networks, and demonstrates that pNML can improve the performance and robustness of these models on various tasks. Moreover, the pNML provides an accurate confidence measure for its output, showcasing state-of-the-art results for out-of-distribution detection, resistance to adversarial attacks, and active learning.