# Uncertainty Quantification with Proper Scoring Rules: Adjusting Measures to Prediction Tasks

arXiv: https://arxiv.org/abs/2505.22538 · HTML: https://mlsanity.com/paper/2505.22538/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Builds on 34 of its 56 references that are in the map.
- First posted on arXiv 28 May 2025.

## Builds on (most cited first, 30 of 34)

- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [PyTorch: An Imperative Style, High-Performance Deep Learning Library](https://mlsanity.com/paper/1912.01703/) (2019)
- [SGDR: Stochastic Gradient Descent with Warm Restarts](https://mlsanity.com/paper/1608.03983/) (2016)
- [Dropout as a Bayesian Approximation: Representing Model Uncertainty in Deep Learning](https://mlsanity.com/paper/1506.02142/) (2015)
- [Fashion-MNIST: a Novel Image Dataset for Benchmarking Machine Learning Algorithms](https://mlsanity.com/paper/1708.07747/) (2017)
- [Simple and Scalable Predictive Uncertainty Estimation using Deep Ensembles](https://mlsanity.com/paper/1612.01474/) (2016)
- [A Baseline for Detecting Misclassified and Out-of-Distribution Examples in Neural Networks](https://mlsanity.com/paper/1610.02136/) (2016)
- [Natural Adversarial Examples](https://mlsanity.com/paper/1907.07174/) (2019)
- [Aleatoric and Epistemic Uncertainty in Machine Learning: An Introduction to Concepts and Methods](https://mlsanity.com/paper/1910.09457/) (2019)
- [Selective Classification for Deep Neural Networks](https://mlsanity.com/paper/1705.08500/) (2017)
- [Deep Bayesian Active Learning with Image Data](https://mlsanity.com/paper/1703.02910/) (2017)
- [Evidential Deep Learning to Quantify Classification Uncertainty](https://mlsanity.com/paper/1806.01768/) (2018)
- [MedMNIST v2 -- A large-scale lightweight benchmark for 2D and 3D biomedical image classification](https://mlsanity.com/paper/2110.14795/) (2021)
- [ImageNet-21K Pretraining for the Masses](https://mlsanity.com/paper/2104.10972/) (2021)
- [BatchBALD: Efficient and Diverse Batch Acquisition for Deep Bayesian Active Learning](https://mlsanity.com/paper/1906.08158/) (2019)
- [Revisiting the Calibration of Modern Neural Networks](https://mlsanity.com/paper/2106.07998/) (2021)
- [Laplace Redux -- Effortless Bayesian Deep Learning](https://mlsanity.com/paper/2106.14806/) (2021)
- [Deep Evidential Regression](https://mlsanity.com/paper/1910.02600/) (2019)
- [Understanding Measures of Uncertainty for Adversarial Example Detection](https://mlsanity.com/paper/1803.08533/) (2018)
- [Quantifying Aleatoric and Epistemic Uncertainty in Machine Learning: Are Conditional Entropy and Mutual Information Appropriate Measures?](https://mlsanity.com/paper/2209.03302/) (2022)
- [DEUP: Direct Epistemic Uncertainty Prediction](https://mlsanity.com/paper/2102.08501/) (2021)
- [Benchmarking Uncertainty Disentanglement: Specialized Uncertainties for Specialized Tasks](https://mlsanity.com/paper/2402.19460/) (2024)
- [Pitfalls of Epistemic Uncertainty Quantification through Loss Minimisation](https://mlsanity.com/paper/2203.06102/) (2022)
- [Rethinking Aleatoric and Epistemic Uncertainty](https://mlsanity.com/paper/2412.20892/) (2024)
- [Second-Order Uncertainty Quantification: A Distance-Based Approach](https://mlsanity.com/paper/2312.00995/) (2023)
- [Learn to Accumulate Evidence from All Training Samples: Theory and Practice](https://mlsanity.com/paper/2306.11113/) (2023)
- [From Risk to Uncertainty: Generating Predictive Uncertainty Measures via Bayesian Estimation](https://mlsanity.com/paper/2402.10727/) (2024)
- [Epistemic Uncertainty Sampling](https://mlsanity.com/paper/1909.00218/) (2019)
- [Label-wise Aleatoric and Epistemic Uncertainty Quantification](https://mlsanity.com/paper/2406.02354/) (2024)
- [Uncertainty Estimates of Predictions via a General Bias-Variance Decomposition](https://mlsanity.com/paper/2210.12256/) (2022)

## Built on by (most cited first, 1 of 1)

- [ST-LoRA: Single Trajectory LoRA Ensemble for Uncertainty Aware Agricultural Segmentation](https://mlsanity.com/paper/2608.01530/) (2026)

## Abstract (arXiv)

We address the problem of uncertainty quantification and propose measures of total, aleatoric, and epistemic uncertainty based on a known decomposition of (strictly) proper scoring rules, a specific type of loss function, into a divergence and an entropy component. This leads to a flexible framework for uncertainty quantification that can be instantiated with different losses (scoring rules), which makes it possible to tailor uncertainty quantification to the use case at hand. We show that this flexibility is indeed advantageous. In particular, we analyze the task of selective prediction and show that the scoring rule should ideally match the task loss. In addition, we perform experiments on two other common tasks. For out-of-distribution detection, our results confirm that a widely used measure of epistemic uncertainty, mutual information, performs best. Moreover, in the setting of active learning, our measure of epistemic uncertainty based on the zero-one-loss consistently outperforms other uncertainty measures.