# Scoring Rules and Calibration for Imprecise Probabilities

arXiv: https://arxiv.org/abs/2410.23001 · HTML: https://mlsanity.com/paper/2410.23001/

## Key facts

- Cited by 10 papers in the map (data through 14 Oct 2026).
- Builds on 23 of its 86 references that are in the map.
- First posted on arXiv 30 Oct 2024.

## Builds on (most cited first, 23 of 23)

- [Communication-Efficient Learning of Deep Networks from Decentralized Data](https://mlsanity.com/paper/1602.05629/) (2016)
- [Distributionally Robust Neural Networks for Group Shifts: On the Importance of Regularization for Worst-Case Generalization](https://mlsanity.com/paper/1911.08731/) (2019)
- [Aleatoric and Epistemic Uncertainty in Machine Learning: An Introduction to Concepts and Methods](https://mlsanity.com/paper/1910.09457/) (2019)
- [Retiring Adult: New Datasets for Fair Machine Learning](https://mlsanity.com/paper/2108.04884/) (2021)
- [Learning Loss for Active Learning](https://mlsanity.com/paper/1905.03677/) (2019)
- [Learning Models with Uniform Performance via Distributionally Robust Optimization](https://mlsanity.com/paper/1810.08750/) (2018)
- [Wasserstein Distributionally Robust Optimization: Theory and Applications in Machine Learning](https://mlsanity.com/paper/1908.08729/) (2019)
- [Calibrating Predictions to Decisions: A Novel Approach to Multi-Class Calibration](https://mlsanity.com/paper/2107.05719/) (2021)
- [U-Calibration: Forecasting for an Unknown Agent](https://mlsanity.com/paper/2307.00168/) (2023)
- [CreINNs: Credal-Set Interval Neural Networks for Uncertainty Estimation in Classification Tasks](https://mlsanity.com/paper/2401.05043/) (2024)
- [Ensemble-based Uncertainty Quantification: Bayesian versus Credal Inference](https://mlsanity.com/paper/2107.10384/) (2021)
- [Distributionally Robust Post-hoc Classifiers under Prior Shifts](https://mlsanity.com/paper/2309.08825/) (2023)
- [Domain Generalisation via Imprecise Learning](https://mlsanity.com/paper/2404.04669/) (2024)
- [Pretrained Visual Uncertainties](https://mlsanity.com/paper/2402.16569/) (2024)
- [Risk Measures and Upper Probabilities: Coherence and Stratification](https://mlsanity.com/paper/2206.03183/) (2022)
- [On the Richness of Calibration](https://mlsanity.com/paper/2302.04118/) (2023)
- [Right Decisions from Wrong Predictions: A Mechanism Design Alternative to Individual Calibration](https://mlsanity.com/paper/2011.07476/) (2020)
- [The Geometry and Calculus of Losses](https://mlsanity.com/paper/2209.00238/) (2022)
- [Constant Regret, Generalized Mixability, and Mirror Descent](https://mlsanity.com/paper/1802.06965/) (2018)
- [Continuity of Generalized Entropy and Statistical Learning](https://mlsanity.com/paper/2012.15829/) (2020)
- [Information Processing Equalities and the Information-Risk Bridge](https://mlsanity.com/paper/2207.11987/) (2022)
- [Insights From Insurance for Fair Machine Learning](https://mlsanity.com/paper/2306.14624/) (2023)
- [Evaluation Metrics as Averaged Outcomes of Fair Gambles](https://mlsanity.com/paper/2401.14483/) (2024)

## Built on by (most cited first, 10 of 10)

- [ConfTuner: Training Large Language Models to Express Their Confidence Verbally](https://mlsanity.com/paper/2508.18847/) (2025)
- [Integral Imprecise Probability Metrics](https://mlsanity.com/paper/2505.16156/) (2025)
- [Truthful Elicitation of Imprecise Forecasts](https://mlsanity.com/paper/2503.16395/) (2025)
- [Three Types of Calibration with Properties and their Semantic and Formal Relationships](https://mlsanity.com/paper/2504.18395/) (2025)
- [Beyond Prediction-Set Size: Quantifying Conformal Uncertainty through Credal Geometry](https://mlsanity.com/paper/2602.01667/) (2026)
- [Property Elicitation on Imprecise Probabilities](https://mlsanity.com/paper/2507.05857/) (2025)
- [On Calibration in Multi-Distribution Learning](https://mlsanity.com/paper/2412.14142/) (2024)
- [When Do Credal Sets Stabilize? Fixed-Point Theorems for Credal Set Updates](https://mlsanity.com/paper/2510.04769/) (2025)
- [RadFusion: Towards Threshold-Controllable Radiology Report Generation](https://mlsanity.com/paper/2608.10505/) (2026)
- [How Much Imprecision is Enough Imprecision in my Classifier? A Practical Elicitation Procedure](https://mlsanity.com/paper/2609.33352/) (2026)

## Abstract (arXiv)

What does it mean to say that, for example, the probability for rain tomorrow is between 20% and 30%? The theory for the evaluation of precise probabilistic forecasts is well-developed and is grounded in the key concepts of proper scoring rules and calibration. For the case of imprecise probabilistic forecasts (sets of probabilities), such theory is still lacking. In this work, we therefore generalize proper scoring rules and calibration to the imprecise case. We develop these concepts as relative to data models and decision problems. As a consequence, the imprecision is embedded in a clear context. We establish a close link to the paradigm of (group) distributional robustness and in doing so provide new insights for it. We argue that proper scoring rules and calibration serve two distinct goals, which are aligned in the precise case, but intriguingly are not necessarily aligned in the imprecise case. The concept of decision-theoretic entropy plays a key role for both goals. Finally, we demonstrate the theoretical insights in machine learning practice, in particular we illustrate subtle pitfalls relating to the choice of loss function in distributional robustness.