# Calibrating sufficiently

arXiv: https://arxiv.org/abs/2105.07283 · HTML: https://mlsanity.com/paper/2105.07283/

## Key facts

- Cited by 10 papers in the map (data through 14 Oct 2026).
- Builds on 6 of its 68 references that are in the map.
- First posted on arXiv 15 May 2021.

## Builds on (most cited first, 6 of 6)

- [Evaluating model calibration in classification](https://mlsanity.com/paper/1902.06977/) (2019)
- [Mitigating Bias in Calibration Error Estimation](https://mlsanity.com/paper/2012.08668/) (2020)
- [Of Quantiles and Expectiles: Consistent Scoring Functions, Choquet Representations, and Forecast Rankings](https://mlsanity.com/paper/1503.08195/) (2015)
- [A Generalized Neyman-Pearson Criterion for Optimal Domain Adaptation](https://mlsanity.com/paper/1810.01545/) (2018)
- [Generic Conditions for Forecast Dominance](https://mlsanity.com/paper/1805.09902/) (2018)
- [A plug-in approach to maximising precision at the top and recall at the top](https://mlsanity.com/paper/1804.03077/) (2018)

## Built on by (most cited first, 10 of 10)

- [Class Prior Estimation under Covariate Shift: No Problem?](https://mlsanity.com/paper/2206.02449/) (2022)
- [Isotonic Recalibration under a Low Signal-to-Noise Ratio](https://mlsanity.com/paper/2301.02692/) (2023)
- [Sparse joint shift in multinomial classification](https://mlsanity.com/paper/2303.16971/) (2023)
- [Minimising quantifier variance under prior probability shift](https://mlsanity.com/paper/2107.08209/) (2021)
- [Factorizable Joint Shift in Multinomial Classification](https://mlsanity.com/paper/2207.14514/) (2022)
- [Invariance assumptions for class distribution estimation](https://mlsanity.com/paper/2311.17225/) (2023)
- [On the Interconnections of Calibration, Quantification, and Classifier Accuracy Prediction under Dataset Shift](https://mlsanity.com/paper/2505.11380/) (2025)
- [Recalibrating binary probabilistic classifiers](https://mlsanity.com/paper/2505.19068/) (2025)
- [Factorizable joint shift revisited](https://mlsanity.com/paper/2601.15036/) (2026)
- [Decomposing Probabilistic Scores: Reliability, Information Loss and Uncertainty](https://mlsanity.com/paper/2603.15232/) (2026)

## Abstract (arXiv)

When probabilistic classifiers are trained and calibrated, the so-called grouping loss component of the calibration loss can easily be overlooked. Grouping loss refers to the gap between observable information and information actually exploited in the calibration exercise. We investigate the relation between grouping loss and the concept of sufficiency, identifying comonotonicity as a useful criterion for sufficiency. We revisit the probing reduction approach of Langford & Zadrozny (2005) and find that it produces an estimator of probabilistic classifiers that reduces grouping loss. Finally, we discuss Brier curves as tools to support training and 'sufficient' calibration of probabilistic classifiers.