# Beyond calibration: estimating the grouping loss of modern neural networks

arXiv: https://arxiv.org/abs/2210.16315 · HTML: https://mlsanity.com/paper/2210.16315/

## Key facts

- Cited by 32 papers in the map (data through 14 Oct 2026).
- Builds on 14 of its 37 references that are in the map.
- First posted on arXiv 28 Oct 2022.

## Builds on (most cited first, 14 of 14)

- [PyTorch: An Imperative Style, High-Performance Deep Learning Library](https://mlsanity.com/paper/1912.01703/) (2019)
- [BART: Denoising Sequence-to-Sequence Pre-training for Natural Language Generation, Translation, and Comprehension](https://mlsanity.com/paper/1910.13461/) (2019)
- [On Calibration of Modern Neural Networks](https://mlsanity.com/paper/1706.04599/) (2017)
- [Benchmarking Neural Network Robustness to Common Corruptions and Perturbations](https://mlsanity.com/paper/1903.12261/) (2019)
- [A Broad-Coverage Challenge Corpus for Sentence Understanding through Inference](https://mlsanity.com/paper/1704.05426/) (2017)
- [The Many Faces of Robustness: A Critical Analysis of Out-of-Distribution Generalization](https://mlsanity.com/paper/2006.16241/) (2020)
- [Inherent Trade-Offs in the Fair Determination of Risk Scores](https://mlsanity.com/paper/1609.05807/) (2016)
- [Revisiting the Calibration of Modern Neural Networks](https://mlsanity.com/paper/2106.07998/) (2021)
- [Beyond temperature scaling: Obtaining well-calibrated multiclass probabilities with Dirichlet calibration](https://mlsanity.com/paper/1910.12656/) (2019)
- [Verified Uncertainty Calibration](https://mlsanity.com/paper/1909.10155/) (2019)
- [Evaluating model calibration in classification](https://mlsanity.com/paper/1902.06977/) (2019)
- [Mitigating Bias in Calibration Error Estimation](https://mlsanity.com/paper/2012.08668/) (2020)
- [Calibration of Neural Networks using Splines](https://mlsanity.com/paper/2006.12800/) (2020)
- [Diagnostic Uncertainty Calibration: Towards Reliable Machine Predictions in Medical Domain](https://mlsanity.com/paper/2007.01659/) (2020)

## Built on by (most cited first, 30 of 32)

- [Beyond Confidence: Reliable Models Should Also Consider Atypicality](https://mlsanity.com/paper/2305.18262/) (2023)
- [Experts Don't Cheat: Learning What You Don't Know By Predicting Pairs](https://mlsanity.com/paper/2402.08733/) (2024)
- [Proximity-Informed Calibration for Deep Neural Networks](https://mlsanity.com/paper/2306.04590/) (2023)
- [When is Multicalibration Post-Processing Necessary?](https://mlsanity.com/paper/2406.06487/) (2024)
- [Rethinking Early Stopping: Refine, Then Calibrate](https://mlsanity.com/paper/2501.19195/) (2025)
- [Anthropomimetic Uncertainty: What Verbalized Uncertainty in Language Models is Missing](https://mlsanity.com/paper/2507.10587/) (2025)
- [Reconfidencing LLMs from the Grouping Loss Perspective](https://mlsanity.com/paper/2402.04957/) (2024)
- [Minimum-Risk Recalibration of Classifiers](https://mlsanity.com/paper/2305.10886/) (2023)
- [Beyond Probability Partitions: Calibrating Neural Networks with Semantic Aware Grouping](https://mlsanity.com/paper/2306.04985/) (2023)
- [Revisiting Confidence Estimation: Towards Reliable Failure Prediction](https://mlsanity.com/paper/2403.02886/) (2024)
- [Cross-Validation Conformal Risk Control](https://mlsanity.com/paper/2401.11974/) (2024)
- [Reassessing How to Compare and Improve the Calibration of Machine Learning Models](https://mlsanity.com/paper/2406.04068/) (2024)
- [Performance Metrics for Probabilistic Ordinal Classifiers](https://mlsanity.com/paper/2309.08701/) (2023)
- [Improving Predictor Reliability with Selective Recalibration](https://mlsanity.com/paper/2410.05407/) (2024)
- [Decision from Suboptimal Classifiers: Excess Risk Pre- and Post-Calibration](https://mlsanity.com/paper/2503.18025/) (2025)
- [Probability Contracts: Accuracy, Coherence, and Decisions Across LLM Interfaces](https://mlsanity.com/paper/2609.37470/) (2026)
- [How to select predictive models for causal inference?](https://mlsanity.com/paper/2302.00370/) (2023)
- [Multiclass Local Calibration with the Jensen-Shannon Distance](https://mlsanity.com/paper/2510.26566/) (2025)
- [Multi-Layer Confidence Scoring for Detection of Out-of-Distribution Samples, Adversarial Attacks, and In-Distribution Misclassifications](https://mlsanity.com/paper/2512.19472/) (2025)
- [On Calibration of Modern Quantized Efficient Neural Networks](https://mlsanity.com/paper/2309.13866/) (2023)
- [On the calibration of powerset speaker diarization models](https://mlsanity.com/paper/2409.15885/) (2024)
- [Towards Precise Prediction Uncertainty in GNNs: Refining GNNs with Topology-grouping Strategy](https://mlsanity.com/paper/2412.14223/) (2024)
- [I-trustworthy Models. A framework for trustworthiness evaluation of probabilistic classifiers](https://mlsanity.com/paper/2501.15617/) (2025)
- [Calibrated and uncertain? Evaluating uncertainty estimates in binary classification models](https://mlsanity.com/paper/2508.11460/) (2025)
- [On the calibration of survival models with competing risks](https://mlsanity.com/paper/2602.00194/) (2026)
- [Calibrating AI Models for Wireless Communications via Conformal Prediction](https://mlsanity.com/paper/2212.07775/) (2022)
- [Uncertainty Estimation based on Geometric Separation](https://mlsanity.com/paper/2301.04452/) (2023)
- [Calibration improves detection of mislabeled examples](https://mlsanity.com/paper/2511.02738/) (2025)
- [Nonparametric Distribution Regression Re-calibration](https://mlsanity.com/paper/2602.13362/) (2026)
- [Decomposing Probabilistic Scores: Reliability, Information Loss and Uncertainty](https://mlsanity.com/paper/2603.15232/) (2026)

## Abstract (arXiv)

The ability to ensure that a classifier gives reliable confidence scores is essential to ensure informed decision-making. To this end, recent work has focused on miscalibration, i.e., the over or under confidence of model scores. Yet calibration is not enough: even a perfectly calibrated classifier with the best possible accuracy can have confidence scores that are far from the true posterior probabilities. This is due to the grouping loss, created by samples with the same confidence scores but different true posterior probabilities. Proper scoring rule theory shows that given the calibration loss, the missing piece to characterize individual errors is the grouping loss. While there are many estimators of the calibration loss, none exists for the grouping loss in standard settings. Here, we propose an estimator to approximate the grouping loss. We show that modern neural network architectures in vision and NLP exhibit grouping loss, notably in distribution shifts settings, which highlights the importance of pre-production validation.