mlsanityData to 7 Oct 2026

Beyond calibration: estimating the grouping loss of modern neural networks

arXiv 2210.16315 ↗First posted 28 Oct 2022cs.LG cs.AI stat.MLcited by 32

Not on the map. Why?

Abstract (arXiv)

The ability to ensure that a classifier gives reliable confidence scores is essential to ensure informed decision-making. To this end, recent work has focused on miscalibration, i.e., the over or under confidence of model scores. Yet calibration is not enough: even a perfectly calibrated classifier with the best possible accuracy can have confidence scores that are far from the true posterior probabilities. This is due to the grouping loss, created by samples with the same confidence scores but different true posterior probabilities. Proper scoring rule theory shows that given the calibration loss, the missing piece to characterize individual errors is the grouping loss. While there are many estimators of the calibration loss, none exists for the grouping loss in standard settings. Here, we propose an estimator to approximate the grouping loss. We show that modern neural network architectures in vision and NLP exhibit grouping loss, notably in distribution shifts settings, which highlights the importance of pre-production validation.

Key facts. Cited by 32 papers in the map (data through 7 Oct 2026). Builds on 14 of its 37 references that are in the map. First posted on arXiv 28 Oct 2022.

Builds on

14 of 37 references.

References · 14

  1. 2019PyTorch: An Imperative Style, High-Performance Deep Learning Librarycited by 20,160
  2. 2019BART: Denoising Sequence-to-Sequence Pre-training for Natural Language Generation, Translation, and Comprehensioncited by 5,874
  3. 2017On Calibration of Modern Neural Networkscited by 4,860
  4. 2019Benchmarking Neural Network Robustness to Common Corruptions and Perturbationscited by 2,857
  5. 2017A Broad-Coverage Challenge Corpus for Sentence Understanding through Inferencecited by 2,665
  6. 2020The Many Faces of Robustness: A Critical Analysis of Out-of-Distribution Generalizationcited by 1,842
  7. 2016Inherent Trade-Offs in the Fair Determination of Risk Scorescited by 582
  8. 2021Revisiting the Calibration of Modern Neural Networkscited by 387
  9. 2019Beyond temperature scaling: Obtaining well-calibrated multiclass probabilities with Dirichlet calibrationcited by 359
  10. 2019Verified Uncertainty Calibrationcited by 307
  11. 2019Evaluating model calibration in classificationcited by 165
  12. 2020Mitigating Bias in Calibration Error Estimationcited by 100
2 more
  1. 2020Calibration of Neural Networks using Splinescited by 97
  2. 2020Diagnostic Uncertainty Calibration: Towards Reliable Machine Predictions in Medical Domaincited by 5

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2022: 1; 2023: 8; 2024: 9; 2025: 8; 2026: 6.

32 papers.

Sort:

  1. 2023Beyond Confidence: Reliable Models Should Also Consider Atypicalitycited by 25
  2. 2023Proximity-Informed Calibration for Deep Neural Networkscited by 18
  3. 2024Experts Don't Cheat: Learning What You Don't Know By Predicting Pairscited by 18
  4. 2024When is Multicalibration Post-Processing Necessary?cited by 16
  5. 2025Anthropomimetic Uncertainty: What Verbalized Uncertainty in Language Models is Missingcited by 14
  6. 2025Rethinking Early Stopping: Refine, Then Calibratecited by 13
  7. 2024Reconfidencing LLMs from the Grouping Loss Perspectivecited by 11
  8. 2023Minimum-Risk Recalibration of Classifierscited by 10
  9. 2023Beyond Probability Partitions: Calibrating Neural Networks with Semantic Aware Groupingcited by 10
  10. 2024Revisiting Confidence Estimation: Towards Reliable Failure Predictioncited by 9
  11. 2024Cross-Validation Conformal Risk Controlcited by 8
  12. 2024Reassessing How to Compare and Improve the Calibration of Machine Learning Modelscited by 7
20 more
  1. 2023Performance Metrics for Probabilistic Ordinal Classifierscited by 5
  2. 2024Improving Predictor Reliability with Selective Recalibrationcited by 3
  3. 2025Decision from Suboptimal Classifiers: Excess Risk Pre- and Post-Calibrationcited by 3
  4. 2023How to select predictive models for causal inference?cited by 2
  5. 2025Multi-Layer Confidence Scoring for Detection of Out-of-Distribution Samples, Adversarial Attacks, and In-Distribution Misclassificationscited by 2
  6. 2023On Calibration of Modern Quantized Efficient Neural Networkscited by 1
  7. 2024On the calibration of powerset speaker diarization modelscited by 1
  8. 2024Towards Precise Prediction Uncertainty in GNNs: Refining GNNs with Topology-grouping Strategycited by 1
  9. 2025I-trustworthy Models. A framework for trustworthiness evaluation of probabilistic classifierscited by 1
  10. 2025Calibrated and uncertain? Evaluating uncertainty estimates in binary classification modelscited by 1
  11. 2025Multiclass Local Calibration with the Jensen-Shannon Distancecited by 1
  12. 2022Calibrating AI Models for Wireless Communications via Conformal Predictioncited by 0
  13. 2023Uncertainty Estimation based on Geometric Separationcited by 0
  14. 2025Calibration improves detection of mislabeled examplescited by 0
  15. 2026On the calibration of survival models with competing riskscited by 0
  16. 2026Nonparametric Distribution Regression Re-calibrationcited by 0
  17. 2026Decomposing Probabilistic Scores: Reliability, Information Loss and Uncertaintycited by 0
  18. 2026Tracing sources of epistemic uncertainty in deep learning predictions: homo- and hetero-scedastic linearized estimatorscited by 0
  19. 2026A Convolutional Layer Activation Dimensionality Reduction for Out-of-Distribution and Adversarial Attack Detection Methodscited by 0
  20. 2026Probability Contracts: Accuracy, Coherence, and Decisions Across LLM Interfacescited by 0