# Rethinking Early Stopping: Refine, Then Calibrate

arXiv: https://arxiv.org/abs/2501.19195 · HTML: https://mlsanity.com/paper/2501.19195/

## Key facts

- Cited by 17 papers in the map (data through 14 Oct 2026).
- Builds on 39 of its 74 references that are in the map.
- First posted on arXiv 31 Jan 2025.

## Builds on (most cited first, 30 of 39)

- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [Training language models to follow instructions with human feedback](https://mlsanity.com/paper/2203.02155/) (2022)
- [XGBoost: A Scalable Tree Boosting System](https://mlsanity.com/paper/1603.02754/) (2016)
- [On Calibration of Modern Neural Networks](https://mlsanity.com/paper/1706.04599/) (2017)
- [Wide Residual Networks](https://mlsanity.com/paper/1605.07146/) (2016)
- [Convex optimization](https://mlsanity.com/paper/2106.01946/) (2021)
- [Improved Regularization of Convolutional Neural Networks with Cutout](https://mlsanity.com/paper/1708.04552/) (2017)
- [Revisiting Deep Learning Models for Tabular Data](https://mlsanity.com/paper/2106.11959/) (2021)
- [Benign Overfitting in Linear Regression](https://mlsanity.com/paper/1906.11300/) (2019)
- [Revisiting the Calibration of Modern Neural Networks](https://mlsanity.com/paper/2106.07998/) (2021)
- [AutoGluon-Tabular: Robust and Accurate AutoML for Structured Data](https://mlsanity.com/paper/2003.06505/) (2020)
- [Beyond temperature scaling: Obtaining well-calibrated multiclass probabilities with Dirichlet calibration](https://mlsanity.com/paper/1910.12656/) (2019)
- [Verified Uncertainty Calibration](https://mlsanity.com/paper/1909.10155/) (2019)
- [When Do Neural Nets Outperform Boosted Trees on Tabular Data?](https://mlsanity.com/paper/2305.02997/) (2023)
- [Evaluating model calibration in classification](https://mlsanity.com/paper/1902.06977/) (2019)
- [High-Dimensional Asymptotics of Prediction: Ridge Regression and Classification](https://mlsanity.com/paper/1507.03003/) (2015)
- [Mitigating Bias in Calibration Error Estimation](https://mlsanity.com/paper/2012.08668/) (2020)
- [Better by Default: Strong Pre-Tuned MLPs and Boosted Trees on Tabular Data](https://mlsanity.com/paper/2407.04491/) (2024)
- [Classifier Calibration: A survey on how to assess and improve predicted class probabilities](https://mlsanity.com/paper/2112.10327/) (2021)
- [Better Uncertainty Calibration via Proper Scores for Classification and Beyond](https://mlsanity.com/paper/2203.07835/) (2022)
- [Smooth ECE: Principled Reliability Diagrams via Kernel Smoothing](https://mlsanity.com/paper/2309.12236/) (2023)
- [High-dimensional analysis of double descent for linear regression with random projections](https://mlsanity.com/paper/2303.01372/) (2023)
- [A Closer Look at Deep Learning Methods on Tabular Datasets](https://mlsanity.com/paper/2407.00956/) (2024)
- [When Does Optimizing a Proper Loss Yield Calibration?](https://mlsanity.com/paper/2305.18764/) (2023)
- [The Impact of Regularization on High-dimensional Logistic Regression](https://mlsanity.com/paper/1906.03761/) (2019)
- [Beyond calibration: estimating the grouping loss of modern neural networks](https://mlsanity.com/paper/2210.16315/) (2022)
- [Evaluating probabilistic classifiers: Reliability diagrams and score decompositions revisited](https://mlsanity.com/paper/2008.03033/) (2020)
- [Classifier Calibration with ROC-Regularized Isotonic Regression](https://mlsanity.com/paper/2311.12436/) (2023)
- [Large-scale probabilistic predictors with and without guarantees of validity](https://mlsanity.com/paper/1511.00213/) (2015)
- [The Calibration Generalization Gap](https://mlsanity.com/paper/2210.01964/) (2022)

## Built on by (most cited first, 17 of 17)

- [Beyond IID: How General Are Tabular Foundation Models, Really?](https://mlsanity.com/paper/2606.30410/) (2026)
- [Structured Matrix Scaling for Multi-Class Calibration](https://mlsanity.com/paper/2511.03685/) (2025)
- [When Shift Happens - Confounding Is to Blame](https://mlsanity.com/paper/2505.21422/) (2025)
- [Efficient Calibration for Decision Making](https://mlsanity.com/paper/2511.13699/) (2025)
- [Too Sharp, Too Sure: When Calibration Follows Curvature](https://mlsanity.com/paper/2604.20614/) (2026)
- [h-calibration: Rethinking Classifier Recalibration with Probabilistic Error-Bounded Objective](https://mlsanity.com/paper/2506.17968/) (2025)
- [CalArena: A Large-Scale Post-Hoc Calibration Benchmark](https://mlsanity.com/paper/2605.30188/) (2026)
- [On the calibration of survival models with competing risks](https://mlsanity.com/paper/2602.00194/) (2026)
- [The Well-Tempered Classifier: Some Elementary Properties of Temperature Scaling](https://mlsanity.com/paper/2602.14862/) (2026)
- [Enforcing Calibration in Multi-Output Probabilistic Regression with Pre-rank Regularization](https://mlsanity.com/paper/2510.21273/) (2025)
- [Don't stop me now: How Validation Criteria Affect Checkpoint Selection and Early Stopping](https://mlsanity.com/paper/2602.22107/) (2026)
- [V-ECE: Estimating General Expected Calibration Errors](https://mlsanity.com/paper/2602.24230/) (2026)
- [Decomposing Probabilistic Scores: Reliability, Information Loss and Uncertainty](https://mlsanity.com/paper/2603.15232/) (2026)
- [Bandwidth Selection in Kernel Density Estimation for Model Calibration](https://mlsanity.com/paper/2606.29925/) (2026)
- [ufakzeka-karar: An Open Turkish Typed-Decision Model with Order-Invariant Option Scoring](https://mlsanity.com/paper/2610.06744/) (2026)
- [Calibration in One Go: Jointly Learning Accurate and Calibrated Neural Networks](https://mlsanity.com/paper/2610.13693/) (2026)
- [Revisiting Tabular Deep Regression: Objectives and the Limits of Global Early Stopping](https://mlsanity.com/paper/2610.13909/) (2026)

## Abstract (arXiv)

Machine learning classifiers often produce probabilistic predictions that are critical for accurate and interpretable decision-making in various domains. The quality of these predictions is generally evaluated with proper losses, such as cross-entropy, which decompose into two components: calibration error assesses general under/overconfidence, while refinement error measures the ability to distinguish different classes. In this paper, we present a novel variational formulation of the calibration-refinement decomposition that sheds new light on post-hoc calibration, and enables rapid estimation of the different terms. Equipped with this new perspective, we provide theoretical and empirical evidence that calibration and refinement errors are not minimized simultaneously during training. Selecting the best epoch based on validation loss thus leads to a compromise point that is suboptimal for both terms. To address this, we propose minimizing refinement error only during training (Refine,...), before minimizing calibration error post hoc, using standard techniques (...then Calibrate). Our method integrates seamlessly with any classifier and consistently improves performance across diverse classification tasks.