# Don't stop me now: How Validation Criteria Affect Checkpoint Selection and Early Stopping

arXiv: https://arxiv.org/abs/2602.22107 · HTML: https://mlsanity.com/paper/2602.22107/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 19 of its 56 references that are in the map.
- First posted on arXiv 25 Feb 2026.

## Builds on (most cited first, 19 of 19)

- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [High-dimensional dynamics of generalization error in neural networks](https://mlsanity.com/paper/1710.03667/) (2017)
- [Qualitatively characterizing neural network optimization problems](https://mlsanity.com/paper/1412.6544/) (2014)
- [PolyLoss: A Polynomial Expansion Perspective of Classification Loss Functions](https://mlsanity.com/paper/2204.12511/) (2022)
- [Loss Functions and Metrics in Deep Learning](https://mlsanity.com/paper/2307.02694/) (2023)
- [Early-stopped neural networks are consistent](https://mlsanity.com/paper/2106.05932/) (2021)
- [Addressing the Loss-Metric Mismatch with Adaptive Loss Alignment](https://mlsanity.com/paper/1905.05895/) (2019)
- [No bad local minima: Data independent training error guarantees for multilayer neural networks](https://mlsanity.com/paper/1605.08361/) (2016)
- [How deep is deep enough? -- Quantifying class separability in the hidden layers of deep neural networks](https://mlsanity.com/paper/1811.01753/) (2018)
- [Early stopping for kernel boosting algorithms: A general analysis with localized complexities](https://mlsanity.com/paper/1707.01543/) (2017)
- [Local minima in training of neural networks](https://mlsanity.com/paper/1611.06310/) (2016)
- [Rethinking Early Stopping: Refine, Then Calibrate](https://mlsanity.com/paper/2501.19195/) (2025)
- [On The Effects Of Data Normalisation For Domain Adaptation On EEG Data](https://mlsanity.com/paper/2210.01081/) (2022)
- [Don't Push the Button! Exploring Data Leakage Risks in Machine Learning and Transfer Learning](https://mlsanity.com/paper/2401.13796/) (2024)
- [Score-oriented loss (SOL) functions](https://mlsanity.com/paper/2103.15522/) (2021)
- [A Surprising Linear Relationship Predicts Test Performance in Deep Networks](https://mlsanity.com/paper/1807.09659/) (2018)
- [A comprehensive theoretical framework for the optimization of neural networks classification performance with respect to weighted metrics](https://mlsanity.com/paper/2305.13472/) (2023)
- [Pre-interpolation loss behaviour in neural networks](https://mlsanity.com/paper/2103.07986/) (2021)
- [PACMAN: PAC-style bounds accounting for the Mismatch between Accuracy and Negative log-loss](https://mlsanity.com/paper/2112.05547/) (2021)

## Abstract (arXiv)

Checkpoint selection is a standard component of neural network training, yet the validation criterion used to select a checkpoint is often chosen heuristically. Moreover, the same criterion may be used either only to rank checkpoints after completion of a predefined training run or also to determine when training should stop, thereby affecting both the selected checkpoint and the set of checkpoints available for selection. In this work, we systematically investigate the role of validation criteria under these two settings. We separately vary the training loss, the validation criterion, and the target evaluation metric, and compare post-hoc checkpoint selection, in which training proceeds for all predefined epochs, with patience-based early stopping, in which the validation criterion also controls training termination. We consider three Cross-Entropy, C-Loss, and PolyLoss as training losses, and accuracy, macro-F1, and Matthews correlation coefficient as target metrics. Selection quality is assessed through the relative gap between the test performance of the validation-selected checkpoint and the best-observed test performance for the same target metric over the complete predefined training run.