# Early Stopping without a Validation Set

arXiv: https://arxiv.org/abs/1703.09580 · HTML: https://mlsanity.com/paper/1703.09580/

## Key facts

- Cited by 39 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 28 Mar 2017.

## Built on by (most cited first, 30 of 39)

- [Surgical Fine-Tuning Improves Adaptation to Distribution Shifts](https://mlsanity.com/paper/2210.11466/) (2022)
- [Early Stopping Against Label Noise Without Validation Data](https://mlsanity.com/paper/2502.07551/) (2025)
- [It's All In the Teacher: Zero-Shot Quantization Brought Closer to the Teacher](https://mlsanity.com/paper/2203.17008/) (2022)
- [Rethinking Few-Shot Adaptation of Vision-Language Models in Two Stages](https://mlsanity.com/paper/2503.11609/) (2025)
- [VICE: Variational Interpretable Concept Embeddings](https://mlsanity.com/paper/2205.00756/) (2022)
- [Image-free Classifier Injection for Zero-Shot Classification](https://mlsanity.com/paper/2308.10599/) (2023)
- [Teaching Models to Teach Themselves: Reasoning at the Edge of Learnability](https://mlsanity.com/paper/2601.18778/) (2026)
- [Channel-Wise Early Stopping without a Validation Set via NNK Polytope Interpolation](https://mlsanity.com/paper/2107.12972/) (2021)
- [Beyond Myopia: Learning from Positive and Unlabeled Data through Holistic Predictive Trends](https://mlsanity.com/paper/2310.04078/) (2023)
- [Towards Understanding How Knowledge Evolves in Large Vision-Language Models](https://mlsanity.com/paper/2504.02862/) (2025)
- [Disparity Between Batches as a Signal for Early Stopping](https://mlsanity.com/paper/2107.06665/) (2021)
- [How Many Validation Labels Do You Need? Exploring the Design Space of Label-Efficient Model Ranking](https://mlsanity.com/paper/2312.01619/) (2023)
- [Cockpit: A Practical Debugging Tool for the Training of Deep Neural Networks](https://mlsanity.com/paper/2102.06604/) (2021)
- [A Layer Selection Approach to Test Time Adaptation](https://mlsanity.com/paper/2404.03784/) (2024)
- [A Realistic Protocol for Evaluation of Weakly Supervised Object Localization](https://mlsanity.com/paper/2404.10034/) (2024)
- [Isotonic Data Augmentation for Knowledge Distillation](https://mlsanity.com/paper/2107.01412/) (2021)
- [Intersection of Parallels as an Early Stopping Criterion](https://mlsanity.com/paper/2208.09529/) (2022)
- [Exploring Selective Layer Fine-Tuning in Federated Learning](https://mlsanity.com/paper/2408.15600/) (2024)
- [Improving Meta-Learning Generalization with Activation-Based Early-Stopping](https://mlsanity.com/paper/2208.02377/) (2022)
- [SWAP: Sparse Entropic Wasserstein Regression for Robust Network Pruning](https://mlsanity.com/paper/2310.04918/) (2023)
- [Addressing caveats of neural persistence with deep graph persistence](https://mlsanity.com/paper/2307.10865/) (2023)
- [Differentiating Policies for Non-Myopic Bayesian Optimization](https://mlsanity.com/paper/2408.07812/) (2024)
- [Noisy Early Stopping for Noisy Labels](https://mlsanity.com/paper/2409.06830/) (2024)
- [Assessing the Effectiveness of Deep Embeddings for Tree Species Classification in the Dutch Forest Inventory](https://mlsanity.com/paper/2508.18829/) (2025)
- [Gradient-Weight Alignment as a Train-Time Proxy for Generalization in Classification Tasks](https://mlsanity.com/paper/2510.25480/) (2025)
- [Exploiting All Samples in Low-Resource Sentence Classification: Early Stopping and Initialization Parameters](https://mlsanity.com/paper/2111.06971/) (2021)
- [Double Descent of Discrepancy: A Task-, Data-, and Model-Agnostic Phenomenon](https://mlsanity.com/paper/2305.15907/) (2023)
- [SANE: The phases of gradient descent through Sharpness Adjusted Number of Effective parameters](https://mlsanity.com/paper/2305.18490/) (2023)
- [When do Convolutional Neural Networks Stop Learning?](https://mlsanity.com/paper/2403.02473/) (2024)
- [Twin: Tuning Learning Rate and Weight Decay of Deep Homogeneous Classifiers without Validation](https://mlsanity.com/paper/2403.05532/) (2024)

## Abstract (arXiv)

Early stopping is a widely used technique to prevent poor generalization performance when training an over-expressive model by means of gradient-based optimization. To find a good point to halt the optimizer, a common practice is to split the dataset into a training and a smaller validation set to obtain an ongoing estimate of the generalization performance. We propose a novel early stopping criterion based on fast-to-compute local statistics of the computed gradients and entirely removes the need for a held-out validation set. Our experiments show that this is a viable approach in the setting of least-squares and logistic regression, as well as neural networks.