# Early Stopping in Deep Networks: Double Descent and How to Eliminate it

arXiv: https://arxiv.org/abs/2007.10099 · HTML: https://mlsanity.com/paper/2007.10099/

## Key facts

- Cited by 41 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 20 Jul 2020.

## Built on by (most cited first, 30 of 41)

- [Data Determines Distributional Robustness in Contrastive Language Image Pre-training (CLIP)](https://mlsanity.com/paper/2205.01397/) (2022)
- [Unifying Grokking and Double Descent](https://mlsanity.com/paper/2303.06173/) (2023)
- [On the Impossible Safety of Large AI Models](https://mlsanity.com/paper/2209.15259/) (2022)
- [Multi-scale Feature Learning Dynamics: Insights for Double Descent](https://mlsanity.com/paper/2112.03215/) (2021)
- [When and how epochwise double descent happens](https://mlsanity.com/paper/2108.12006/) (2021)
- [Provable Continual Learning via Sketched Jacobian Approximations](https://mlsanity.com/paper/2112.05095/) (2021)
- [Over-training with Mixup May Hurt Generalization](https://mlsanity.com/paper/2303.01475/) (2023)
- [Defects of Convolutional Decoder Networks in Frequency Representation](https://mlsanity.com/paper/2210.09020/) (2022)
- [Disparity Between Batches as a Signal for Early Stopping](https://mlsanity.com/paper/2107.06665/) (2021)
- [Regularization-wise double descent: Why it occurs and how to eliminate it](https://mlsanity.com/paper/2206.01378/) (2022)
- [Post-Hoc Reversal: Are We Selecting Models Prematurely?](https://mlsanity.com/paper/2404.07815/) (2024)
- [What Can Grokking Teach Us About Learning Under Nonstationarity?](https://mlsanity.com/paper/2507.20057/) (2025)
- [Benign Overfitting in Classification: Provably Counter Label Noise with Larger Models](https://mlsanity.com/paper/2206.00501/) (2022)
- [Swing-by Dynamics in Concept Learning and Compositional Generalization](https://mlsanity.com/paper/2410.08309/) (2024)
- [Optimization Variance: Exploring Generalization Properties of DNNs](https://mlsanity.com/paper/2106.01714/) (2021)
- [Trap of Feature Diversity in the Learning of MLPs](https://mlsanity.com/paper/2112.00980/) (2021)
- [Unified Neural Network Scaling Laws and Scale-time Equivalence](https://mlsanity.com/paper/2409.05782/) (2024)
- [Image Augmentation for Satellite Images](https://mlsanity.com/paper/2207.14580/) (2022)
- [Representation Learning Dynamics of Self-Supervised Models](https://mlsanity.com/paper/2309.02011/) (2023)
- [Towards Understanding Epoch-wise Double descent in Two-layer Linear Neural Networks](https://mlsanity.com/paper/2407.09845/) (2024)
- [Not All Explanations for Deep Learning Phenomena Are Equally Valuable](https://mlsanity.com/paper/2506.23286/) (2025)
- [Towards Theoretical Analysis of Transformation Complexity of ReLU DNNs](https://mlsanity.com/paper/2205.01940/) (2022)
- [Dropout Drops Double Descent](https://mlsanity.com/paper/2305.16179/) (2023)
- [On Local Overfitting and Forgetting in Deep Neural Networks](https://mlsanity.com/paper/2412.12968/) (2024)
- [A dynamic view of some anomalous phenomena in SGD](https://mlsanity.com/paper/2505.01751/) (2025)
- [Information FOMO: The unhealthy fear of missing out on information. A method for removing misleading data for healthier models](https://mlsanity.com/paper/2208.13080/) (2022)
- [Early-stopping for Transformer model training](https://mlsanity.com/paper/2510.16074/) (2025)
- [Generalizing similarity in noisy setups: the DIBS phenomenon](https://mlsanity.com/paper/2201.12803/) (2022)
- [Feature Weaken: Vicinal Data Augmentation for Classification](https://mlsanity.com/paper/2211.10944/) (2022)
- [Double Descent of Discrepancy: A Task-, Data-, and Model-Agnostic Phenomenon](https://mlsanity.com/paper/2305.15907/) (2023)

## Abstract (arXiv)

Over-parameterized models, such as large deep networks, often exhibit a double descent phenomenon, whereas a function of model size, error first decreases, increases, and decreases at last. This intriguing double descent behavior also occurs as a function of training epochs and has been conjectured to arise because training epochs control the model complexity. In this paper, we show that such epoch-wise double descent arises for a different reason: It is caused by a superposition of two or more bias-variance tradeoffs that arise because different parts of the network are learned at different epochs, and eliminating this by proper scaling of stepsizes can significantly improve the early stopping performance. We show this analytically for i) linear regression, where differently scaled features give rise to a superposition of bias-variance tradeoffs, and for ii) a two-layer neural network, where the first and second layer each govern a bias-variance tradeoff. Inspired by this theory, we study two standard convolutional networks empirically and show that eliminating epoch-wise double descent through adjusting stepsizes of different layers improves the early stopping performance significantly.