# When and how epochwise double descent happens

arXiv: https://arxiv.org/abs/2108.12006 · HTML: https://mlsanity.com/paper/2108.12006/

## Key facts

- Cited by 21 papers in the map (data through 14 Oct 2026).
- Builds on 20 of its 32 references that are in the map.
- First posted on arXiv 26 Aug 2021.

## Builds on (most cited first, 20 of 20)

- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [PyTorch: An Imperative Style, High-Performance Deep Learning Library](https://mlsanity.com/paper/1912.01703/) (2019)
- [Rethinking the Inception Architecture for Computer Vision](https://mlsanity.com/paper/1512.00567/) (2015)
- [Understanding deep learning requires rethinking generalization](https://mlsanity.com/paper/1611.03530/) (2016)
- [An overview of gradient descent optimization algorithms](https://mlsanity.com/paper/1609.04747/) (2016)
- [When Does Label Smoothing Help?](https://mlsanity.com/paper/1906.02629/) (2019)
- [Reconciling modern machine learning practice and the bias-variance trade-off](https://mlsanity.com/paper/1812.11118/) (2018)
- [Wide Neural Networks of Any Depth Evolve as Linear Models Under Gradient Descent](https://mlsanity.com/paper/1902.06720/) (2019)
- [Deep Double Descent: Where Bigger Models and More Data Hurt](https://mlsanity.com/paper/1912.02292/) (2019)
- [Gradient Descent Provably Optimizes Over-parameterized Neural Networks](https://mlsanity.com/paper/1810.02054/) (2018)
- [The generalization error of random features regression: Precise asymptotics and double descent curve](https://mlsanity.com/paper/1908.05355/) (2019)
- [Pervasive Label Errors in Test Sets Destabilize Machine Learning Benchmarks](https://mlsanity.com/paper/2103.14749/) (2021)
- [Insights on representational similarity in neural networks with canonical correlation](https://mlsanity.com/paper/1806.05759/) (2018)
- [High-dimensional dynamics of generalization error in neural networks](https://mlsanity.com/paper/1710.03667/) (2017)
- [SGD on Neural Networks Learns Functions of Increasing Complexity](https://mlsanity.com/paper/1905.11604/) (2019)
- [To understand deep learning we need to understand kernel learning](https://mlsanity.com/paper/1802.01396/) (2018)
- [On the geometry of generalization and memorization in deep neural networks](https://mlsanity.com/paper/2105.14602/) (2021)
- [A Brief Prehistory of Double Descent](https://mlsanity.com/paper/2004.04328/) (2020)
- [Early Stopping in Deep Networks: Double Descent and How to Eliminate it](https://mlsanity.com/paper/2007.10099/) (2020)
- [More Data Can Hurt for Linear Regression: Sample-wise Double Descent](https://mlsanity.com/paper/1912.07242/) (2019)

## Built on by (most cited first, 21 of 21)

- [Unifying Grokking and Double Descent](https://mlsanity.com/paper/2303.06173/) (2023)
- [Multi-scale Feature Learning Dynamics: Insights for Double Descent](https://mlsanity.com/paper/2112.03215/) (2021)
- [Over-training with Mixup May Hurt Generalization](https://mlsanity.com/paper/2303.01475/) (2023)
- [Benign Overfitting in Classification: Provably Counter Label Noise with Larger Models](https://mlsanity.com/paper/2206.00501/) (2022)
- [Unified Neural Network Scaling Laws and Scale-time Equivalence](https://mlsanity.com/paper/2409.05782/) (2024)
- [Towards Understanding Epoch-wise Double descent in Two-layer Linear Neural Networks](https://mlsanity.com/paper/2407.09845/) (2024)
- [United We Stand: Using Epoch-wise Agreement of Ensembles to Combat Overfit](https://mlsanity.com/paper/2310.11077/) (2023)
- [On Local Overfitting and Forgetting in Deep Neural Networks](https://mlsanity.com/paper/2412.12968/) (2024)
- [A dynamic view of some anomalous phenomena in SGD](https://mlsanity.com/paper/2505.01751/) (2025)
- [Double Descent and Overparameterization in Particle Physics Data](https://mlsanity.com/paper/2509.01397/) (2025)
- [Two Speeds of Learning: A Representation-Readout Decomposition of Grokking and Double Descent](https://mlsanity.com/paper/2605.27078/) (2026)
- [Noisy Early Stopping for Noisy Labels](https://mlsanity.com/paper/2409.06830/) (2024)
- [With Greater Distance Comes Worse Performance: On the Perspective of Layer Utilization and Model Generalization](https://mlsanity.com/paper/2201.11939/) (2022)
- [Double Descent of Discrepancy: A Task-, Data-, and Model-Agnostic Phenomenon](https://mlsanity.com/paper/2305.15907/) (2023)
- [Relearning Forgotten Knowledge: on Forgetting, Overfit and Training-Free Ensembles of DNNs](https://mlsanity.com/paper/2310.11094/) (2023)
- [Deep Double Descent for Time Series Forecasting: Avoiding Undertrained Models](https://mlsanity.com/paper/2311.01442/) (2023)
- [Double Descent Meets Out-of-Distribution Detection: Theoretical Insights and Empirical Analysis on the role of model complexity](https://mlsanity.com/paper/2411.02184/) (2024)
- [On the Relationship Between Double Descent of CNNs and Shape/Texture Bias Under Learning Process](https://mlsanity.com/paper/2503.02302/) (2025)
- [Forget Me Not: Fighting Local Overfitting with Knowledge Fusion and Distillation](https://mlsanity.com/paper/2507.08686/) (2025)
- [Deep Exploration of Epoch-wise Double Descent in Noisy Data: Signal Separation, Large Activation, and Benign Overfitting](https://mlsanity.com/paper/2601.08316/) (2026)
- [A Data-dependent Early Stopping Rule using Rademacher Complexity with L1-norm](https://mlsanity.com/paper/2608.24210/) (2026)

## Abstract (arXiv)

Deep neural networks are known to exhibit a `double descent' behavior as the number of parameters increases. Recently, it has also been shown that an `epochwise double descent' effect exists in which the generalization error initially drops, then rises, and finally drops again with increasing training time. This presents a practical problem in that the amount of time required for training is long, and early stopping based on validation performance may result in suboptimal generalization. In this work we develop an analytically tractable model of epochwise double descent that allows us to characterise theoretically when this effect is likely to occur. This model is based on the hypothesis that the training data contains features that are slow to learn but informative. We then show experimentally that deep neural networks behave similarly to our theoretical model. Our findings indicate that epochwise double descent requires a critical amount of noise to occur, but above a second critical noise level early stopping remains effective. Using insights from theory, we give two methods by which epochwise double descent can be removed: one that removes slow to learn features from the input and reduces generalization performance, and another that instead modifies the training dynamics and matches or exceeds the generalization performance of standard training. Taken together, our results suggest a new picture of how epochwise double descent emerges from the interplay between the dynamics of training and noise in the training data.