# A dynamic view of some anomalous phenomena in SGD

arXiv: https://arxiv.org/abs/2505.01751 · HTML: https://mlsanity.com/paper/2505.01751/

## Key facts

- Cited by 2 papers in the map (data through 14 Oct 2026).
- Builds on 22 of its 51 references that are in the map.
- First posted on arXiv 3 May 2025.

## Builds on (most cited first, 22 of 22)

- [Reconciling modern machine learning practice and the bias-variance trade-off](https://mlsanity.com/paper/1812.11118/) (2018)
- [Deep Double Descent: Where Bigger Models and More Data Hurt](https://mlsanity.com/paper/1912.02292/) (2019)
- [Grokking: Generalization Beyond Overfitting on Small Algorithmic Datasets](https://mlsanity.com/paper/2201.02177/) (2022)
- [Surprises in High-Dimensional Ridgeless Least Squares Interpolation](https://mlsanity.com/paper/1903.08560/) (2019)
- [The generalization error of random features regression: Precise asymptotics and double descent curve](https://mlsanity.com/paper/1908.05355/) (2019)
- [Towards Understanding Grokking: An Effective Theory of Representation Learning](https://mlsanity.com/paper/2205.10343/) (2022)
- [Two models of double descent for weak features](https://mlsanity.com/paper/1903.07571/) (2019)
- [Explaining grokking through circuit efficiency](https://mlsanity.com/paper/2309.02390/) (2023)
- [Triple descent and the two kinds of overfitting: Where & why do they appear?](https://mlsanity.com/paper/2006.03509/) (2020)
- [Multiple Descent: Design Your Own Generalization Curve](https://mlsanity.com/paper/2008.01036/) (2020)
- [A Brief Prehistory of Double Descent](https://mlsanity.com/paper/2004.04328/) (2020)
- [Early Stopping in Deep Networks: Double Descent and How to Eliminate it](https://mlsanity.com/paper/2007.10099/) (2020)
- [Double Descent Demystified: Identifying, Interpreting & Ablating the Sources of a Deep Learning Puzzle](https://mlsanity.com/paper/2303.14151/) (2023)
- [Emergence in non-neural models: grokking modular arithmetic via average gradient outer product](https://mlsanity.com/paper/2407.20199/) (2024)
- [Multi-scale Feature Learning Dynamics: Insights for Double Descent](https://mlsanity.com/paper/2112.03215/) (2021)
- [When and how epochwise double descent happens](https://mlsanity.com/paper/2108.12006/) (2021)
- [Quadratic models for understanding catapult dynamics of neural networks](https://mlsanity.com/paper/2205.11787/) (2022)
- [Concentration bounds for two time scale stochastic approximation](https://mlsanity.com/paper/1806.10798/) (2018)
- [The global convergence time of stochastic gradient descent in non-convex landscapes: Sharp estimates via large deviations](https://mlsanity.com/paper/2503.16398/) (2025)
- [On the Role of Optimization in Double Descent: A Least Squares Study](https://mlsanity.com/paper/2107.12685/) (2021)
- [Understanding the Double Descent Phenomenon in Deep Learning](https://mlsanity.com/paper/2403.10459/) (2024)
- [Towards Understanding Epoch-wise Double descent in Two-layer Linear Neural Networks](https://mlsanity.com/paper/2407.09845/) (2024)

## Built on by (most cited first, 2 of 2)

- [Deep Exploration of Epoch-wise Double Descent in Noisy Data: Signal Separation, Large Activation, and Benign Overfitting](https://mlsanity.com/paper/2601.08316/) (2026)
- [Adynamical systems view of training generativemodels and the memorization phenomenon](https://mlsanity.com/paper/2605.19483/) (2026)

## Abstract (arXiv)

It has been observed by Belkin et al. that over-parametrized neural networks exhibit a `double descent' phenomenon. That is, as the model complexity (as reflected in the number of features) increases, the test error initially decreases, then increases, and then decreases again. A counterpart of this phenomenon in the time domain has been noted in the context of epoch-wise training, viz., the test error decreases with the number of iterates, then increases, then decreases again. Another anomalous phenomenon is that of textitgrokking wherein two regimes of descent are interrupted by a third regime wherein the mean loss remains almost constant. This note presents a plausible explanation for these and related phenomena by using the theory of two time scale stochastic approximation, applied to the continuous time limit of the gradient dynamics. This gives a novel perspective for an already well studied theme.