# Dynamical Decoupling of Generalization and Overfitting in Large Two-Layer Networks

arXiv: https://arxiv.org/abs/2502.21269 · HTML: https://mlsanity.com/paper/2502.21269/

## Key facts

- Cited by 25 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 28 Feb 2025.

## Built on by (most cited first, 25 of 25)

- [Bigger Isn't Always Memorizing: Early Stopping Overparameterized Diffusion Models](https://mlsanity.com/paper/2505.16959/) (2025)
- [The Hidden Width of Deep ResNets: Tight Error Bounds and Phase Diagram](https://mlsanity.com/paper/2509.10167/) (2025)
- [Theory of Scaling Laws for In-Context Regression: Depth, Width, Context and Time](https://mlsanity.com/paper/2510.01098/) (2025)
- [Provable Scaling Laws of Feature Emergence from Learning Dynamics of Grokking](https://mlsanity.com/paper/2509.21519/) (2025)
- [The Nuclear Route: Sharp Asymptotics of ERM in Overparameterized Quadratic Networks](https://mlsanity.com/paper/2505.17958/) (2025)
- [Asymptotics of SGD in Sequence-Single Index Models and Single-Layer Attention Networks](https://mlsanity.com/paper/2506.02651/) (2025)
- [Spectral Dynamics in Deep Networks: Feature Learning, Outlier Escape, and Learning Rate Transfer](https://mlsanity.com/paper/2605.07870/) (2026)
- [AI4SLT: Empirical Processes in Lean 4 for Formal Statistical Learning Theory](https://mlsanity.com/paper/2602.02285/) (2026)
- [Biased Generalization in Diffusion Models](https://mlsanity.com/paper/2603.03469/) (2026)
- [A statistical physics framework for optimal learning](https://mlsanity.com/paper/2507.07907/) (2025)
- [Implicit bias produces neural scaling laws in learning curves, from perceptrons to deep networks](https://mlsanity.com/paper/2505.13230/) (2025)
- [Generalization performance of narrow one-hidden layer networks in the teacher-student setting](https://mlsanity.com/paper/2507.00629/) (2025)
- [Precise Dynamics of Diagonal Linear Networks: A Unifying Analysis by Dynamical Mean-Field Theory](https://mlsanity.com/paper/2510.01930/) (2025)
- [Sharp description of local minima in the loss landscape of high-dimensional two-layer ReLU neural networks](https://mlsanity.com/paper/2604.09412/) (2026)
- [The Interplay of Data Structure and Imbalance in the Learning Dynamics of Diffusion Models](https://mlsanity.com/paper/2605.06367/) (2026)
- [Escape dynamics and implicit bias of one-pass SGD in overparameterized quadratic networks](https://mlsanity.com/paper/2604.03068/) (2026)
- [The two clocks and the innovation window: When and how generative models learn rules](https://mlsanity.com/paper/2605.10019/) (2026)
- [Generalization Bound of Gradient Flow through Training Trajectory and Data-dependent Kernel](https://mlsanity.com/paper/2506.11357/) (2025)
- [A simple mean field model of feature learning](https://mlsanity.com/paper/2510.15174/) (2025)
- [Dichotomy of Feature Learning and Unlearning: Fast-Slow Analysis on Neural Networks with Stochastic Gradient Descent](https://mlsanity.com/paper/2602.07378/) (2026)
- [Decoupled Descent: Exact Test Error Tracking Via Approximate Message Passing](https://mlsanity.com/paper/2604.27883/) (2026)
- [Spherical Boltzmann machines: a solvable theory of learning and generation in energy-based models](https://mlsanity.com/paper/2605.09031/) (2026)
- [Theory of learning of high-dimensional controlled non-linear dynamical systems (I): models and methods](https://mlsanity.com/paper/2606.07247/) (2026)
- [Approximate Message Passing with Random Initialization for Phase Retrieval](https://mlsanity.com/paper/2608.01654/) (2026)
- [Learning Orthogonal Multi-Index Models Beyond Small Initialization: Incremental Learning, Competitive Dynamics and Symmetry](https://mlsanity.com/paper/2609.10879/) (2026)

## Abstract (arXiv)

Understanding the inductive bias and generalization properties of large overparametrized machine learning models requires to characterize the dynamics of the training algorithm. We study the learning dynamics of large two-layer neural networks via dynamical mean field theory, a well established technique of non-equilibrium statistical physics. We show that, for large network width m, and large number of samples per input dimension n/d, the training dynamics exhibits a separation of timescales which implies: (i)~The emergence of a slow time scale associated with the growth in Gaussian/Rademacher complexity of the network; (ii)~Inductive bias towards small complexity if the initialization has small enough complexity; (iii)~A dynamical decoupling between feature learning and overfitting regimes; (iv)~A non-monotone behavior of the test error, associated `feature unlearning' regime at large times.