# On the Dynamics of Delayed Generalization in Linear Models

arXiv: https://arxiv.org/abs/2602.08302 · HTML: https://mlsanity.com/paper/2602.08302/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Builds on 21 of its 24 references that are in the map.
- First posted on arXiv 9 Feb 2026.

## Builds on (most cited first, 21 of 21)

- [Scaling Laws for Neural Language Models](https://mlsanity.com/paper/2001.08361/) (2020)
- [Towards Deep Learning Models Resistant to Adversarial Attacks](https://mlsanity.com/paper/1706.06083/) (2017)
- [Understanding deep learning requires rethinking generalization](https://mlsanity.com/paper/1611.03530/) (2016)
- [Progress measures for grokking via mechanistic interpretability](https://mlsanity.com/paper/2301.05217/) (2023)
- [Prevalence of Neural Collapse during the terminal phase of deep learning training](https://mlsanity.com/paper/2008.08186/) (2020)
- [Deep Double Descent: Where Bigger Models and More Data Hurt](https://mlsanity.com/paper/1912.02292/) (2019)
- [The Implicit Bias of Gradient Descent on Separable Data](https://mlsanity.com/paper/1710.10345/) (2017)
- [Grokking: Generalization Beyond Overfitting on Small Algorithmic Datasets](https://mlsanity.com/paper/2201.02177/) (2022)
- [Exploring Generalization in Deep Learning](https://mlsanity.com/paper/1706.08947/) (2017)
- [Towards Understanding Grokking: An Effective Theory of Representation Learning](https://mlsanity.com/paper/2205.10343/) (2022)
- [The Power of Interpolation: Understanding the Effectiveness of SGD in Modern Over-parametrized Learning](https://mlsanity.com/paper/1712.06559/) (2017)
- [Grokking as the Transition from Lazy to Rich Training Dynamics](https://mlsanity.com/paper/2310.06110/) (2023)
- [The Slingshot Mechanism: An Empirical Study of Adaptive Optimizers and the Grokking Phenomenon](https://mlsanity.com/paper/2206.04817/) (2022)
- [Dichotomy of Early and Late Phase Implicit Biases Can Provably Induce Grokking](https://mlsanity.com/paper/2311.18817/) (2023)
- [Why Do You Grok? A Theoretical Analysis of Grokking Modular Addition](https://mlsanity.com/paper/2407.12332/) (2024)
- [Grokking in Linear Estimators -- A Solvable Model that Groks without Understanding](https://mlsanity.com/paper/2310.16441/) (2023)
- [To grok or not to grok: Disentangling generalization and memorization on corrupted algorithmic datasets](https://mlsanity.com/paper/2310.13061/) (2023)
- [The Complexity Dynamics of Grokking](https://mlsanity.com/paper/2412.09810/) (2024)
- [Deep Grokking: Would Deep Neural Networks Generalize Better?](https://mlsanity.com/paper/2405.19454/) (2024)
- [Grokking Beyond the Euclidean Norm of Model Parameters](https://mlsanity.com/paper/2506.05718/) (2025)
- [Grokking at the Edge of Linear Separability](https://mlsanity.com/paper/2410.04489/) (2024)

## Built on by (most cited first, 1 of 1)

- [Biologically Inspired Mechanisms for Facilitating Grokking in Multilayer Perceptrons](https://mlsanity.com/paper/2608.28184/) (2026)

## Abstract (arXiv)

Delayed generalization, where training accuracy becomes nearly perfect substantially before test accuracy, has often been studied in the context of the depth and compositional structure of deep networks. In this paper, we ask whether delayed generalization can occur in simpler settings. We show that it can arise in a linear model trained with gradient descent and an exponential-tail loss on linearly separable data. We characterize the finite-time hyperplane dynamics into three phases: a population phase, a support vector dominated unlearning phase and a generalization phase. The convergence of the residual beyond the maximum-margin direction is sufficient for the unlearning phase to begin. In the support vector dominated phase, we characterize the hyperplane dynamics. The time scale in these dynamics causes delayed generalization. We validate our theory through controlled experiments varying population and support-vector distributions, and further verify the predicted bias behavior in a deep model with a single output neuron on MNIST. Our results provide a unified characterization of the asymptotic and finite-time dynamics underlying delayed generalization in linear models.