mlsanityData to 7 Oct 2026

Understanding the Generalization Benefits of Late Learning Rate Decay

arXiv 2401.11600 ↗First posted 21 Jan 2024cs.LG stat.MLcited by 6

Not on the map. Why?

Abstract (arXiv)

Why do neural networks trained with large learning rates for a longer time often lead to better generalization? In this paper, we delve into this question by examining the relation between training and testing loss in neural networks. Through visualization of these losses, we note that the training trajectory with a large learning rate navigates through the minima manifold of the training loss, finally nearing the neighborhood of the testing loss minimum. Motivated by these findings, we introduce a nonlinear model whose loss landscapes mirror those observed for real neural networks. Upon investigating the training process using SGD on our model, we demonstrate that an extended phase with a large learning rate steers our model towards the minimum norm solution of the training loss, which may achieve near-optimal generalization, thereby affirming the empirically observed benefits of late learning rate decay.

Key facts. Cited by 6 papers in the map (data through 7 Oct 2026). Builds on 56 of its 79 references that are in the map. First posted on arXiv 21 Jan 2024.

Builds on

56 of 79 references.

References · 56

  1. 2014Adam: A Method for Stochastic Optimizationcited by 36,960
  2. 2014Very Deep Convolutional Networks for Large-Scale Image Recognitioncited by 15,267
  3. 2016Layer Normalizationcited by 5,579
  4. 2016On Large-Batch Training for Deep Learning: Generalization Gap and Sharp Minimacited by 1,337
  5. 2017Visualizing the Loss Landscape of Neural Netscited by 1,126
  6. 2015Cyclical Learning Rates for Training Neural Networkscited by 627
  7. 2022Grokking: Generalization Beyond Overfitting on Small Algorithmic Datasetscited by 599
  8. 2018Gradient Descent Finds Global Minima of Deep Neural Networkscited by 572
  9. 2019Benign Overfitting in Linear Regressioncited by 544
  10. 2016Weight Normalization: A Simple Reparameterization to Accelerate Training of Deep Neural Networkscited by 541
  11. 2019Surprises in High-Dimensional Ridgeless Least Squares Interpolationcited by 508
  12. 2021Gradient Descent on Neural Networks Typically Occurs at the Edge of Stabilitycited by 413
38 more
  1. 2017Don't Decay the Learning Rate, Increase the Batch Sizecited by 313
  2. 2019Gradient Descent Maximizes the Margin of Homogeneous Neural Networkscited by 308
  3. 2020Kernel and Rich Regimes in Overparametrized Modelscited by 302
  4. 2017Train longer, generalize better: closing the generalization gap in large batch training of neural networkscited by 278
  5. 2018Implicit Bias of Gradient Descent on Linear Convolutional Networkscited by 251
  6. 2017Non-convex learning via Stochastic Gradient Langevin Dynamics: a nonasymptotic analysiscited by 242
  7. 2022Towards Understanding Grokking: An Effective Theory of Representation Learningcited by 229
  8. 2018Just Interpolate: Kernel "Ridgeless" Regression Can Generalizecited by 190
  9. 2021On the Origin of Implicit Regularization in Stochastic Gradient Descentcited by 183
  10. 2019Towards Explaining the Regularization Effect of Initial Large Learning Rate in Training Neural Networkscited by 162
  11. 2022Self-Stabilization: The Implicit Bias of Gradient Descent at the Edge of Stabilitycited by 151
  12. 2015Stochastic modified equations and adaptive stochastic gradient algorithmscited by 141
  13. 2017A Bayesian Perspective on Generalization and Stochastic Gradient Descentcited by 134
  14. 2022Understanding Gradient Descent on Edge of Stability in Deep Learningcited by 126
  15. 2020Towards Theoretically Understanding Why SGD Generalizes Better Than ADAM in Deep Learningcited by 125
  16. 2019Implicit regularization for deep neural networks driven by an Ornstein-Uhlenbeck like processcited by 123
  17. 2021Label Noise SGD Provably Prefers Flat Global Minimizerscited by 121
  18. 2018An Alternative View: When Does SGD Escape Local Minima?cited by 115
  19. 2018Stochastic Modified Equations and Dynamics of Stochastic Gradient Algorithms I: Mathematical Foundationscited by 109
  20. 2021Implicit Bias of SGD for Diagonal Linear Networks: a Provable Benefit of Stochasticitycited by 107
  21. 2021What Happens after SGD Reaches Zero Loss? --A Mathematical Frameworkcited by 106
  22. 2020On the Optimal Weighted ℓ2 Regularization in Overparameterized Linear Regressioncited by 97
  23. 2020Shape Matters: Understanding the Implicit Bias of the Noise Covariancecited by 88
  24. 2019Implicit Regularization for Optimal Sparse Recoverycited by 78
  25. 2021On the Validity of Modeling SGD with Stochastic Differential Equations (SDEs)cited by 73
  26. 2017A Hitting Time Analysis of Stochastic Gradient Langevin Dynamicscited by 70
  27. 2023Why Do We Need Weight Decay in Modern Deep Learning?cited by 70
  28. 2022SGD with Large Step Sizes Learns Sparse Featurescited by 66
  29. 2022The alignment property of SGD noise and how it helps select flat minima: A stability analysiscited by 64
  30. 2020Reconciling Modern Deep Learning with Traditional Optimization Analyses: The Intrinsic Learning Ratecited by 59
  31. 2016A Variational Analysis of Stochastic Gradient Algorithmscited by 56
  32. 2022Understanding Edge-of-Stability Training Dynamics with a Minimalist Examplecited by 54
  33. 2021Large Learning Rate Tames Homogeneity: Convergence and Balancing Effectcited by 48
  34. 2022Beyond the Quadratic Approximation: the Multiscale Structure of Neural Network Loss Landscapescited by 42
  35. 2022Towards understanding how momentum improves generalization in deep learningcited by 40
  36. 2021Power-law escape rate of SGDcited by 29
  37. 2022Grokking phase transitions in learning local rules with gradient descentcited by 28
  38. 2019Implicit Regularization and Convergence for Weight Normalizationcited by 20

50 most cited of 56 All 56 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 3; 2025: 2; 2026: 1.

6 papers.

  1. 2024Large Stepsize Gradient Descent for Non-Homogeneous Two-Layer Networks: Margin Improvement and Fast Optimizationcited by 17
  2. 2024Understanding the Language Model to Solve the Symbolic Multi-Step Reasoning Problem from the Perspective of Buffer Mechanismcited by 15
  3. 2024Where Do Large Learning Rates Lead Us?cited by 6
  4. 2025Achilles' Heel of Mamba: Essential difficulties of the Mamba architecture demonstrated by synthetic datacited by 3
  5. 2025PBiLoss: Popularity-Aware Regularization to Improve Fairness in Graph-Based Recommender Systemscited by 0
  6. 2026Graph neural network for colliding particles with an application to sea ice floe modelingcited by 0