mlsanityData to 7 Oct 2026

Special Properties of Gradient Descent with Large Learning Rates

arXiv 2205.15142 ↗First posted 30 May 2022cs.LG math.OCcited by 10

Not on the map. Why?

Abstract (arXiv)

When training neural networks, it has been widely observed that a large step size is essential in stochastic gradient descent (SGD) for obtaining superior models. However, the effect of large step sizes on the success of SGD is not well understood theoretically. Several previous works have attributed this success to the stochastic noise present in SGD. However, we show through a novel set of experiments that the stochastic noise is not sufficient to explain good non-convex training, and that instead the effect of a large learning rate itself is essential for obtaining best performance.We demonstrate the same effects also in the noise-less case, i.e. for full-batch GD. We formally prove that GD with large step size -- on certain non-convex function classes -- follows a different trajectory than GD with a small step size, which can lead to convergence to a global minimum instead of a local one. Our settings provide a framework for future analysis which allows comparing algorithms based on behaviors that can not be observed in the traditional settings.

Key facts. Cited by 10 papers in the map (data through 7 Oct 2026). Builds on 39 of its 49 references that are in the map. First posted on arXiv 30 May 2022.

Builds on

39 of 49 references.

References · 39

  1. 2015Deep Residual Learning for Image Recognitioncited by 51,809
  2. 2018Neural Tangent Kernel: Convergence and Generalization in Neural Networkscited by 2,597
  3. 2016On Large-Batch Training for Deep Learning: Generalization Gap and Sharp Minimacited by 1,337
  4. 2016Optimization Methods for Large-Scale Machine Learningcited by 1,140
  5. 2017Visualizing the Loss Landscape of Neural Netscited by 1,126
  6. 2017Computing Nonvacuous Generalization Bounds for Deep (Stochastic) Neural Networks with Many More Parameters than Training Datacited by 527
  7. 2017Sharp Minima Can Generalize For Deep Netscited by 462
  8. 2021Gradient Descent on Neural Networks Typically Occurs at the Edge of Stabilitycited by 413
  9. 2016Entropy-SGD: Biasing Gradient Descent Into Wide Valleyscited by 357
  10. 2017Three Factors Influencing Minima in SGDcited by 270
  11. 2020The large learning rate phase of deep learning: the catapult mechanismcited by 206
  12. 2014Qualitatively characterizing neural network optimization problemscited by 186
27 more
  1. 2021On the Origin of Implicit Regularization in Stochastic Gradient Descentcited by 183
  2. 2019Towards Explaining the Regularization Effect of Initial Large Learning Rate in Training Neural Networkscited by 162
  3. 2020Implicit Gradient Regularizationcited by 145
  4. 2018Fast and Faster Convergence of SGD for Over-Parameterized Models and an Accelerated Perceptroncited by 143
  5. 2022Understanding Gradient Descent on Edge of Stability in Deep Learningcited by 126
  6. 2020The Heavy-Tail Phenomenon in SGDcited by 122
  7. 2020Implicit Regularization in Deep Learning May Not Be Explainable by Normscited by 118
  8. 2018An Alternative View: When Does SGD Escape Local Minima?cited by 115
  9. 2021Implicit Bias of SGD for Diagonal Linear Networks: a Provable Benefit of Stochasticitycited by 107
  10. 2017Spurious Local Minima are Common in Two-Layer ReLU Neural Networkscited by 104
  11. 2018On the Relation Between the Sharpest Directions of DNN Loss and the SGD Step Lengthcited by 85
  12. 2017Gradient Descent Can Take Exponential Time to Escape Saddle Pointscited by 74
  13. 2018Escaping Saddles with Stochastic Gradientscited by 66
  14. 2017Implicit Regularization in Nonconvex Statistical Estimation: Gradient Descent Converges Linearly for Phase Retrieval, Matrix Completion, and Blind Deconvolutioncited by 63
  15. 2019Normalized Flat Minima: Exploring Scale Invariant Definition of Flat Minima for Neural Networks using PAC-Bayesian Analysiscited by 58
  16. 2021Large Learning Rate Tames Homogeneity: Convergence and Balancing Effectcited by 48
  17. 2022Anticorrelated Noise Injection for Improved Generalizationcited by 41
  18. 2019Sharp Analysis for Nonconvex SGD Escaping from Saddle Pointscited by 40
  19. 2019Near-Optimal Methods for Minimizing Star-Convex Functions and Beyondcited by 36
  20. 2021Continuous vs. Discrete Optimization of Deep Neural Networkscited by 34
  21. 2020The Effects of Mild Over-parameterization on the Optimization Landscape of Shallow ReLU Neural Networkscited by 33
  22. 2021Analyzing Monotonic Linear Interpolation in Neural Network Loss Landscapescited by 27
  23. 2016Gradient Descent Converges to Minimizerscited by 25
  24. 2016Local minima in training of neural networkscited by 19
  25. 2020Multiplicative noise and heavy tails in stochastic optimizationcited by 15
  26. 2021Eliminating Sharp Minima from SGD with Truncated Heavy-tailed Noisecited by 13
  27. 2022Tackling benign nonconvexity with smoothing and stochastic gradientscited by 2

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 1; 2024: 3; 2025: 5; 2026: 1.

10 papers.

  1. 2024Neural Redshift: Random Networks are not Random Functionscited by 43
  2. 2025SFT Doesn't Always Hurt General Capabilities: Revisiting Domain-Specific Fine-Tuning in LLMscited by 23
  3. 2024Towards Understanding Inductive Bias in Transformers: A View From Infinitycited by 13
  4. 2024Where Do Large Learning Rates Lead Us?cited by 6
  5. 2025Glocal Smoothness: Line search and adaptive step sizes can help in theory too!cited by 2
  6. 2025Large Learning Rates Simultaneously Achieve Robustness to Spurious Correlations and Compressibilitycited by 2
  7. 2026Fixed Aggregation Features Can Rival GNNscited by 2
  8. 2023The Disharmony between BN and ReLU Causes Gradient Explosion, but is Offset by the Correlation between Activationscited by 1
  9. 2025Feature Learning Beyond the Edge of Stabilitycited by 0
  10. 2025Non-Singularity of the Gradient Descent map for Neural Networks with Piecewise Analytic Activationscited by 0