mlsanityData to 14 Oct 2026

Large Learning Rates Improve Generalization: But How Large Are We Talking About?

arXiv 2311.11303 ↗First posted 19 Nov 2023cs.LG stat.MLcited by 1

Not on the map. Why?

Abstract (arXiv)

Inspired by recent research that recommends starting neural networks training with large learning rates (LRs) to achieve the best generalization, we explore this hypothesis in detail. Our study clarifies the initial LR ranges that provide optimal results for subsequent training with a small LR or weight averaging. We find that these ranges are in fact significantly narrower than generally assumed. We conduct our main experiments in a simplified setup that allows precise control of the learning rate hyperparameter and validate our key findings in a more practical setting.

Key facts. Cited by 1 paper in the map (data through 14 Oct 2026). Builds on 14 of its 18 references that are in the map. First posted on arXiv 19 Nov 2023.

Builds on

14 of 18 references.

References · 14

  1. 2015Deep Residual Learning for Image Recognitioncited by 52,096
  2. 2018Averaging Weights Leads to Wider Optima and Better Generalizationcited by 1,209
  3. 2021The Role of Permutation Invariance in Linear Mode Connectivity of Neural Networkscited by 291
  4. 2020The large learning rate phase of deep learning: the catapult mechanismcited by 216
  5. 2021On the Origin of Implicit Regularization in Stochastic Gradient Descentcited by 184
  6. 2019Towards Explaining the Regularization Effect of Initial Large Learning Rate in Training Neural Networkscited by 162
  7. 2020Implicit Gradient Regularizationcited by 146
  8. 2022SGD with Large Step Sizes Learns Sparse Featurescited by 68
  9. 2018Theoretical Analysis of Auto Rate-Tuning by Batch Normalizationcited by 65
  10. 2020Reconciling Modern Deep Learning with Traditional Optimization Analyses: The Intrinsic Learning Ratecited by 59
  11. 2023Stochastic Collapse: How Gradient Noise Attracts SGD Dynamics Towards Simpler Subnetworkscited by 39
  12. 2020Wide-minima Density Hypothesis and the Explore-Exploit Learning Rate Schedulecited by 25
2 more
  1. 2021On the Periodic Behavior of Neural Network Training with Batch Normalization and Weight Decaycited by 20
  2. 2022Training Scale-Invariant Neural Networks on the Sphere Can Happen in Three Regimescited by 19

Built on by

Cited by year

Citing papers per year: 2024: 1.

1 paper.

  1. 2024KV Shifting Attention Enhances Language Modelingcited by 8