mlsanityData to 7 Oct 2026

Maximal Initial Learning Rates in Deep ReLU Networks

arXiv 2212.07295 ↗First posted 14 Dec 2022stat.ML cs.LGcited by 11

Not on the map. Why?

Abstract (arXiv)

Training a neural network requires choosing a suitable learning rate, which involves a trade-off between speed and effectiveness of convergence. While there has been considerable theoretical and empirical analysis of how large the learning rate can be, most prior work focuses only on late-stage training. In this work, we introduce the maximal initial learning rate ηast - the largest learning rate at which a randomly initialized neural network can successfully begin training and achieve (at least) a given threshold accuracy. Using a simple approach to estimate ηast, we observe that in constant-width fully-connected ReLU networks, ηast behaves differently from the maximum learning rate later in training. Specifically, we find that ηast is well predicted as a power of depth × width, provided that (i) the width of the network is sufficiently large compared to the depth, and (ii) the input layer is trained at a relatively small learning rate. We further analyze the relationship between ηast and the sharpness λ1 of the network at initialization, indicating they are closely though not inversely related. We formally prove bounds for λ1 in terms of depth × width that align with our empirical results.

Key facts. Cited by 11 papers in the map (data through 7 Oct 2026). Builds on 33 of its 44 references that are in the map. First posted on arXiv 14 Dec 2022.

Builds on

33 of 44 references.

References · 33

  1. 2015Deep Residual Learning for Image Recognitioncited by 51,809
  2. 2015Delving Deep into Rectifiers: Surpassing Human-Level Performance on ImageNet Classificationcited by 3,590
  3. 2017Accurate, Large Minibatch SGD: Training ImageNet in 1 Hourcited by 1,479
  4. 2021Gradient Descent on Neural Networks Typically Occurs at the Edge of Stabilitycited by 413
  5. 2017Don't Decay the Learning Rate, Increase the Batch Sizecited by 313
  6. 2016On the Expressive Power of Deep Neural Networkscited by 302
  7. 2019PyHessian: Neural Networks Through the Lens of the Hessiancited by 291
  8. 2017Train longer, generalize better: closing the generalization gap in large batch training of neural networkscited by 278
  9. 2017Three Factors Influencing Minima in SGDcited by 270
  10. 2020The large learning rate phase of deep learning: the catapult mechanismcited by 206
  11. 2017Super-Convergence: Very Fast Training of Neural Networks Using Large Learning Ratescited by 196
  12. 2021On the Origin of Implicit Regularization in Stochastic Gradient Descentcited by 183
21 more
  1. 2021The Principles of Deep Learning Theorycited by 169
  2. 2019Complexity of Linear Regions in Deep Networkscited by 166
  3. 2019Towards Explaining the Regularization Effect of Initial Large Learning Rate in Training Neural Networkscited by 162
  4. 2019Deep ReLU Networks Have Surprisingly Few Activation Patternscited by 160
  5. 2020The Break-Even Point on Optimization Trajectories of Deep Neural Networkscited by 149
  6. 2022Understanding Gradient Descent on Edge of Stability in Deep Learningcited by 126
  7. 2019Finite Depth and Width Corrections to the Neural Tangent Kernelcited by 114
  8. 2020The Early Phase of Neural Network Trainingcited by 114
  9. 2018How to Start Training: The Effect of Initialization and Architecturecited by 109
  10. 2018Which Neural Net Architectures Give Rise To Exploding and Vanishing Gradients?cited by 94
  11. 2018Universal Statistics of Fisher Information in Deep Neural Networks: Mean Field Approachcited by 89
  12. 2022Adaptive Gradient Methods at the Edge of Stabilitycited by 88
  13. 2022Understanding the unstable convergence of gradient descentcited by 84
  14. 2017Failures of Gradient-Based Deep Learningcited by 79
  15. 2022Analyzing Sharpness along GD Trajectory: Progressive Sharpening and Edge of Stabilitycited by 61
  16. 2020Reconciling Modern Deep Learning with Traditional Optimization Analyses: The Intrinsic Learning Ratecited by 59
  17. 2018Products of Many Large Random Matrices and Gradients in Deep Neural Networkscited by 49
  18. 2021Large Learning Rate Tames Homogeneity: Convergence and Balancing Effectcited by 48
  19. 2020On the infinite width limit of neural networks with a standard parameterizationcited by 38
  20. 2021Deep ReLU Networks Preserve Expected Lengthcited by 17
  21. 2019The Effect of Network Width on Stochastic Gradient Descent and Generalization: an Empirical Studycited by 16

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 3; 2024: 3; 2025: 3; 2026: 2.

11 papers.

  1. 2024Super Consistency of Neural Network Landscapes and Learning Rate Transfercited by 23
  2. 2023Phase diagram of early training dynamics in deep neural networks: effect of the learning rate, depth, and widthcited by 19
  3. 2023On the Parameterization of Second-Order Optimization Effective Towards the Infinite Widthcited by 14
  4. 2025The Butterfly Effect: Neural Network Training Trajectories Are Highly Sensitive to Initial Conditionscited by 9
  5. 2023Catapult Dynamics and Phase Transitions in Quadratic Netscited by 8
  6. 2024Principled Architecture-aware Scaling of Hyperparameterscited by 4
  7. 2024ExpTest: Loss-Curve Hypothesis Testing for Autonomous Learning-Rate Selection in Deep Neural Networkscited by 0
  8. 2025Enlightenment Period Improving DNN Performancecited by 0
  9. 2025Beyond Gaussian Initializations: Signal Preserving Weight Initialization for Odd-Sigmoid Activationscited by 0
  10. 2026A Journey to the Edge of Stabilitycited by 0
  11. 2026Convergence of Practical Muon with Finite Newton-Schulz Iterations and Nesterov Momentumcited by 0