mlsanityData to 7 Oct 2026

Large Learning Rate Tames Homogeneity: Convergence and Balancing Effect

arXiv 2110.03677 ↗First posted 7 Oct 2021cs.LG math.DS math.OCcited by 48

Not on the map. Why?

Abstract (arXiv)

Recent empirical advances show that training deep models with large learning rate often improves generalization performance. However, theoretical justifications on the benefits of large learning rate are highly limited, due to challenges in analysis. In this paper, we consider using Gradient Descent (GD) with a large learning rate on a homogeneous matrix factorization problem, i.e., minX, Y |A - XY^top|sf F2. We prove a convergence theory for constant large learning rates well beyond 2/L, where L is the largest eigenvalue of Hessian at the initialization. Moreover, we rigorously establish an implicit bias of GD induced by such a large learning rate, termed 'balancing', meaning that magnitudes of X and Y at the limit of GD iterations will be close even if their initialization is significantly unbalanced. Numerical experiments are provided to support our theory.

Key facts. Cited by 48 papers in the map (data through 7 Oct 2026). Builds on 32 of its 55 references that are in the map. First posted on arXiv 7 Oct 2021.

Builds on

32 of 55 references.

References · 32

  1. 2018Neural Tangent Kernel: Convergence and Generalization in Neural Networkscited by 2,597
  2. 2021Convex optimizationcited by 2,479
  3. 2016Optimization Methods for Large-Scale Machine Learningcited by 1,140
  4. 2019On the Variance of the Adaptive Learning Rate and Beyondcited by 763
  5. 2017The Implicit Bias of Gradient Descent on Separable Datacited by 649
  6. 2015Cyclical Learning Rates for Training Neural Networkscited by 627
  7. 2014In Search of the Real Inductive Bias: On the Role of Implicit Regularization in Deep Learningcited by 339
  8. 2017Implicit Regularization in Matrix Factorizationcited by 318
  9. 2017Three Factors Influencing Minima in SGDcited by 270
  10. 2018A disciplined approach to neural network hyper-parameters: Part 1 -- learning rate, batch size, momentum, and weight decaycited by 223
  11. 2020The large learning rate phase of deep learning: the catapult mechanismcited by 206
  12. 2018Gradient descent aligns the layers of deep linear networkscited by 182
20 more
  1. 2018Algorithmic Regularization in Learning Deep Homogeneous Models: Layers are Automatically Balancedcited by 172
  2. 2019Towards Explaining the Regularization Effect of Initial Large Learning Rate in Training Neural Networkscited by 162
  3. 2017No Spurious Local Minima in Nonconvex Low Rank Problems: A Unified Geometric Analysiscited by 127
  4. 2020Towards Theoretically Understanding Why SGD Generalizes Better Than ADAM in Deep Learningcited by 125
  5. 2014Phase Retrieval via Wirtinger Flow: Theory and Algorithmscited by 123
  6. 2019An Exponential Learning Rate Schedule for Deep Learningcited by 121
  7. 2018A Closer Look at Deep Learning Heuristics: Learning rate restarts, Warmup and Distillationcited by 110
  8. 2015Low-rank Solutions of Linear Matrix Equations via Procrustes Flowcited by 97
  9. 2016A Geometric Analysis of Phase Retrievalcited by 92
  10. 2015Fast low-rank estimation by projected gradient descent: General statistical and algorithmic guaranteescited by 68
  11. 2021Global Convergence of Gradient Descent for Asymmetric Low-Rank Matrix Factorizationcited by 44
  12. 2015Dropping Convexity for Faster Semi-definite Optimizationcited by 40
  13. 2013Understanding Alternating Minimization for Matrix Completioncited by 32
  14. 2020Stochasticity of Deterministic Gradient Descent: Large Learning Rate for Multiscale Objective Functioncited by 26
  15. 2020A Comparison of Optimization Algorithms for Deep Learningcited by 23
  16. 2021Noisy Gradient Descent Converges to Flat Minima for Nonconvex Matrix Factorizationcited by 15
  17. 2020Learning Rate Annealing Can Provably Help Generalization, Even for Convex Problemscited by 11
  18. 2021Implicit Regularization of Bregman Proximal Point Algorithm and Mirror Descent on Separable Datacited by 6
  19. 2018On Landscape of Lagrangian Functions and Stochastic Search for Constrained Nonconvex Optimizationcited by 2
  20. 2020The Landscape of Matrix Factorization Revisitedcited by 1

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 1; 2022: 11; 2023: 13; 2024: 6; 2025: 11; 2026: 6.

48 papers.

Sort:

  1. 2022Understanding Gradient Descent on Edge of Stability in Deep Learningcited by 126
  2. 2022Understanding the Generalization Benefit of Normalization Layers: Sharpness Reductioncited by 86
  3. 2022On the Implicit Bias in Deep-Learning Algorithmscited by 84
  4. 2022SGD with Large Step Sizes Learns Sparse Featurescited by 66
  5. 2022Understanding Edge-of-Stability Training Dynamics with a Minimalist Examplecited by 54
  6. 2022Learning threshold neurons via the "edge of stability"cited by 50
  7. 2022Beyond the Quadratic Approximation: the Multiscale Structure of Neural Network Loss Landscapescited by 42
  8. 2023Implicit Bias of Gradient Descent for Logistic Regression at the Edge of Stabilitycited by 42
  9. 2023(S)GD over Diagonal Linear Networks: Implicit Regularisation, Large Stepsizes and Edge of Stabilitycited by 35
  10. 2024Large Stepsize Gradient Descent for Logistic Loss: Non-Monotonicity of the Loss Improves Optimization Efficiencycited by 34
  11. 2025Symmetry in Neural Network Parameter Spacescited by 31
  12. 2023Catapults in SGD: spikes in the training loss and their impact on generalization through feature learningcited by 29
36 more
  1. 2022Beyond spectral gap: The role of the topology in decentralized learningcited by 27
  2. 2023Good regularity creates large learning rate implicit biases: edge of stability, balancing, and catapultcited by 23
  3. 2021More is Less: Inducing Sparsity via Overparameterizationcited by 22
  4. 2023Universal Sharpness Dynamics in Neural Network Training: Fixed Point Analysis, Edge of Stability, and Route to Chaoscited by 21
  5. 2023Gradient Descent Monotonically Decreases the Sharpness of Gradient Flow Solutions in Scalar Networks and Beyondcited by 20
  6. 2024Large Stepsize Gradient Descent for Non-Homogeneous Two-Layer Networks: Margin Improvement and Fast Optimizationcited by 17
  7. 2023From Stability to Chaos: Analyzing Gradient Descent Dynamics in Quadratic Regressioncited by 16
  8. 2023How to escape sharp minima with random perturbationscited by 13
  9. 2023Convergence of Alternating Gradient Descent for Matrix Factorizationcited by 12
  10. 2022Maximal Initial Learning Rates in Deep ReLU Networkscited by 11
  11. 2022Special Properties of Gradient Descent with Large Learning Ratescited by 10
  12. 2022On the generalization of learning algorithms that do not convergecited by 9
  13. 2023Catapult Dynamics and Phase Transitions in Quadratic Netscited by 8
  14. 2023Noise Balance and Stationary Distribution of Stochastic Gradient Descentcited by 6
  15. 2024Understanding the Generalization Benefits of Late Learning Rate Decaycited by 6
  16. 2024Where Do Large Learning Rates Lead Us?cited by 6
  17. 2025Minimax Optimal Convergence of Gradient Descent in Logistic Regression via Large and Adaptive Stepsizescited by 5
  18. 2025Gradient Descent with Large Step Sizes: Chaos and Fractal Convergence Regioncited by 5
  19. 2025A Minimalist Example of Edge-of-Stability and Progressive Sharpeningcited by 4
  20. 2023On the Effect of Initialization: The Scaling Path of 2-Layer Neural Networkscited by 3
  21. 2024Provable Acceleration of Nesterov's Accelerated Gradient for Rectangular Matrix Factorization and Linear Neural Networkscited by 3
  22. 2025Convergence Rates for Gradient Descent on the Edge of Stability in Overparametrised Least Squarescited by 3
  23. 2025Gradient Descent Converges Linearly to Flatter Minima than Gradient Flow in Shallow Linear Networkscited by 2
  24. 2025Variational Learning Finds Flatter Solutions at the Edge of Stabilitycited by 2
  25. 2025Sharpness of Minima in Deep Matrix Factorizationcited by 2
  26. 2025Statistical Guarantees for High-Dimensional Stochastic Gradient Descentcited by 2
  27. 2025Any-stepsize Gradient Descent for Separable Data under Fenchel-Young Lossescited by 1
  28. 2026Does SGD Seek Flatness or Sharpness? An Exactly Solvable Modelcited by 1
  29. 2026Product-Stability: Provable Convergence for Gradient Descent on the Edge of Stabilitycited by 1
  30. 2026Dynamics of Gradient Descent with Large Step Size Near a Manifold of Flat Minimacited by 1
  31. 2023Implicit Balancing and Regularization: Generalization and Convergence Guarantees for Overparameterized Asymmetric Matrix Sensingcited by 0
  32. 2024Data-induced multiscale losses and efficient multirate gradient descent schemescited by 0
  33. 2025Data Uniformity Improves Training Efficiency and More, with a Convergence Framework Beyond the NTK Regimecited by 0
  34. 2026Centre manifold theorem for maps along manifolds of fixed pointscited by 0
  35. 2026Unbiased Gradients, Moving Stability Boundaries: Exact Mini-Batch Geometry in Linear Self-Attentioncited by 0
  36. 2026A Bifurcation Theory Framework for Gradient Descent on the Edge of Stabilitycited by 0