mlsanityData to 7 Oct 2026

Provable Super-Convergence with a Large Cyclical Learning Rate

arXiv 2102.10734 ↗First posted 22 Feb 2021cs.LGcited by 10

Not on the map. Why?

Abstract (arXiv)

Conventional wisdom dictates that learning rate should be in the stable regime so that gradient-based algorithms don't blow up. This letter introduces a simple scenario where an unstably large learning rate scheme leads to a super fast convergence, with the convergence rate depending only logarithmically on the condition number of the problem. Our scheme uses a Cyclical Learning Rate (CLR) where we periodically take one large unstable step and several small stable steps to compensate for the instability. These findings also help explain the empirical observations of [Smith and Topin, 2019] where they show that CLR with a large maximum learning rate can dramatically accelerate learning and lead to so-called "super-convergence". We prove that our scheme excels in the problems where Hessian exhibits a bimodal spectrum and the eigenvalues can be grouped into two clusters (small and large). The unstably large step is the key to enabling fast convergence over the small eigen-spectrum.

Key facts. Cited by 10 papers in the map (data through 7 Oct 2026). Builds on 21 of its 25 references that are in the map. First posted on arXiv 22 Feb 2021.

Builds on

21 of 25 references.

References · 21

  1. 2014Very Deep Convolutional Networks for Large-Scale Image Recognitioncited by 15,267
  2. 2016SGDR: Stochastic Gradient Descent with Warm Restartscited by 4,993
  3. 2018Neural Tangent Kernel: Convergence and Generalization in Neural Networkscited by 2,597
  4. 2018Averaging Weights Leads to Wider Optima and Better Generalizationcited by 1,194
  5. 2018Reconciling modern machine learning practice and the bias-variance trade-offcited by 919
  6. 2019Wide Neural Networks of Any Depth Evolve as Linear Models Under Gradient Descentcited by 732
  7. 2015Cyclical Learning Rates for Training Neural Networkscited by 627
  8. 2016Linear Convergence of Gradient and Proximal-Gradient Methods Under the Polyak-Lojasiewicz Conditioncited by 526
  9. 2017Empirical Analysis of the Hessian of Over-Parametrized Neural Networkscited by 283
  10. 2018Gradient Descent Happens in a Tiny Subspacecited by 208
  11. 2017Super-Convergence: Very Fast Training of Neural Networks Using Large Learning Ratescited by 196
  12. 2019Cyclical Annealing Schedule: A Simple Approach to Mitigating KL Vanishingcited by 191
9 more
  1. 2019Gradient Descent with Early Stopping is Provably Robust to Label Noise for Overparameterized Neural Networkscited by 158
  2. 2019Global Convergence of Policy Gradient Methods to (Almost) Locally Optimal Policiescited by 117
  3. 2018Overfitting or perfect fitting? Risk bounds for classification and regression rules that interpolatecited by 99
  4. 2019Generalization Guarantees for Neural Networks via Harnessing the Low-rank Structure of the Jacobiancited by 67
  5. 2018Escaping Saddles with Stochastic Gradientscited by 66
  6. 2019Hessian based analysis of SGD for Deep Nets: Dynamics and Generalizationcited by 37
  7. 2017Theory of Deep Learning III: explaining the non-overfitting puzzlecited by 32
  8. 2020The Two Regimes of Deep Network Trainingcited by 30
  9. 2019Neural Spectrum Alignment: Empirical Studycited by 6

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2022: 3; 2023: 2; 2024: 2; 2025: 2; 2026: 1.

10 papers.

  1. 2022Compute-Efficient Deep Learning: Algorithmic Trends and Opportunitiescited by 38
  2. 2023Provably Faster Gradient Descent via Long Stepscited by 20
  3. 2023From Stability to Chaos: Analyzing Gradient Descent Dynamics in Quadratic Regressioncited by 16
  4. 2022Chaotic Regularization and Heavy-Tailed Limits for Deterministic Gradient Descentcited by 9
  5. 2024Gradient Descent on Logistic Regression with Non-Separable Data and Large Step Sizescited by 9
  6. 2025Negative Stepsizes Make Gradient-Descent-Ascent Convergecited by 9
  7. 2026POSEIDON: Physics-Optimized Seismic Energy Inference and Detection Operating Networkcited by 3
  8. 2022When is Momentum Extragradient Optimal? A Polynomial-Based Analysiscited by 1
  9. 2025Gradient Descent with Provably Tuned Learning-rate Schedulescited by 1
  10. 2024Data-induced multiscale losses and efficient multirate gradient descent schemescited by 0