mlsanityData to 7 Oct 2026

Acceleration via Fractal Learning Rate Schedules

arXiv 2103.01338 ↗First posted 1 Mar 2021cs.LG math.OC stat.MLcited by 13

Not on the map. Why?

Abstract (arXiv)

In practical applications of iterative first-order optimization, the learning rate schedule remains notoriously difficult to understand and expensive to tune. We demonstrate the presence of these subtleties even in the innocuous case when the objective is a convex quadratic. We reinterpret an iterative algorithm from the numerical analysis literature as what we call the Chebyshev learning rate schedule for accelerating vanilla gradient descent, and show that the problem of mitigating instability leads to a fractal ordering of step sizes. We provide some experiments to challenge conventional beliefs about stable learning rates in deep learning: the fractal schedule enables training to converge with locally unstable updates which make negative progress on the objective.

Key facts. Cited by 13 papers in the map (data through 7 Oct 2026). Builds on 44 of its 94 references that are in the map. First posted on arXiv 1 Mar 2021.

Builds on

44 of 94 references.

References · 44

  1. 2014Adam: A Method for Stochastic Optimizationcited by 36,960
  2. 2020Language Models are Few-Shot Learnerscited by 32,199
  3. 2016SGDR: Stochastic Gradient Descent with Warm Restartscited by 4,993
  4. 2016Identity Mappings in Deep Residual Networkscited by 1,960
  5. 2016On Large-Batch Training for Deep Learning: Generalization Gap and Sharp Minimacited by 1,337
  6. 2019Large Batch Optimization for Deep Learning: Training BERT in 76 minutescited by 671
  7. 2015Cyclical Learning Rates for Training Neural Networkscited by 627
  8. 2014Convex Optimization: Algorithms and Complexitycited by 569
  9. 2015Train faster, generalize better: Stability of stochastic gradient descentcited by 547
  10. 2021Gradient Descent on Neural Networks Typically Occurs at the Edge of Stabilitycited by 413
  11. 2017Don't Decay the Learning Rate, Increase the Batch Sizecited by 313
  12. 2019Momentum-Based Variance Reduction in Non-Convex SGDcited by 302
32 more
  1. 2017How to Escape Saddle Points Efficientlycited by 284
  2. 2015A Differential Equation for Modeling Nesterov's Accelerated Gradient Method: Theory and Insightscited by 203
  3. 2019Cyclical Annealing Schedule: A Simple Approach to Mitigating KL Vanishingcited by 191
  4. 2018Measuring the Effects of Data Parallelism on Neural Network Trainingcited by 190
  5. 2018AdaGrad stepsizes: Sharp convergence over nonconvex landscapescited by 182
  6. 2019Towards Explaining the Regularization Effect of Initial Large Learning Rate in Training Neural Networkscited by 162
  7. 2015Stochastic modified equations and adaptive stochastic gradient algorithmscited by 141
  8. 2018Black-Box Reductions for Parameter-free Online Learning in Banach Spacescited by 124
  9. 2019An Exponential Learning Rate Schedule for Deep Learningcited by 121
  10. 2019Which Algorithmic Choices Matter at Which Batch Sizes? Insights From a Noisy Quadratic Modelcited by 116
  11. 2014Analysis and Design of Optimization Algorithms via Integral Quadratic Constraintscited by 105
  12. 2020Characterizing Structural Regularities of Labeled Data in Overparameterized Modelscited by 93
  13. 2019A Unified Theory of SGD: Variance Reduction, Sampling, Quantization and Coordinate Descentcited by 84
  14. 2017Neural Optimizer Search with Reinforcement Learningcited by 82
  15. 2019The Step Decay Schedule: A Near Optimal, Geometrically Decaying Learning Rate Procedure For Least Squarescited by 67
  16. 2020Reconciling Modern Deep Learning with Traditional Optimization Analyses: The Intrinsic Learning Ratecited by 59
  17. 2014Linear Coupling: An Ultimate Unification of Gradient and Mirror Descentcited by 58
  18. 2019Revisiting the Polyak step sizecited by 55
  19. 2018Stability and Convergence Trade-off of Iterative Optimization Algorithmscited by 53
  20. 2021Acceleration Methodscited by 48
  21. 2017Catalyst Acceleration for First-order Convex Optimization: from Theory to Practicecited by 39
  22. 2020Disentangling Adaptive Gradient Methods from Learning Ratescited by 39
  23. 2019Escaping Saddle Points with Adaptive Gradient Methodscited by 34
  24. 2015A geometric alternative to Nesterov's accelerated gradient descentcited by 21
  25. 2016Optimal Black-Box Reductions Between Optimization Objectivescited by 21
  26. 2019Faster Neural Network Training with Data Echoingcited by 19
  27. 2018Near-optimal method for highly smooth convex optimizationcited by 16
  28. 2019At Stability's Edge: How to Adjust Hyperparameters to Preserve Minima Selection in Asynchronous Training of Neural Networks?cited by 16
  29. 2020Complexity Guarantees for Polyak Steps with Momentumcited by 7
  30. 2020Stochastic Optimization with Laggard Data Pipelinescited by 7
  31. 2015On Accelerated Methods in Optimizationcited by 5
  32. 2018A Fast Anderson-Chebyshev Acceleration for Nonlinear Optimizationcited by 5

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 1; 2022: 5; 2023: 2; 2024: 2; 2025: 1; 2026: 2.

13 papers.

Sort:

  1. 2022SGD with Large Step Sizes Learns Sparse Featurescited by 66
  2. 2021Fractal Structure and Generalization Properties of Stochastic Optimization Algorithmscited by 23
  3. 2023Provably Faster Gradient Descent via Long Stepscited by 20
  4. 2023From Stability to Chaos: Analyzing Gradient Descent Dynamics in Quadratic Regressioncited by 16
  5. 2022The Curse of Unrolling: Rate of Differentiating Through Optimizationcited by 15
  6. 2022Chaotic Regularization and Heavy-Tailed Limits for Deterministic Gradient Descentcited by 9
  7. 2025Negative Stepsizes Make Gradient-Descent-Ascent Convergecited by 9
  8. 2022Adaptive Gradient Methods with Local Guaranteescited by 8
  9. 2022Accelerating Hamiltonian Monte Carlo via Chebyshev Integration Timecited by 7
  10. 2024Understanding the Generalization Benefits of Late Learning Rate Decaycited by 6
  11. 2024Dynamical stability and chaos in artificial neural network trajectories along trainingcited by 6
  12. 2026A lower bound for stepsize-based acceleration of gradient descentcited by 4
1 more
  1. 2026Improved Gradient Descent Lower Bounds Beyond Nesterovcited by 2