mlsanityData to 14 Oct 2026

An Automatic Learning Rate Schedule Algorithm for Achieving Faster Convergence and Steeper Descent

arXiv 2310.11291 ↗First posted 17 Oct 2023cs.LGcited by 9

Not on the map. Why?

Abstract (arXiv)

The delta-bar-delta algorithm is recognized as a learning rate adaptation technique that enhances the convergence speed of the training process in optimization by dynamically scheduling the learning rate based on the difference between the current and previous weight updates. While this algorithm has demonstrated strong competitiveness in full data optimization when compared to other state-of-the-art algorithms like Adam and SGD, it may encounter convergence issues in mini-batch optimization scenarios due to the presence of noisy gradients. In this study, we thoroughly investigate the convergence behavior of the delta-bar-delta algorithm in real-world neural network optimization. To address any potential convergence challenges, we propose a novel approach called RDBD (Regrettable Delta-Bar-Delta). Our approach allows for prompt correction of biased learning rate adjustments and ensures the convergence of the optimization process. Furthermore, we demonstrate that RDBD can be seamlessly integrated with any optimization algorithm and significantly improve the convergence speed. By conducting extensive experiments and evaluations, we validate the effectiveness and efficiency of our proposed RDBD approach. The results showcase its capability to overcome convergence issues in mini-batch optimization and its potential to enhance the convergence speed of various optimization algorithms. This research contributes to the advancement of optimization techniques in neural network training, providing practitioners with a reliable automatic learning rate scheduler for achieving faster convergence and improved optimization outcomes.

Key facts. Cited by 9 papers in the map (data through 14 Oct 2026). Builds on 73 of its 106 references that are in the map. First posted on arXiv 17 Oct 2023.

Builds on

73 of 106 references.

References · 73

  1. 2014Adam: A Method for Stochastic Optimizationcited by 37,240
  2. 2019PyTorch: An Imperative Style, High-Performance Deep Learning Librarycited by 20,305
  3. 2017Model-Agnostic Meta-Learning for Fast Adaptation of Deep Networkscited by 4,897
  4. 2020Meta-Learning in Neural Networks: A Surveycited by 950
  5. 2019On the Convergence of Adam and Beyondcited by 841
  6. 2018A Convergence Theory for Deep Learning via Over-Parameterizationcited by 787
  7. 2019On the Variance of the Adaptive Learning Rate and Beyondcited by 772
  8. 2018Gradient Descent Provably Optimizes Over-parameterized Neural Networkscited by 659
  9. 2019On Exact Computation with an Infinitely Wide Neural Netcited by 573
  10. 2019Fine-Grained Analysis of Optimization and Generalization for Overparameterized Two-Layer Neural Networkscited by 566
  11. 2023Symbolic Discovery of Optimization Algorithmscited by 434
  12. 2019Why gradient clipping accelerates training: A theoretical justification for adaptivitycited by 341
38 more
  1. 2017Don't Decay the Learning Rate, Increase the Batch Sizecited by 320
  2. 2017Three Factors Influencing Minima in SGDcited by 273
  3. 2018Learning Overparameterized Neural Networks via Stochastic Gradient Descent on Structured Datacited by 261
  4. 2018Meta-Learning: A Surveycited by 256
  5. 2019Generalization Bounds of Stochastic Gradient Descent for Wide and Deep Neural Networkscited by 222
  6. 2019Adaptive Gradient Methods with Dynamic Bound of Learning Ratecited by 183
  7. 2019Towards moderate overparameterization: global convergence guarantees for training shallow neural networkscited by 176
  8. 2019Towards Explaining the Regularization Effect of Initial Large Learning Rate in Training Neural Networkscited by 162
  9. 2018On the Convergence of Stochastic Gradient Descent with Adaptive Stepsizescited by 156
  10. 2020The Break-Even Point on Optimization Trajectories of Deep Neural Networkscited by 150
  11. 2016Stochastic Variance Reduction for Nonconvex Optimizationcited by 145
  12. 2016Katyusha: The First Direct Acceleration of Stochastic Gradient Methodscited by 135
  13. 2019An Improved Analysis of Training Over-parameterized Deep Neural Networkscited by 121
  14. 2019Polylogarithmic width suffices for gradient descent to achieve arbitrarily small test error with shallow ReLU networkscited by 121
  15. 2019An Exponential Learning Rate Schedule for Deep Learningcited by 121
  16. 2017Online Learning Rate Adaptation with Hypergradient Descentcited by 118
  17. 2023Learning-Rate-Free Learning by D-Adaptationcited by 113
  18. 2023A Tutorial on Meta-Reinforcement Learningcited by 108
  19. 2017Recovery Guarantees for One-hidden-layer Neural Networkscited by 94
  20. 2018On the Convergence Rate of Training Recurrent Neural Networkscited by 82
  21. 2020On the Generalization Benefit of Noise in Stochastic Gradient Descentcited by 82
  22. 2019Fast Convergence of Natural Gradient Descent for Overparameterized Neural Networkscited by 76
  23. 2019The Step Decay Schedule: A Near Optimal, Geometrically Decaying Learning Rate Procedure For Least Squarescited by 68
  24. 2021Training Multi-Layer Over-Parametrized Neural Network in Subquadratic Timecited by 62
  25. 2019Quadratic Suffices for Over-parametrization via Matrix Chernoff Boundcited by 59
  26. 2020Training (Overparametrized) Neural Networks in Near-Linear Timecited by 58
  27. 2019Time Matters in Regularizing Deep Networks: Weight Decay and Data Augmentation Affect Early Learning Dynamics, Matter Little Near Convergencecited by 54
  28. 2023Attention Scheme Inspired Softmax Regressioncited by 49
  29. 2023An Over-parameterized Exponential Regressioncited by 45
  30. 2023Solving Regularized Exp, Cosh and Sinh Regression Problemscited by 44
  31. 2023The Closeness of In-Context Learning and Weight Shifting for Softmax Regressioncited by 41
  32. 2019Gram-Gauss-Newton Method: Learning Overparameterized Neural Networks for Regression Problemscited by 40
  33. 2021Strength of Minibatch Noise in SGDcited by 39
  34. 2020Over-parameterized Adversarial Training: An Analysis Overcoming the Curse of Dimensionalitycited by 36
  35. 2021Does Preprocessing Help Training Over-parameterized Neural Networks?cited by 32
  36. 2023In-Context Learning for Attention Scheme: from Single Softmax Regression to Multiple Softmax Regression via a Tensor Trickcited by 32
  37. 2023InfoPrompt: Information-Theoretic Soft Prompt Tuning for Natural Language Understandingcited by 30
  38. 2023A Fast Optimization View: Reformulating Single Layer Attention in LLM Based on Tensor and SVM Trick, and Solving It in Matrix Multiplication Timecited by 30

50 most cited of 73 All 73 (JSON)

Built on by

Cited by year

Citing papers per year: 2023: 1; 2024: 3; 2025: 5.

9 papers.

  1. 2024Circuit Complexity Bounds for RoPE-based Transformer Architecturecited by 26
  2. 2024The Computational Limits of State-Space Models and Mamba via the Lens of Circuit Complexitycited by 21
  3. 2024Theoretical Constraints on the Expressive Power of mathsfRoPE-based Tensor Attention Transformerscited by 12
  4. 2025Theoretical Guarantees for High Order Trajectory Refinement in Generative Flowscited by 10
  5. 2023A Theoretical Insight into Attack and Defense of Gradient Leakage in Transformercited by 6
  6. 2025Neural Algorithmic Reasoning for Hypergraphs with Looped Transformerscited by 4
  7. 2025Scaling Law Phenomena Across Regression Paradigms: Multiple and Kernel Approachescited by 4
  8. 2025Time and Memory Trade-off of KV-Cache Compression in Tensor Transformer Decodingcited by 2
  9. 2025Theoretical Foundation of Flow-Based Time Series Generation: Provable Approximation, Generalization, and Efficiencycited by 2