# Dynamic Learning Rate Scheduling based on Loss Changes Leads to Faster Convergence

arXiv: https://arxiv.org/abs/2512.14527 · HTML: https://mlsanity.com/paper/2512.14527/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 15 of its 25 references that are in the map.
- First posted on arXiv 16 Dec 2025.

## Builds on (most cited first, 15 of 15)

- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [Spider: A Large-Scale Human-Labeled Dataset for Complex and Cross-Domain Semantic Parsing and Text-to-SQL Task](https://mlsanity.com/paper/1809.08887/) (2018)
- [WizardLM: Empowering large pre-trained language models to follow complex instructions](https://mlsanity.com/paper/2304.12244/) (2023)
- [Seq2SQL: Generating Structured Queries from Natural Language using Reinforcement Learning](https://mlsanity.com/paper/1709.00103/) (2017)
- [Cyclical Learning Rates for Training Neural Networks](https://mlsanity.com/paper/1506.01186/) (2015)
- [Identifying and attacking the saddle point problem in high-dimensional non-convex optimization](https://mlsanity.com/paper/1406.2572/) (2014)
- [The Marginal Value of Adaptive Gradient Methods in Machine Learning](https://mlsanity.com/paper/1705.08292/) (2017)
- [Painless Stochastic Gradient: Interpolation, Line-Search, and Convergence Rates](https://mlsanity.com/paper/1905.09997/) (2019)
- [The CoT Collection: Improving Zero-shot and Few-shot Learning of Language Models via Chain-of-Thought Fine-Tuning](https://mlsanity.com/paper/2305.14045/) (2023)
- [An Empirical Study of Instruction-tuning Large Language Models in Chinese](https://mlsanity.com/paper/2310.07328/) (2023)
- [AutoLRS: Automatic Learning-Rate Schedule by Bayesian Optimization on the Fly](https://mlsanity.com/paper/2105.10762/) (2021)
- [Automated Learning Rate Scheduler for Large-batch Training](https://mlsanity.com/paper/2107.05855/) (2021)
- [AutoLR: An Evolutionary Approach to Learning Rate Policies](https://mlsanity.com/paper/2007.04223/) (2020)
- [LipschitzLR: Using theoretically computed adaptive learning rates for fast convergence](https://mlsanity.com/paper/1902.07399/) (2019)
- [Training Aware Sigmoidal Optimizer](https://mlsanity.com/paper/2102.08716/) (2021)

## Abstract (arXiv)

Despite significant advances in optimizers for training, most research works use common scheduler choices like Cosine or exponential decay. In this paper, we study GreedyLR, a novel scheduler that adaptively adjusts the learning rate during training based on the current loss. To validate the effectiveness of our proposed scheduler, we conduct experiments on several NLP, CV, and LLM tasks with up to 7B parameters, including both fine-tuning and pre-training experiments. The results show that our approach outperforms several state-of-the-art schedulers in terms of accuracy, speed, and convergence. We also provide a theoretical analysis of the GreedyLR algorithm, including a proof of convergence and derivation of the optimal scaling factor F that maximizes the convergence rate, along with experiments to show robustness of the algorithm to realistic noisy landscapes. Our scheduler is easy to implement, computationally efficient, and could be considered a good default scheduler for training.