# Learning an Adaptive Learning Rate Schedule

arXiv: https://arxiv.org/abs/1909.09712 · HTML: https://mlsanity.com/paper/1909.09712/

## Key facts

- Cited by 29 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 20 Sep 2019.

## Built on by (most cited first, 29 of 29)

- [Learning to Optimize: A Primer and A Benchmark](https://mlsanity.com/paper/2103.12828/) (2021)
- [Learning to Learn with Generative Models of Neural Network Checkpoints](https://mlsanity.com/paper/2209.12892/) (2022)
- [VeLO: Training Versatile Learned Optimizers by Scaling Up](https://mlsanity.com/paper/2211.09760/) (2022)
- [AutoBalance: Optimized Loss Functions for Imbalanced Data](https://mlsanity.com/paper/2201.01212/) (2022)
- [Optimal Linear Decay Learning Rate Schedules and Further Refinements](https://mlsanity.com/paper/2310.07831/) (2023)
- [Automated Dynamic Algorithm Configuration](https://mlsanity.com/paper/2205.13881/) (2022)
- [Practical tradeoffs between memory, compute, and performance in learned optimizers](https://mlsanity.com/paper/2203.11860/) (2022)
- [A Closer Look at Learned Optimization: Stability, Robustness, and Inductive Biases](https://mlsanity.com/paper/2209.11208/) (2022)
- [A Generalizable Approach to Learning Optimizers](https://mlsanity.com/paper/2106.00958/) (2021)
- [A Multi-Power Law for Loss Curve Prediction Across Learning Rate Schedules](https://mlsanity.com/paper/2503.12811/) (2025)
- [Reinforced Attention for Few-Shot Learning and Beyond](https://mlsanity.com/paper/2104.04192/) (2021)
- [Dynamic Layer Tying for Parameter-Efficient Transformers](https://mlsanity.com/paper/2401.12819/) (2024)
- [A Simple Guard for Learned Optimizers](https://mlsanity.com/paper/2201.12426/) (2022)
- [Narrowing the Focus: Learned Optimizers for Pretrained Models](https://mlsanity.com/paper/2408.09310/) (2024)
- [Controlling changes to attention logits](https://mlsanity.com/paper/2511.21377/) (2025)
- [Studying K-FAC Heuristics by Viewing Adam through a Second-Order Lens](https://mlsanity.com/paper/2310.14963/) (2023)
- [Efficient Meta Subspace Optimization](https://mlsanity.com/paper/2110.14920/) (2021)
- [Generalisable Agents for Neural Network Optimisation](https://mlsanity.com/paper/2311.18598/) (2023)
- [Interactive Training: Feedback-Driven Neural Network Optimization](https://mlsanity.com/paper/2510.02297/) (2025)
- [Reinforcement Learning for Micro-Level Claims Reserving](https://mlsanity.com/paper/2601.07637/) (2026)
- [To Raise or Not To Raise: The Autonomous Learning Rate Question](https://mlsanity.com/paper/2106.08767/) (2021)
- [Intelligent gradient amplification for deep neural networks](https://mlsanity.com/paper/2305.18445/) (2023)
- [Neural Optimizer Equation, Decay Function, and Learning Rate Schedule Joint Evolution](https://mlsanity.com/paper/2404.06679/) (2024)
- [Curriculum Is More Influential Than Haptic Information During Reinforcement Learning of Object Manipulation Against Gravity](https://mlsanity.com/paper/2407.09986/) (2024)
- [SparseDoctor: Towards Efficient Chat Doctor with Mixture of Experts Enhanced Large Language Models](https://mlsanity.com/paper/2509.14269/) (2025)
- [Optimization as a Dynamical System: Generative Schedules from Latent ODEs](https://mlsanity.com/paper/2509.23052/) (2025)
- [Meta-Cognitive Reinforcement Learning with Self-Doubt and Recovery](https://mlsanity.com/paper/2601.20193/) (2026)
- [POP: Prior-Fitted First-Order Optimization Policies](https://mlsanity.com/paper/2602.15473/) (2026)
- [SOLAR: A State-Driven Online Learning Rate Scheduler for LLM Pretraining](https://mlsanity.com/paper/2609.34681/) (2026)

## Abstract (arXiv)

The learning rate is one of the most important hyper-parameters for model training and generalization. However, current hand-designed parametric learning rate schedules offer limited flexibility and the predefined schedule may not match the training dynamics of high dimensional and non-convex optimization problems. In this paper, we propose a reinforcement learning based framework that can automatically learn an adaptive learning rate schedule by leveraging the information from past training histories. The learning rate dynamically changes based on the current training dynamics. To validate this framework, we conduct experiments with different neural network architectures on the Fashion MINIST and CIFAR10 datasets. Experimental results show that the auto-learned learning rate controller can achieve better test results. In addition, the trained controller network is generalizable -- able to be trained on one data set and transferred to new problems.