mlsanityData to 7 Oct 2026

Learning an Adaptive Learning Rate Schedule

arXiv 1909.09712 ↗First posted 20 Sep 2019cs.LG stat.MLcited by 29

Not on the map. Why?

Abstract (arXiv)

The learning rate is one of the most important hyper-parameters for model training and generalization. However, current hand-designed parametric learning rate schedules offer limited flexibility and the predefined schedule may not match the training dynamics of high dimensional and non-convex optimization problems. In this paper, we propose a reinforcement learning based framework that can automatically learn an adaptive learning rate schedule by leveraging the information from past training histories. The learning rate dynamically changes based on the current training dynamics. To validate this framework, we conduct experiments with different neural network architectures on the Fashion MINIST and CIFAR10 datasets. Experimental results show that the auto-learned learning rate controller can achieve better test results. In addition, the trained controller network is generalizable -- able to be trained on one data set and transferred to new problems.

Key facts. Cited by 29 papers in the map (data through 7 Oct 2026). First posted on arXiv 20 Sep 2019.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 5; 2022: 7; 2023: 4; 2024: 4; 2025: 5; 2026: 4.

29 papers.

Sort:

By area:

  1. 2021Learning to Optimize: A Primer and A Benchmarkcited by 206
  2. 2022Learning to Learn with Generative Models of Neural Network Checkpointscited by 85
  3. 2022VeLO: Training Versatile Learned Optimizers by Scaling Upcited by 57
  4. 2022AutoBalance: Optimized Loss Functions for Imbalanced Datacited by 49
  5. 2023Optimal Linear Decay Learning Rate Schedules and Further Refinementscited by 41
  6. 2022Automated Dynamic Algorithm Configurationcited by 37
  7. 2022Practical tradeoffs between memory, compute, and performance in learned optimizerscited by 32
  8. 2022A Closer Look at Learned Optimization: Stability, Robustness, and Inductive Biasescited by 30
  9. 2021A Generalizable Approach to Learning Optimizerscited by 23
  10. 2025A Multi-Power Law for Loss Curve Prediction Across Learning Rate Schedulescited by 22
  11. 2021Reinforced Attention for Few-Shot Learning and Beyondcited by 18
  12. 2024Dynamic Layer Tying for Parameter-Efficient Transformerscited by 10
17 more
  1. 2022A Simple Guard for Learned Optimizerscited by 9
  2. 2024Narrowing the Focus: Learned Optimizers for Pretrained Modelscited by 3
  3. 2025Controlling changes to attention logitscited by 3
  4. 2023Studying K-FAC Heuristics by Viewing Adam through a Second-Order Lenscited by 2
  5. 2021Efficient Meta Subspace Optimizationcited by 1
  6. 2023Generalisable Agents for Neural Network Optimisationcited by 1
  7. 2025Interactive Training: Feedback-Driven Neural Network Optimizationcited by 1
  8. 2026Reinforcement Learning for Micro-Level Claims Reservingcited by 1
  9. 2021To Raise or Not To Raise: The Autonomous Learning Rate Questioncited by 0
  10. 2023Intelligent gradient amplification for deep neural networkscited by 0
  11. 2024Neural Optimizer Equation, Decay Function, and Learning Rate Schedule Joint Evolutioncited by 0
  12. 2024Curriculum Is More Influential Than Haptic Information During Reinforcement Learning of Object Manipulation Against Gravitycited by 0
  13. 2025SparseDoctor: Towards Efficient Chat Doctor with Mixture of Experts Enhanced Large Language Modelscited by 0
  14. 2025Optimization as a Dynamical System: Generative Schedules from Latent ODEscited by 0
  15. 2026Meta-Cognitive Reinforcement Learning with Self-Doubt and Recoverycited by 0
  16. 2026POP: Prior-Fitted First-Order Optimization Policiescited by 0
  17. 2026SOLAR: A State-Driven Online Learning Rate Scheduler for LLM Pretrainingcited by 0