mlsanityData to 7 Oct 2026

Online Learning Rate Adaptation with Hypergradient Descent

arXiv 1703.04782 ↗First posted 14 Mar 2017cs.LG stat.MLcited by 117

Not on the map. Why?

Abstract (arXiv)

We introduce a general method for improving the convergence rate of gradient-based optimizers that is easy to implement and works well in practice. We demonstrate the effectiveness of the method in a range of optimization problems by applying it to stochastic gradient descent, stochastic gradient descent with Nesterov momentum, and Adam, showing that it significantly reduces the need for the manual tuning of the initial learning rate for these commonly used algorithms. Our method works by dynamically updating the learning rate during optimization using the gradient with respect to the learning rate of the update rule itself. Computing this "hypergradient" needs little additional computation, requires only one extra copy of the original gradient to be stored in memory, and relies upon nothing more than what is provided by reverse-mode automatic differentiation.

Key facts. Cited by 117 papers in the map (data through 7 Oct 2026). First posted on arXiv 14 Mar 2017.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 22; 2022: 23; 2023: 24; 2024: 18; 2025: 16; 2026: 14.

117 papers.

Sort:

By area:

  1. 2021Graph Contrastive Learning Automatedcited by 287
  2. 2021Investigating Bi-Level Optimization for Learning and Vision from a Unified Perspective: A Survey and Beyondcited by 151
  3. 2022Learning to Learn with Generative Models of Neural Network Checkpointscited by 85
  4. 2021BERT Learns to Teach: Knowledge Distillation with Meta Learningcited by 61
  5. 2021Unbiased Gradient Estimation in Unrolled Computation Graphs with Persistent Evolution Strategiescited by 52
  6. 2023Optimal Linear Decay Learning Rate Schedules and Further Refinementscited by 41
  7. 2022Automated Dynamic Algorithm Configurationcited by 37
  8. 2022DDG-DA: Data Distribution Generation for Predictable Concept Drift Adaptationcited by 36
  9. 2022A Closer Look at Learned Optimization: Stability, Robustness, and Inductive Biasescited by 30
  10. 2022DLTTA: Dynamic Learning Rate for Test-time Adaptation on Cross-domain Medical Imagescited by 28
  11. 2022Betty: An Automatic Differentiation Library for Multilevel Optimizationcited by 27
  12. 2024B-STaR: Monitoring and Balancing Exploration and Exploitation in Self-Taught Reasonerscited by 26
38 more
  1. 2021Meta Propagation Networks for Graph Few-shot Semi-supervised Learningcited by 25
  2. 2023Meta-learning approaches for few-shot learning: A survey of recent advancescited by 25
  3. 2021A Generalizable Approach to Learning Optimizerscited by 23
  4. 2023Learning How to Propagate Messages in Graph Neural Networkscited by 23
  5. 2023Fast Trainable Projection for Robust Fine-Tuningcited by 18
  6. 2021Meta Back-translationcited by 16
  7. 2021Doubly Adaptive Scaled Algorithm for Machine Learning Using Second-Order Informationcited by 16
  8. 2023Don't be so Monotone: Relaxing Stochastic Line Search in Over-Parameterized Modelscited by 15
  9. 2021AutoLRS: Automatic Learning-Rate Schedule by Bayesian Optimization on the Flycited by 14
  10. 2021Automated Deep Learning: Neural Architecture Search Is Not the Endcited by 13
  11. 2023Towards Constituting Mathematical Structures for Learning to Optimizecited by 13
  12. 2024Tuning-Free Stochastic Optimizationcited by 13
  13. 2024FIT-RAG: Black-Box RAG with Factual Information and Token Reductioncited by 13
  14. 2021A Theoretical Understanding of Gradient Bias in Meta-Reinforcement Learningcited by 11
  15. 2022Selecting and Composing Learning Rate Policies for Deep Neural Networkscited by 11
  16. 2024Gradient Methods with Online Scalingcited by 11
  17. 2021Automated Learning Rate Scheduler for Large-batch Trainingcited by 10
  18. 2021Learning an Explicit Hyperparameter Prediction Function Conditioned on Taskscited by 9
  19. 2022Amortized Proximal Optimizationcited by 9
  20. 2022Grad-GradaGrad? A Non-Monotone Adaptive Stochastic Gradient Methodcited by 9
  21. 2022Learning Symbolic Model-Agnostic Loss Functions via Meta-Learningcited by 9
  22. 2023Optimal Sets and Solution Paths of ReLU Networkscited by 9
  23. 2023Searching for Optimal Per-Coordinate Step-sizes with Multidimensional Backtrackingcited by 9
  24. 2023An Automatic Learning Rate Schedule Algorithm for Achieving Faster Convergence and Steeper Descentcited by 9
  25. 2021Robust MAML: Prioritization task buffer with adaptive learning process for model-agnostic meta-learningcited by 8
  26. 2021AngularGrad: A New Optimization Technique for Angular Convergence of Convolutional Neural Networkscited by 8
  27. 2023FedHyper: A Universal and Robust Learning Rate Scheduler for Federated Learning with Hypergradient Descentcited by 8
  28. 2024Stepping on the Edge: Curvature Aware Learning Rate Tunerscited by 8
  29. 2021Scalable One-Pass Optimisation of High-Dimensional Weight-Update Hyperparameters by Implicit Differentiationcited by 7
  30. 2022Step-size Adaptation Using Exponentiated Gradient Updatescited by 7
  31. 2023Low-Variance Gradient Estimation in Unrolled Computation Graphs with ES-Singlecited by 7
  32. 2023QLABGrad: a Hyperparameter-Free and Convergence-Guaranteed Scheme for Deep Learningcited by 6
  33. 2023Contrastive Learning with Consistent Representationscited by 6
  34. 2024MADA: Meta-Adaptive Optimizers through hyper-gradient Descentcited by 6
  35. 2024MetaOptimize: A Framework for Optimizing Step Sizes and Other Meta-parameterscited by 5
  36. 2025Provable and Practical Online Learning Rate Adaptation with Hypergradient Descentcited by 5
  37. 2023A Nonstochastic Control Approach to Optimizationcited by 4
  38. 2023Meta-Learning Adaptive Loss Functionscited by 4

50 most cited of 117 All 117 (JSON)