mlsanityData to 7 Oct 2026

MetaOptimize: A Framework for Optimizing Step Sizes and Other Meta-parameters

arXiv 2402.02342 ↗First posted 4 Feb 2024cs.LG cs.AI math.OCcited by 5

Not on the map. Why?

Abstract (arXiv)

We address the challenge of optimizing meta-parameters (hyperparameters) in machine learning, a key factor for efficient training and high model performance. Rather than relying on expensive meta-parameter search methods, we introduce MetaOptimize: a dynamic approach that adjusts meta-parameters, particularly step sizes (also known as learning rates), during training. More specifically, MetaOptimize can wrap around any first-order optimization algorithm, tuning step sizes on the fly to minimize a specific form of regret that considers the long-term impact of step sizes on training, through a discounted sum of future losses. We also introduce lower-complexity variants of MetaOptimize that, in conjunction with its adaptability to various optimization algorithms, achieve performance comparable to those of the best hand-crafted learning rate schedules across diverse machine learning tasks.

Key facts. Cited by 5 papers in the map (data through 7 Oct 2026). Builds on 31 of its 63 references that are in the map. First posted on arXiv 4 Feb 2024.

Builds on

31 of 63 references.

References · 31

  1. 2016Learning to learn by gradient descent by gradient descentcited by 616
  2. 2023TinyStories: How Small Can Language Models Be and Still Speak Coherent English?cited by 425
  3. 2023Symbolic Discovery of Optimization Algorithmscited by 423
  4. 2015Gradient-based Hyperparameter Optimization through Reversible Learningcited by 357
  5. 2019Optimizing Millions of Hyperparameters by Implicit Differentiationcited by 303
  6. 2016Hyperparameter optimization with approximate gradientcited by 231
  7. 2018Meta-Gradient Reinforcement Learningcited by 156
  8. 2020Stochastic Polyak Step-size for SGD: An Adaptive Learning Rate for Fast Convergencecited by 137
  9. 2016Coin Betting and Parameter-Free Online Learningcited by 131
  10. 2023Prodigy: An Expeditiously Adaptive Parameter-Free Learnercited by 129
  11. 2019Painless Stochastic Gradient: Interpolation, Line-Search, and Convergence Ratescited by 119
  12. 2017Online Learning Rate Adaptation with Hypergradient Descentcited by 117
19 more
  1. 2023Learning-Rate-Free Learning by D-Adaptationcited by 113
  2. 2023DoG is SGD's Best Friend: A Parameter-Free Dynamic Step Size Schedulecited by 85
  3. 2020Tasks, stability, architecture, and compute: Training more effective learned optimizers, and using them to train themselvescited by 57
  4. 2023Benchmarking Neural Network Training Algorithmscited by 46
  5. 2019Training Neural Networks for and by Interpolationcited by 45
  6. 2019The importance of better models in stochastic optimizationcited by 32
  7. 2022Practical tradeoffs between memory, compute, and performance in learned optimizerscited by 32
  8. 2018L4: Practical loss-based stepsize adaptation for deep learningcited by 26
  9. 2023Mechanic: A Learning Rate Tunercited by 20
  10. 2015Layer-Specific Adaptive Learning Rates for Deep Networkscited by 19
  11. 2020Gradient-based Hyperparameter Optimization Over Long Horizonscited by 12
  12. 2023Searching for Optimal Per-Coordinate Step-sizes with Multidimensional Backtrackingcited by 9
  13. 2022Step-size Adaptation Using Exponentiated Gradient Updatescited by 7
  14. 2022A History of Meta-gradient: Gradient Methods for Meta-learningcited by 7
  15. 2022Meta Mirror Descent: Optimiser Learning for Fast Convergencecited by 7
  16. 2024MADA: Meta-Adaptive Optimizers through hyper-gradient Descentcited by 6
  17. 2024Step-size Optimization for Continual Learningcited by 6
  18. 2018Metatrace Actor-Critic: Online Step-size Tuning by Meta-gradient Descent for Reinforcement Learning Controlcited by 5
  19. 2022Hyperparameter Importance for Machine Learning Algorithmscited by 4

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 2; 2025: 0; 2026: 3.

5 papers.

  1. 2024Streaming Deep Reinforcement Learning Finally Workscited by 24
  2. 2024Your Transformer is Secretly Linearcited by 23
  3. 2026Learning to Forget: Continual Learning with Adaptive Weight Decaycited by 1
  4. 2026Intentional Updates for Streaming Reinforcement Learningcited by 0
  5. 2026Extending Differential Temporal Difference Methods for Episodic Problemscited by 0