mlsanityData to 14 Oct 2026

Mechanic: A Learning Rate Tuner

arXiv 2306.00144 ↗First posted 31 May 2023cs.LGcited by 20

Not on the map. Why?

Abstract (arXiv)

We introduce a technique for tuning the learning rate scale factor of any base optimization algorithm and schedule automatically, which we call textscmechanic. Our method provides a practical realization of recent theoretical reductions for accomplishing a similar goal in online convex optimization. We rigorously evaluate textscmechanic on a range of large scale deep learning tasks with varying batch sizes, schedules, and base optimization algorithms. These experiments demonstrate that depending on the problem, textscmechanic either comes very close to, matches or even improves upon manual tuning of learning rates.

Key facts. Cited by 20 papers in the map (data through 14 Oct 2026). Builds on 31 of its 45 references that are in the map. First posted on arXiv 31 May 2023.

Builds on

31 of 45 references.

References · 31

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,399
  2. 2014Adam: A Method for Stochastic Optimizationcited by 37,240
  3. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 30,186
  4. 2019RoBERTa: A Robustly Optimized BERT Pretraining Approachcited by 13,372
  5. 2018GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understandingcited by 4,899
  6. 2017Revisiting Unreasonable Effectiveness of Data in Deep Learning Eracited by 919
  7. 2015Automatic differentiation in machine learning: a surveycited by 855
  8. 2019Introduction to Online Convex Optimizationcited by 803
  9. 2013Stochastic First- and Zeroth-order Methods for Nonconvex Stochastic Programmingcited by 724
  10. 2023Symbolic Discovery of Optimization Algorithmscited by 434
  11. 2019Online Learning: A Modern Introduction Using Convex Optimizationcited by 417
  12. 2019Lower Bounds for Non-Convex Stochastic Optimizationcited by 307
19 more
  1. 2016Coin Betting and Parameter-Free Online Learningcited by 133
  2. 2018Black-Box Reductions for Parameter-free Online Learning in Banach Spacescited by 125
  3. 2023Learning-Rate-Free Learning by D-Adaptationcited by 113
  4. 2023DoG is SGD's Best Friend: A Parameter-Free Dynamic Step Size Schedulecited by 85
  5. 2020Lipschitz and Comparator-Norm Adaptivity in Online Learningcited by 57
  6. 2021Impossible Tuning Made Possible: A New Expert Algorithm and Its Applicationscited by 52
  7. 2022Making SGD Parameter-Freecited by 50
  8. 2022Parameter-free Mirror Descentcited by 45
  9. 2020Disentangling Adaptive Gradient Methods from Learning Ratescited by 42
  10. 2015MuProp: Unbiased Backpropagation for Stochastic Neural Networkscited by 37
  11. 2019Lipschitz Adaptivity with Multiple Learning Rates in Online Learningcited by 37
  12. 2017Online Learning Without Prior Informationcited by 36
  13. 2021Parameter-free Stochastic Optimization of Variationally Coherent Functionscited by 27
  14. 2019Combining Online Learning Guaranteescited by 22
  15. 2019Adaptive scale-invariant online algorithms for learning linear modelscited by 19
  16. 2017Online Convex Optimization with Unconstrained Domains and Lossescited by 14
  17. 2013Minimax Optimal Algorithms for Unconstrained Linear Optimizationcited by 12
  18. 2022Adaptive Gradient Methods with Local Guaranteescited by 8
  19. 2023A Nonstochastic Control Approach to Optimizationcited by 4

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2023: 3; 2024: 10; 2025: 4; 2026: 3.

20 papers.

Sort:

By area:

  1. 2023Optimal Linear Decay Learning Rate Schedules and Further Refinementscited by 43
  2. 2024Fast TRAC: A Parameter-Free Optimizer for Lifelong Reinforcement Learningcited by 17
  3. 2024Tuning-Free Stochastic Optimizationcited by 13
  4. 2024Fully Unconstrained Online Learningcited by 10
  5. 2023Improving Adaptive Online Learning Using Refined Discretizationcited by 8
  6. 2024Stepping on the Edge: Curvature Aware Learning Rate Tunerscited by 8
  7. 2023SANIA: Polyak-type Optimization Framework Leads to Scale Invariant Stochastic Algorithmscited by 5
  8. 2024MetaOptimize: A Framework for Optimizing Step Sizes and Other Meta-parameterscited by 5
  9. 2024textttskwdro: a library for Wasserstein distributionally robust machine learningcited by 4
  10. 2025How far away are truly hyperparameter-free learning algorithms?cited by 2
  11. 2025Online Learning-guided Learning Rate Adaptation via Gradient Alignmentcited by 1
  12. 2024Corridor Geometry in Gradient-Based Optimizationcited by 0
8 more
  1. 2024Adaptive Gradient Regularization: A Faster and Generalizable Optimization Technique for Deep Neural Networkscited by 0
  2. 2024Super Level Sets and Exponential Decay: A Synergistic Approach to Stable Neural Network Trainingcited by 0
  3. 2024State-free Reinforcement Learningcited by 0
  4. 2025Efficient Deep Learning Approaches for Processing Ultra-Widefield Retinal Imagingcited by 0
  5. 2025Neural Velocity for hyperparameter tuningcited by 0
  6. 2026Large-scale empirical tuning and comparison of default optimizers for variational inferencecited by 0
  7. 2026Directional Curvature from Armijo Backtracking: A Low-Cost Sharpness Probe and a Calibration-Free Learning-Rate Safeguard for Adamcited by 0
  8. 2026SOLAR: A State-Driven Online Learning Rate Scheduler for LLM Pretrainingcited by 0