mlsanityData to 7 Oct 2026

Learning-Rate-Free Learning by D-Adaptation

arXiv 2301.07733 ↗First posted 18 Jan 2023cs.LG cs.AI math.OCcited by 113

Not on the map. Why?

Abstract (arXiv)

D-Adaptation is an approach to automatically setting the learning rate which asymptotically achieves the optimal rate of convergence for minimizing convex Lipschitz functions, with no back-tracking or line searches, and no additional function value or gradient evaluations per step. Our approach is the first hyper-parameter free method for this class without additional multiplicative log factors in the convergence rate. We present extensive experiments for SGD and Adam variants of our method, where the method automatically matches hand-tuned learning rates across more than a dozen diverse machine learning problems, including large-scale vision and language problems. An open-source implementation is available.

Key facts. Cited by 113 papers in the map (data through 7 Oct 2026). Builds on 29 of its 50 references that are in the map. First posted on arXiv 18 Jan 2023.

Builds on

29 of 50 references.

References · 29

  1. 2015Deep Residual Learning for Image Recognitioncited by 51,809
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  3. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 29,915
  4. 2019RoBERTa: A Robustly Optimized BERT Pretraining Approachcited by 13,266
  5. 2015Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networkscited by 9,617
  6. 2014ImageNet Large Scale Visual Recognition Challengecited by 8,441
  7. 2016Densely Connected Convolutional Networkscited by 6,691
  8. 2016Wide Residual Networkscited by 3,057
  9. 2016Aggregated Residual Transformations for Deep Neural Networkscited by 2,540
  10. 2015Aligning Books and Movies: Towards Story-like Visual Explanations by Watching Movies and Reading Bookscited by 1,041
  11. 2018fastMRI: An Open Dataset and Benchmarks for Accelerated MRIcited by 464
  12. 2019Online Learning: A Modern Introduction Using Convex Optimizationcited by 407
17 more
  1. 2019Deep Learning Recommendation Model for Personalization and Recommendation Systemscited by 327
  2. 2016Hyperparameter optimization with approximate gradientcited by 231
  3. 2020End-to-End Variational Networks for Accelerated MRI Reconstructioncited by 201
  4. 2018AdaGrad stepsizes: Sharp convergence over nonconvex landscapescited by 182
  5. 2016Sequence-to-Sequence Learning as Beam-Search Optimizationcited by 126
  6. 2016Scale-Free Online Learningcited by 83
  7. 2019Revisiting the Polyak step sizecited by 55
  8. 2022Making SGD Parameter-Freecited by 50
  9. 2014Unconstrained Online Linear Learning in Hilbert Spaces: Minimax Algorithms and Normal Approximationscited by 46
  10. 2021Parameter-free Stochastic Optimization of Variationally Coherent Functionscited by 27
  11. 2022PDE-Based Optimal Strategy for Unconstrained Online Learningcited by 27
  12. 2020Momentum via Primal Averaging: Theoretical Insights and Learning Rate Schedules for Non-Convex Optimizationcited by 19
  13. 2020Guarantees for Tuning the Step Size using a Learning-to-Learn Approachcited by 10
  14. 2020The Power of Factorial Powers: New Parameter settings for (Stochastic) Optimizationcited by 8
  15. 2019Offset Sampling Improves Deep Learning based Accelerated MRI Reconstructions by Exploiting Symmetrycited by 4
  16. 2022Optimal first-order methods for convex functions with a quadratic upper boundcited by 4
  17. 2018Efficient First-order Methods for Convex Minimization: a Constructive Approachcited by 3

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 25; 2024: 37; 2025: 30; 2026: 21.

113 papers.

Sort:

By area:

  1. 2023Simple and Controllable Music Generationcited by 480
  2. 2023Small-scale proxies for large-scale Transformer training instabilitiescited by 182
  3. 2024Old Optimizer, New Norm: An Anthologycited by 173
  4. 2024The Road Less Scheduledcited by 158
  5. 2023Prodigy: An Expeditiously Adaptive Parameter-Free Learnercited by 129
  6. 2023Multiple Physics Pretraining for Physical Surrogate Modelscited by 90
  7. 2023DoG is SGD's Best Friend: A Parameter-Free Dynamic Step Size Schedulecited by 85
  8. 2024Scalable Optimization in the Modular Normcited by 64
  9. 2023Towards Stability of Autoregressive Neural Operatorscited by 54
  10. 2024Masked Audio Generation using a Single Non-Autoregressive Transformercited by 50
  11. 2023DoWG Unleashed: An Efficient Universal Parameter-Free Gradient Descent Methodcited by 35
  12. 2024Data-Efficient Operator Learning via Unsupervised Pretraining and In-Context Learningcited by 32
38 more
  1. 2024Mocap Everyone Everywhere: Lightweight Motion Capture With Smartwatches and a Head-Mounted Cameracited by 30
  2. 2024Unlearning as multi-task optimization: A normalized gradient difference approach with an adaptive learning ratecited by 29
  3. 2023MoMo: Momentum Models for Adaptive Learning Ratescited by 24
  4. 2023Ego3DPose: Capturing 3D Cues from Binocular Egocentric Viewscited by 22
  5. 2023Adaptive Proximal Gradient Method for Convex Optimizationcited by 21
  6. 2025Why Gradients Rapidly Increase Near the End of Trainingcited by 21
  7. 2023Stochastic Nonsmooth Convex Optimization with Heavy-Tailed Noises: High-Probability Bound, In-Expectation Rate and Initial Distance Adaptationcited by 20
  8. 2023Mechanic: A Learning Rate Tunercited by 20
  9. 2023A simple uniformly optimal method without line search for convex optimizationcited by 18
  10. 2024VMAS: Video-to-Music Generation via Semantic Alignment in Web Music Videoscited by 14
  11. 2024Tuning-Free Stochastic Optimizationcited by 13
  12. 2024How Free is Parameter-Free Stochastic Optimization?cited by 12
  13. 2025SGD with Adaptive Preconditioning: Unified Analysis and Momentum Accelerationcited by 12
  14. 2024Fully Unconstrained Online Learningcited by 10
  15. 2024An Adaptive Stochastic Gradient Method with Non-negative Gauss-Newton Stepsizescited by 10
  16. 2023An Automatic Learning Rate Schedule Algorithm for Achieving Faster Convergence and Steeper Descentcited by 9
  17. 2024Gradient descent with generalized Newton's methodcited by 9
  18. 2024Remove that Square Root: A New Efficient Scale-Invariant Version of AdaGradcited by 8
  19. 2024Stepping on the Edge: Curvature Aware Learning Rate Tunerscited by 8
  20. 2023Adaptive Federated Learning with Auto-Tuned Clientscited by 7
  21. 2025Towards hyperparameter-free optimization with differential privacycited by 7
  22. 2025Position: Machine Learning Conferences Should Establish a "Refutations and Critiques" Trackcited by 7
  23. 2026ScheduleFree+: Scaling Learning-Rate-Free & Schedule-Free Learning to Large Language Modelscited by 6
  24. 2023Non-Uniform Smoothness for Gradient Descentcited by 5
  25. 2023SANIA: Polyak-type Optimization Framework Leads to Scale Invariant Stochastic Algorithmscited by 5
  26. 2024MetaOptimize: A Framework for Optimizing Step Sizes and Other Meta-parameterscited by 5
  27. 2025Analysis of an Idealized Stochastic Polyak Method and its Application to Black-Box Model Distillationcited by 5
  28. 2023μ2-SGD: Stable Stochastic Optimization via a Double Momentum Mechanismcited by 4
  29. 2024Dreamguider: Improved Training free Diffusion-based Conditional Generationcited by 4
  30. 2024CRONOS: Enhancing Deep Learning with Scalable GPU Accelerated Convex Neural Networkscited by 4
  31. 2025LOB-Bench: Benchmarking Generative AI for Finance -- an Application to Limit Order Book Datacited by 4
  32. 2025Nesterov Finds GRAAL: Optimal and Adaptive Gradient Method for Convex Optimizationcited by 4
  33. 2026Where Does Warm-Up Come From? Adaptive Scheduling for Norm-Constrained Optimizerscited by 4
  34. 2024Stochastic Weakly Convex Optimization Beyond Lipschitz Continuitycited by 3
  35. 2024AdaBatchGrad: Combining Adaptive Batch Size and Adaptive Step Sizecited by 3
  36. 2024Parameter-free Clipped Gradient Descent Meets Polyakcited by 3
  37. 2024The High Line: Exact Risk and Learning Rate Curves of Stochastic Adaptive Learning Rate Algorithmscited by 3
  38. 2025DADA: Dual Averaging with Distance Adaptationcited by 3

50 most cited of 113 All 113 (JSON)