mlsanityData to 7 Oct 2026

DoWG Unleashed: An Efficient Universal Parameter-Free Gradient Descent Method

arXiv 2305.16284 ↗First posted 25 May 2023cs.LG math.OC stat.MLcited by 35

Not on the map. Why?

Abstract (arXiv)

This paper proposes a new easy-to-implement parameter-free gradient-based optimizer: DoWG (Distance over Weighted Gradients). We prove that DoWG is efficient -- matching the convergence rate of optimally tuned gradient descent in convex optimization up to a logarithmic factor without tuning any parameters, and universal -- automatically adapting to both smooth and nonsmooth problems. While popular algorithms following the AdaGrad framework compute a running average of the squared gradients to use for normalization, DoWG maintains a new distance-based weighted version of the running average, which is crucial to achieve the desired properties. To complement our theory, we also show empirically that DoWG trains at the edge of stability, and validate its effectiveness on practical machine learning tasks.

Key facts. Cited by 35 papers in the map (data through 7 Oct 2026). Builds on 50 of its 72 references that are in the map. First posted on arXiv 25 May 2023.

Builds on

50 of 72 references.

References · 50

  1. 2015Deep Residual Learning for Image Recognitioncited by 51,809
  2. 2014Adam: A Method for Stochastic Optimizationcited by 36,960
  3. 2019PyTorch: An Imperative Style, High-Performance Deep Learning Librarycited by 20,160
  4. 2014Very Deep Convolutional Networks for Large-Scale Image Recognitioncited by 15,267
  5. 2021Convex optimizationcited by 2,479
  6. 2016Optimization Methods for Large-Scale Machine Learningcited by 1,140
  7. 2019On the Convergence of Adam and Beyondcited by 834
  8. 2014Convex Optimization: Algorithms and Complexitycited by 569
  9. 2021Carbon Emissions and Large Neural Network Trainingcited by 561
  10. 2021Gradient Descent on Neural Networks Typically Occurs at the Edge of Stabilitycited by 413
  11. 2019Why gradient clipping accelerates training: A theoretical justification for adaptivitycited by 335
  12. 2017The Marginal Value of Adaptive Gradient Methods in Machine Learningcited by 248
38 more
  1. 2018AdaGrad stepsizes: Sharp convergence over nonconvex landscapescited by 182
  2. 2020Better Theory for SGD in the Nonconvex Worldcited by 156
  3. 2018On the Convergence of Stochastic Gradient Descent with Adaptive Stepsizescited by 155
  4. 2022Self-Stabilization: The Implicit Bias of Gradient Descent at the Edge of Stabilitycited by 151
  5. 2020Stochastic Polyak Step-size for SGD: An Adaptive Learning Rate for Fast Convergencecited by 137
  6. 2016Coin Betting and Parameter-Free Online Learningcited by 131
  7. 2022Understanding Gradient Descent on Edge of Stability in Deep Learningcited by 126
  8. 2023Noise Is Not the Main Factor Behind the Gap Between SGD and Adam on Transformers, but Sign Descent Might Becited by 114
  9. 2023Learning-Rate-Free Learning by D-Adaptationcited by 113
  10. 2020The Cost of Training NLP Models: A Concise Overviewcited by 97
  11. 2023Convergence of Adam Under Relaxed Assumptionscited by 97
  12. 2022Adaptive Gradient Methods at the Edge of Stabilitycited by 88
  13. 2023DoG is SGD's Best Friend: A Parameter-Free Dynamic Step Size Schedulecited by 85
  14. 2019Adaptive Gradient Descent without Descentcited by 79
  15. 2015Beyond Convexity: Stochastic Quasi-Convex Optimizationcited by 75
  16. 2018Theoretical Analysis of Auto Rate-Tuning by Batch Normalizationcited by 65
  17. 2022The Power of Adaptivity in SGD: Self-Tuning Step Sizes with Unbounded Gradients and Affine Variancecited by 60
  18. 2020Lipschitz and Comparator-Norm Adaptivity in Online Learningcited by 57
  19. 2019UniXGrad: A Universal, Adaptive Algorithm with Optimal Guarantees for Constrained Optimizationcited by 56
  20. 2019Revisiting the Polyak step sizecited by 55
  21. 2018Online Adaptive Methods, Universality and Accelerationcited by 51
  22. 2022Making SGD Parameter-Freecited by 50
  23. 2014Unconstrained Online Linear Learning in Hilbert Spaces: Minimax Algorithms and Normal Approximationscited by 46
  24. 2022Dynamics of SGD with Stochastic Polyak Stepsizes: Truly Adaptive Variants and Convergence to Exact Solutioncited by 40
  25. 2017Online to Offline Conversions, Universality and Adaptive Minibatch Sizescited by 39
  26. 2019Lipschitz Adaptivity with Multiple Learning Rates in Online Learningcited by 36
  27. 2017Revisiting Normalized Gradient Descent: Fast Evasion of Saddle Pointscited by 28
  28. 2021Parameter-free Stochastic Optimization of Variationally Coherent Functionscited by 27
  29. 2018WNGrad: Learn the Learning Rate in Gradient Descentcited by 24
  30. 2017A Unified Approach to Adaptive Regularization in Online and Stochastic Optimizationcited by 19
  31. 2023Adaptive proximal algorithms for convex optimization under local Lipschitz continuity of the gradientcited by 19
  32. 2017Convergence Rates for Deterministic and Stochastic Subgradient Methods Without Lipschitz Continuitycited by 18
  33. 2020Adaptive Gradient Methods for Constrained Convex Optimization and Variational Inequalitiescited by 18
  34. 2018A Simple Nearly-Optimal Restart Scheme For Speeding-Up First Order Methodscited by 14
  35. 2022On the Convergence of AdaGrad(Norm) on Rd: Beyond Convexity, Non-Asymptotic Rate and Accelerationcited by 13
  36. 2022On Optimal Universal First-Order Methods for Minimizing Heterogeneous Sumscited by 9
  37. 2020Sequential convergence of AdaGrad algorithm for smooth convex optimizationcited by 5
  38. 2022Dissecting adaptive methods in GANscited by 3

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 4; 2024: 8; 2025: 15; 2026: 8.

35 papers.

Sort:

  1. 2024Old Optimizer, New Norm: An Anthologycited by 173
  2. 2023Optimal Linear Decay Learning Rate Schedules and Further Refinementscited by 41
  3. 2023Adaptive Proximal Gradient Method for Convex Optimizationcited by 21
  4. 2023Normalized Gradients for Allcited by 18
  5. 2023A simple uniformly optimal method without line search for convex optimizationcited by 18
  6. 2025SGD with Adaptive Preconditioning: Unified Analysis and Momentum Accelerationcited by 12
  7. 2024Gradient descent with generalized Newton's methodcited by 9
  8. 2025Towards hyperparameter-free optimization with differential privacycited by 7
  9. 2026ScheduleFree+: Scaling Learning-Rate-Free & Schedule-Free Learning to Large Language Modelscited by 6
  10. 2025Analysis of an Idealized Stochastic Polyak Method and its Application to Black-Box Model Distillationcited by 5
  11. 2025Nesterov Finds GRAAL: Optimal and Adaptive Gradient Method for Convex Optimizationcited by 4
  12. 2024Parameter-free Clipped Gradient Descent Meets Polyakcited by 3
23 more
  1. 2025DADA: Dual Averaging with Distance Adaptationcited by 3
  2. 2025Parameter-free Algorithms for the Stochastically Extended Adversarial Modelcited by 3
  3. 2026Convex Dominance in Deep Learning I: A Scaling Law of Loss and Learning Ratecited by 3
  4. 2025How far away are truly hyperparameter-free learning algorithms?cited by 2
  5. 2025LightSAM: Parameter-Agnostic Sharpness-Aware Minimizationcited by 2
  6. 2024Towards Stability of Parameter-Free Optimizationcited by 1
  7. 2024Learning-Rate-Free Stochastic Optimization over Riemannian Manifoldscited by 1
  8. 2024Explainable AI for Fraud Detection: An Attention-Based Ensemble of CNNs, GNNs, and A Confidence-Driven Gating Mechanismcited by 1
  9. 2025AutoSGD: Automatic Learning Rate Selection for Stochastic Gradient Descentcited by 1
  10. 2025Online Learning-guided Learning Rate Adaptation via Gradient Alignmentcited by 1
  11. 2026Randomized Feasibility Methods for Constrained Optimization with Adaptive Step Sizescited by 1
  12. 2026Distance-Aware Muon: Adaptive Step Scaling for Normalized Optimizationcited by 1
  13. 2024Tuning-Free Coreset Markov Chain Monte Carlo via Hot DoGcited by 0
  14. 2024MARINA-P: Superior Performance in Non-smooth Federated Optimization with Adaptive Stepsizescited by 0
  15. 2025First-ish Order Methods: Hessian-aware Scalings of Gradient Descentcited by 0
  16. 2025Revisiting Learning Rate Controlcited by 0
  17. 2025Stochastic Adaptive Gradient Descent Without Descentcited by 0
  18. 2025A regret minimization approach to fixed-point iterationscited by 0
  19. 2025AutoGD: Automatic Learning Rate Selection for Gradient Descentcited by 0
  20. 2026ZENITH: Automated Gradient Norm Informed Stochastic Optimizationcited by 0
  21. 2026Stochastic Auto-conditioned Fast Gradient Methods with Optimal Ratescited by 0
  22. 2026Large-scale empirical tuning and comparison of default optimizers for variational inferencecited by 0
  23. 2026Zero-order Parameter-free Optimization for LMO-based Methods: Novel Approach for Efficient Fine-tuningcited by 0