mlsanityData to 7 Oct 2026

DoG is SGD's Best Friend: A Parameter-Free Dynamic Step Size Schedule

arXiv 2302.12022 ↗First posted 8 Feb 2023cs.LG math.OCcited by 85

Not on the map. Why?

Abstract (arXiv)

We propose a tuning-free dynamic SGD step size formula, which we call Distance over Gradients (DoG). The DoG step sizes depend on simple empirical quantities (distance from the initial point and norms of gradients) and have no ``learning rate'' parameter. Theoretically, we show that a slight variation of the DoG formula enjoys strong parameter-free convergence guarantees for stochastic convex optimization assuming only locally bounded stochastic gradients. Empirically, we consider a broad range of vision and language transfer learning tasks, and show that DoG's performance is close to that of SGD with tuned learning rate. We also propose a per-layer variant of DoG that generally outperforms tuned SGD, approaching the performance of tuned Adam. A PyTorch implementation is available at https://github.com/formll/dog

Key facts. Cited by 85 papers in the map (data through 7 Oct 2026). Builds on 62 of its 119 references that are in the map. First posted on arXiv 8 Feb 2023.

Builds on

62 of 119 references.

References · 62

  1. 2017Attention Is All You Needcited by 59,932
  2. 2015Deep Residual Learning for Image Recognitioncited by 51,809
  3. 2014Adam: A Method for Stochastic Optimizationcited by 36,960
  4. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  5. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 29,915
  6. 2019PyTorch: An Imperative Style, High-Performance Deep Learning Librarycited by 20,160
  7. 2014Very Deep Convolutional Networks for Large-Scale Image Recognitioncited by 15,267
  8. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,491
  9. 2019RoBERTa: A Robustly Optimized BERT Pretraining Approachcited by 13,266
  10. 2016Densely Connected Convolutional Networkscited by 6,691
  11. 2018GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understandingcited by 4,854
  12. 2014Going Deeper with Convolutionscited by 4,231
38 more
  1. 2016SQuAD: 100,000+ Questions for Machine Comprehension of Textcited by 4,115
  2. 2022A ConvNet for the 2020scited by 3,545
  3. 2016Wide Residual Networkscited by 3,057
  4. 2017A Broad-Coverage Challenge Corpus for Sentence Understanding through Inferencecited by 2,665
  5. 2013Describing Textures in the Wildcited by 2,110
  6. 2019SuperGLUE: A Stickier Benchmark for General-Purpose Language Understanding Systemscited by 1,718
  7. 2016CLEVR: A Diagnostic Dataset for Compositional Language and Elementary Visual Reasoningcited by 1,445
  8. 2022Model soups: averaging weights of multiple fine-tuned models improves accuracy without increasing inference timecited by 1,366
  9. 2017SemEval-2017 Task 1: Semantic Textual Similarity - Multilingual and Cross-lingual Focused Evaluationcited by 998
  10. 2018Neural Network Acceptability Judgmentscited by 952
  11. 2018Adafactor: Adaptive Learning Rates with Sublinear Memory Costcited by 896
  12. 2019On the Convergence of Adam and Beyondcited by 834
  13. 2017Remote Sensing Image Scene Classification: Benchmark and State of the Artcited by 791
  14. 2019Large Batch Optimization for Deep Learning: Training BERT in 76 minutescited by 671
  15. 2017Large Batch Training of Convolutional Networkscited by 529
  16. 2016Linear Convergence of Gradient and Proximal-Gradient Methods Under the Polyak-Lojasiewicz Conditioncited by 526
  17. 2021Datasets: A Community Library for Natural Language Processingcited by 526
  18. 2019A Large-scale Study of Representation Learning with the Visual Task Adaptation Benchmarkcited by 419
  19. 2019Unlabeled Data Improves Adversarial Robustnesscited by 415
  20. 2018Shampoo: Preconditioned Stochastic Tensor Optimizationcited by 413
  21. 2018Time-uniform, nonparametric, nonasymptotic confidence sequencescited by 220
  22. 2018AdaGrad stepsizes: Sharp convergence over nonconvex landscapescited by 182
  23. 2020Large-Scale Methods for Distributionally Robust Optimizationcited by 179
  24. 2018Stochastic model-based minimization of weakly convex functionscited by 157
  25. 2020Stochastic Polyak Step-size for SGD: An Adaptive Learning Rate for Fast Convergencecited by 137
  26. 2016Coin Betting and Parameter-Free Online Learningcited by 131
  27. 2018Black-Box Reductions for Parameter-free Online Learning in Banach Spacescited by 124
  28. 2019Painless Stochastic Gradient: Interpolation, Line-Search, and Convergence Ratescited by 119
  29. 2018An Alternative View: When Does SGD Escape Local Minima?cited by 115
  30. 2023Learning-Rate-Free Learning by D-Adaptationcited by 113
  31. 2022The Power of Adaptivity in SGD: Self-Tuning Step Sizes with Unbounded Gradients and Affine Variancecited by 60
  32. 2020Lipschitz and Comparator-Norm Adaptivity in Online Learningcited by 57
  33. 2019Revisiting the Polyak step sizecited by 55
  34. 2022Making SGD Parameter-Freecited by 50
  35. 2020On the distance between two neural networks and the stability of learningcited by 46
  36. 2019Training Neural Networks for and by Interpolationcited by 45
  37. 2022Parameter-free Mirror Descentcited by 45
  38. 2019SGD Converges to Global Minimum in Deep Learning via Star-convex Pathcited by 39

50 most cited of 62 All 62 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 15; 2024: 29; 2025: 26; 2026: 15.

85 papers.

Sort:

By area:

  1. 2023Small-scale proxies for large-scale Transformer training instabilitiescited by 182
  2. 2024Old Optimizer, New Norm: An Anthologycited by 173
  3. 2023Prodigy: An Expeditiously Adaptive Parameter-Free Learnercited by 129
  4. 2024Scalable Optimization in the Modular Normcited by 64
  5. 2023Optimal Linear Decay Learning Rate Schedules and Further Refinementscited by 41
  6. 2023DoWG Unleashed: An Efficient Universal Parameter-Free Gradient Descent Methodcited by 35
  7. 2024Unlearning as multi-task optimization: A normalized gradient difference approach with an adaptive learning ratecited by 29
  8. 2023Adaptive Proximal Gradient Method for Convex Optimizationcited by 21
  9. 2023Stochastic Nonsmooth Convex Optimization with Heavy-Tailed Noises: High-Probability Bound, In-Expectation Rate and Initial Distance Adaptationcited by 20
  10. 2023Mechanic: A Learning Rate Tunercited by 20
  11. 2023Normalized Gradients for Allcited by 18
  12. 2023A simple uniformly optimal method without line search for convex optimizationcited by 18
38 more
  1. 2024Tuning-Free Stochastic Optimizationcited by 13
  2. 2024Discounted Adaptive Online Learning: Towards Better Regularizationcited by 12
  3. 2024How Free is Parameter-Free Stochastic Optimization?cited by 12
  4. 2025SGD with Adaptive Preconditioning: Unified Analysis and Momentum Accelerationcited by 12
  5. 2024The Price of Adaptivity in Stochastic Convex Optimizationcited by 11
  6. 2024Fully Unconstrained Online Learningcited by 10
  7. 2024On the Last-Iterate Convergence of Shuffling Gradient Methodscited by 9
  8. 2024Gradient descent with generalized Newton's methodcited by 9
  9. 2024Stepping on the Edge: Curvature Aware Learning Rate Tunerscited by 8
  10. 2025Towards hyperparameter-free optimization with differential privacycited by 7
  11. 2025Understanding Outer Optimizers in Local SGD: Learning Rates, Momentum, and Accelerationcited by 7
  12. 2024Projected gradient methods for nonconvex and stochastic smooth optimization: new complexities and auto-conditioned stepsizescited by 6
  13. 2026ScheduleFree+: Scaling Learning-Rate-Free & Schedule-Free Learning to Large Language Modelscited by 6
  14. 2023SANIA: Polyak-type Optimization Framework Leads to Scale Invariant Stochastic Algorithmscited by 5
  15. 2024MetaOptimize: A Framework for Optimizing Step Sizes and Other Meta-parameterscited by 5
  16. 2024Adaptive Variance Reduction for Stochastic Optimization under Weaker Assumptionscited by 5
  17. 2025Analysis of an Idealized Stochastic Polyak Method and its Application to Black-Box Model Distillationcited by 5
  18. 2025Gradient-Variation Online Adaptivity for Accelerated Optimization with H"older Smoothnesscited by 5
  19. 2023μ2-SGD: Stable Stochastic Optimization via a Double Momentum Mechanismcited by 4
  20. 2024Dreamguider: Improved Training free Diffusion-based Conditional Generationcited by 4
  21. 2025Nesterov Finds GRAAL: Optimal and Adaptive Gradient Method for Convex Optimizationcited by 4
  22. 2023A Whole New Ball Game: A Primal Accelerated Method for Matrix Games and Minimizing the Maximum of Smooth Functionscited by 3
  23. 2024Stochastic Weakly Convex Optimization Beyond Lipschitz Continuitycited by 3
  24. 2024Parameter-free Clipped Gradient Descent Meets Polyakcited by 3
  25. 2024The High Line: Exact Risk and Learning Rate Curves of Stochastic Adaptive Learning Rate Algorithmscited by 3
  26. 2025DADA: Dual Averaging with Distance Adaptationcited by 3
  27. 2025Parameter-free Algorithms for the Stochastically Extended Adversarial Modelcited by 3
  28. 2025Noise-Adaptive Layerwise Learning Rates: Accelerating Geometry-Aware Optimization for Deep Neural Network Trainingcited by 3
  29. 2026Convex Dominance in Deep Learning I: A Scaling Law of Loss and Learning Ratecited by 3
  30. 2023On the convergence of adaptive first order methods: proximal gradient and alternating minimization algorithmscited by 2
  31. 2024Unleash Graph Neural Networks from Heavy Tuningcited by 2
  32. 2025How far away are truly hyperparameter-free learning algorithms?cited by 2
  33. 2025LightSAM: Parameter-Agnostic Sharpness-Aware Minimizationcited by 2
  34. 2025A Robust widetildeO(1/√T) Rate for Unprojected TD Learning with Linear Function Approximationcited by 2
  35. 2025Adaptive Algorithms with Sharp Convergence Rates for Stochastic Hierarchical Optimizationcited by 2
  36. 2025Convergence of Regret Matching in Potential Games and Constrained Optimizationcited by 2
  37. 2023ELRA: Exponential learning rate adaption gradient descent optimization methodcited by 1
  38. 2024Towards Stability of Parameter-Free Optimizationcited by 1

50 most cited of 85 All 85 (JSON)