DADA: Dual Averaging with Distance Adaptation
Not on the map. Why?
Abstract (arXiv)
We present a novel universal gradient method for solving convex optimization problems. Our algorithm, Dual Averaging with Distance Adaptation (DADA), is based on the classical scheme of dual averaging and dynamically adjusts its coefficients based on observed gradients and the distance between iterates and the starting point, eliminating the need for problem-specific parameters. DADA is a universal algorithm that simultaneously works for a broad spectrum of problem classes, provided the local growth of the objective function around its minimizer can be bounded. Particular examples of such problem classes are nonsmooth Lipschitz functions, Lipschitz-smooth functions, H"older-smooth functions, functions with high-order Lipschitz derivative, quasi-self-concordant functions, and (L0,L1)-smooth functions. Crucially, DADA is applicable to both unconstrained and constrained problems, even when the domain is unbounded, without requiring prior knowledge of the number of iterations or desired accuracy.
Key facts. Cited by 3 papers in the map (data through 7 Oct 2026). Builds on 17 of its 27 references that are in the map. First posted on arXiv 17 Jan 2025.
Builds on
17 of 27 references.
References · 17
- 2016Optimization Methods for Large-Scale Machine Learningcited by 1,140
- 2021Carbon Emissions and Large Neural Network Trainingcited by 561
- 2019Why gradient clipping accelerates training: A theoretical justification for adaptivitycited by 335
- 2016Coin Betting and Parameter-Free Online Learningcited by 131
- 2023Prodigy: An Expeditiously Adaptive Parameter-Free Learnercited by 129
- 2018Black-Box Reductions for Parameter-free Online Learning in Banach Spacescited by 124
- 2023Learning-Rate-Free Learning by D-Adaptationcited by 113
- 2020The Cost of Training NLP Models: A Concise Overviewcited by 97
- 2023DoG is SGD's Best Friend: A Parameter-Free Dynamic Step Size Schedulecited by 85
- 2022Making SGD Parameter-Freecited by 50
- 2023DoWG Unleashed: An Efficient Universal Parameter-Free Gradient Descent Methodcited by 35
- 2022Super-Universal Regularized Newton Methodcited by 20
5 more
- 2023Stochastic Nonsmooth Convex Optimization with Heavy-Tailed Noises: High-Probability Bound, In-Expectation Rate and Initial Distance Adaptationcited by 20
- 2024Universality of AdaGrad Stepsizes for Stochastic Optimization: Inexact Oracle, Acceleration and Variance Reductioncited by 9
- 2023Minimizing Quasi-Self-Concordant Functions by Gradient Regularization of Newton Methodcited by 7
- 2019Smoothness parameter of power of Euclidean normcited by 6
- 2023Primal subgradient methods with predefined stepsizescited by 2
Built on by
Citing papers per year: 2025: 1; 2026: 2.
3 papers.