mlsanityData to 7 Oct 2026

A Unified Approach to Adaptive Regularization in Online and Stochastic Optimization

arXiv 1706.06569 ↗First posted 20 Jun 2017cs.LG math.OC stat.MLcited by 19

Not on the map. Why?

Abstract (arXiv)

We describe a framework for deriving and analyzing online optimization algorithms that incorporate adaptive, data-dependent regularization, also termed preconditioning. Such algorithms have been proven useful in stochastic optimization by reshaping the gradients according to the geometry of the data. Our framework captures and unifies much of the existing literature on adaptive online methods, including the AdaGrad and Online Newton Step algorithms as well as their diagonal versions. As a result, we obtain new convergence proofs for these algorithms that are substantially simpler than previous analyses. Our framework also exposes the rationale for the different preconditioned updates used in common stochastic optimization methods.

Key facts. Cited by 19 papers in the map (data through 7 Oct 2026). First posted on arXiv 20 Jun 2017.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 2; 2022: 0; 2023: 2; 2024: 2; 2025: 7; 2026: 6.

19 papers.

Sort:

  1. 2025Training Deep Learning Models with Norm-Constrained LMOscited by 177
  2. 2024Old Optimizer, New Norm: An Anthologycited by 173
  3. 2023DoG is SGD's Best Friend: A Parameter-Free Dynamic Step Size Schedulecited by 85
  4. 2025ASGO: Adaptive Structured Gradient Optimizationcited by 55
  5. 2023DoWG Unleashed: An Efficient Universal Parameter-Free Gradient Descent Methodcited by 35
  6. 2025Structured Preconditioners in Adaptive Optimization: A Unified Analysiscited by 30
  7. 2025SGD with Adaptive Preconditioning: Unified Analysis and Momentum Accelerationcited by 12
  8. 2024The Price of Adaptivity in Stochastic Convex Optimizationcited by 11
  9. 2025Non-Euclidean SGD for Structured Optimization: Unified Analysis and Improved Ratescited by 8
  10. 2021Structured Dropout Variational Inference for Bayesian Neural Networkscited by 7
  11. 2025A Tale of Two Geometries: Adaptive Optimizers and Non-Euclidean Descentcited by 5
  12. 2026A unified convergence theory for adaptive first-order methods in the nonconvex case, including AdaNorm, full and diagonal AdaGrad and Muoncited by 2
7 more
  1. 2025The Sample Complexity of Parameter-Free Stochastic Convex Optimizationcited by 1
  2. 2026Optimal Projection-Free Adaptive SGD for Matrix Optimizationcited by 1
  3. 2021Meta-Regularization: An Approach to Adaptive Choice of the Learning Rate in Gradient Descentcited by 0
  4. 2026Stochastic Auto-conditioned Fast Gradient Methods with Optimal Ratescited by 0
  5. 2026FOAM: Frequency and Operator Error-Based Adaptive Damping Method for Reducing Staleness-Oriented Error for Shampoocited by 0
  6. 2026Clipping the Price of Adaptivity at the Tailcited by 0
  7. 2026AOS: Adaptive Optimizer Switching via Training-State Signals for Faster Convergence and Better Generalizationcited by 0