mlsanityData to 14 Oct 2026

Structured Preconditioners in Adaptive Optimization: A Unified Analysis

arXiv 2503.10537 ↗First posted 13 Mar 2025cs.LGcited by 33

Not on the map. Why?

Abstract (arXiv)

We present a novel unified analysis for a broad class of adaptive optimization algorithms with structured (e.g., layerwise, diagonal, and kronecker-factored) preconditioners for both online regret minimization and offline convex optimization. Our analysis not only provides matching rate to several important structured preconditioned algorithms including diagonal AdaGrad, full-matrix AdaGrad, and AdaGrad-Norm, but also gives an improved convergence rate for a one-sided variant of Shampoo over that of original Shampoo. Interestingly, more structured preconditioners (e.g., diagonal Adagrad, AdaGrad-Norm which use less space and compute) are often presented as computationally efficient approximations to full-matrix Adagrad, aiming for improved optimization performance through better approximations. Our unified analysis challenges this prevailing view and reveals, perhaps surprisingly, that more structured preconditioners, despite using less space and computation per step, can outperform their less structured counterparts. To demonstrate this, we show that one-sided Shampoo, which is relatively much cheaper than full-matrix AdaGrad could outperform it both theoretically and experimentally.

Key facts. Cited by 33 papers in the map (data through 14 Oct 2026). Builds on 23 of its 37 references that are in the map. First posted on arXiv 13 Mar 2025.

Builds on

23 of 37 references.

References · 23

  1. 2018Adafactor: Adaptive Learning Rates with Sublinear Memory Costcited by 905
  2. 2019On the Convergence of Adam and Beyondcited by 841
  3. 2015Optimizing Neural Networks with Kronecker-factored Approximate Curvaturecited by 796
  4. 2019Large Batch Optimization for Deep Learning: Training BERT in 76 minutescited by 674
  5. 2017Large Batch Training of Convolutional Networkscited by 535
  6. 2018Shampoo: Preconditioned Stochastic Tensor Optimizationcited by 426
  7. 2020AdaBelief Optimizer: Adapting Stepsizes by the Belief in Observed Gradientscited by 283
  8. 2023Sophia: A Scalable Stochastic Second-order Optimizer for Language Model Pre-trainingcited by 265
  9. 2020ADAHESSIAN: An Adaptive Second Order Optimizer for Machine Learningcited by 202
  10. 2024Old Optimizer, New Norm: An Anthologycited by 193
  11. 2024SOAP: Improving and Stabilizing Shampoo using Adamcited by 192
  12. 2018AdaGrad stepsizes: Sharp convergence over nonconvex landscapescited by 182
11 more
  1. 2013Revisiting Natural Gradient for Deep Networkscited by 159
  2. 2018Fast Approximate Natural Gradient Descent in a Kronecker-factored Eigenbasiscited by 146
  3. 2024Modular Duality in Deep Learningcited by 77
  4. 2024MARS: Unleashing the Power of Variance Reduction for Training Large Modelscited by 64
  5. 2024A New Perspective on Shampoo's Preconditionercited by 59
  6. 2023Benchmarking Neural Network Training Algorithmscited by 52
  7. 2018Online Adaptive Methods, Universality and Accelerationcited by 51
  8. 2024Adam Exploits ℓ_∞-geometry of Loss Landscape via Coordinate-wise Adaptivitycited by 23
  9. 2017A Unified Approach to Adaptive Regularization in Online and Stochastic Optimizationcited by 19
  10. 2021Doubly Adaptive Scaled Algorithm for Machine Learning Using Second-Order Informationcited by 16
  11. 2020AdaSGD: Bridging the gap between SGD and Adamcited by 8

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 15; 2026: 18.

33 papers.

Sort:

By area:

  1. 2025ASGO: Adaptive Structured Gradient Optimizationcited by 57
  2. 2025MuonBP: Faster Muon via Block-Periodic Orthogonalizationcited by 35
  3. 2025Purifying Shampoo: Investigating Shampoo's Heuristics by Decomposing its Preconditionercited by 23
  4. 2025AdaGrad Meets Muon: Adaptive Stepsizes for Orthogonal Updatescited by 23
  5. 2025Low-rank Orthogonalization for Large-scale Matrix Optimization with Applications to Foundation Model Trainingcited by 18
  6. 2025Understanding and Improving Shampoo and SOAP via Kullback-Leibler Minimizationcited by 16
  7. 2026Adaptive Matrix Online Learning through Smoothing with Guarantees for Nonsmooth Nonconvex Optimizationcited by 14
  8. 2026Sharp Capacity Scaling of Spectral Optimizers in Learning Associative Memorycited by 14
  9. 2025SGD with Adaptive Preconditioning: Unified Analysis and Momentum Accelerationcited by 12
  10. 2026On the Width Scaling of Neural Optimizers Under Matrix Operator Norms I: Row/Column Normalization and Hyperparameter Transfercited by 12
  11. 2025Preconditioned Norms: A Unified Framework for Steepest Descent, Quasi-Newton and Adaptive Methodscited by 11
  12. 2026Enhancing LLM Training via Spectral Clippingcited by 10
21 more
  1. 2025Non-Euclidean SGD for Structured Optimization: Unified Analysis and Improved Ratescited by 8
  2. 2026Clarifying Shampoo: Adapting Spectral Descent to Stochasticity and the Parameter Trajectorycited by 8
  3. 2025On Learning Verifiers and Implications to Chain-of-Thought Reasoningcited by 6
  4. 2025On the Convergence Analysis of Muoncited by 5
  5. 2025Conda: Column-Normalized Adam for Training Large Language Models Fastercited by 5
  6. 2025A Tale of Two Geometries: Adaptive Optimizers and Non-Euclidean Descentcited by 5
  7. 2025Provable Benefit of SignGD: A Minimal Model Under Heavy-Tailed Class Imbalancecited by 5
  8. 2026Optimizer-Model Consistency: Full Finetuning with the Same Optimizer as Pretraining Forgets Lesscited by 5
  9. 2026DASH: Faster Shampoo via Batched Block Preconditioning and Efficient Inverse-Root Solverscited by 2
  10. 2026PRISM: Structured Optimization via Anisotropic Spectral Shapingcited by 2
  11. 2026A unified convergence theory for adaptive first-order methods in the nonconvex case, including AdaNorm, full and diagonal AdaGrad and Muoncited by 2
  12. 2026FOAM: Frequency and Operator Error-Based Adaptive Damping Method for Reducing Staleness-Oriented Error for Shampoocited by 2
  13. 2025Learning by solving differential equationscited by 1
  14. 2026Convergence Rate Analysis of the AdamW-style Shampoo: Unifying One-Sided and Two-Sided Preconditioningcited by 1
  15. 2026Optimal Projection-Free Adaptive SGD for Matrix Optimizationcited by 1
  16. 2026Deterministic Adam-Inspired Methods with Accelerated Convergence Ratecited by 1
  17. 2026Pro-KLShampoo: Projected KL-Shampoo with Whitening Recovered by Orthogonalizationcited by 1
  18. 2026Matrix AdaGrad: Row-wise and Column-wise Adaptive Subgradient Methodscited by 1
  19. 2026Rethinking Bregman Divergences in Kronecker-Factored Optimizerscited by 0
  20. 2026Minimax Optimality of Shampoo for Nonstationary Stochastic Optimizationcited by 0
  21. 2026Memory-Efficient Subspace Optimization with Residual Correction: From Momentum SGD to Muoncited by 0