mlsanityData to 14 Oct 2026

Optimal Projection-Free Adaptive SGD for Matrix Optimization

arXiv 2604.02505 ↗First posted 2 Apr 2026math.OC cs.LGcited by 1

Not on the map. Why?

Abstract (arXiv)

Recently, Jiang et al. [2026] developed Leon, a practical variant of One-sided Shampoo [Xie et al., 2025a, An et al., 2025] algorithm for online convex optimization, which does not require computing a costly quadratic projection at each iteration. Unfortunately, according to the existing analysis, Leon requires tuning an additional hyperparameter in its preconditioner and cannot achieve dimension-independent convergence guarantees for convex optimization problems beyond the bounded gradients assumption. In this paper, we resolve this issue by proving certain stability properties of Leon's preconditioner. Using our improved analysis, we show that tuning the extra hyperparameter can be avoided and, more importantly, develop the first practical variant of One-sided Shampoo with Nesterov acceleration, which does not require computing projections at each iteration. As a side contribution, we obtain improved dimension-independent rates in the non-smooth non-convex setting and develop a unified analysis of the proposed algorithm, which yields accelerated projection-free adaptive SGD with (block-)diagonal preconditioners.

Key facts. Cited by 1 paper in the map (data through 14 Oct 2026). Builds on 24 of its 37 references that are in the map. First posted on arXiv 2 Apr 2026.

Builds on

24 of 37 references.

References · 24

  1. 2025Muon is Scalable for LLM Trainingcited by 428
  2. 2025Kimi-VL Technical Reportcited by 388
  3. 2024Old Optimizer, New Norm: An Anthologycited by 193
  4. 2025Training Deep Learning Models with Norm-Constrained LMOscited by 192
  5. 2024Adam-mini: Use Fewer Learning Rates To Gain Morecited by 137
  6. 2024Why Transformers Need Adam: A Hessian Perspectivecited by 124
  7. 2023A Spectral Condition for Feature Learningcited by 109
  8. 2025Understanding Gradient Orthogonalization for Deep Learning via Non-Euclidean Trust-Region Optimizationcited by 92
  9. 2025Muon Optimizes Under Spectral Norm Constraintscited by 86
  10. 2025ASGO: Adaptive Structured Gradient Optimizationcited by 57
  11. 2019UniXGrad: A Universal, Adaptive Algorithm with Optimal Guarantees for Constrained Optimizationcited by 56
  12. 2025Structured Preconditioners in Adaptive Optimization: A Unified Analysiscited by 33
12 more
  1. 2021ZerO Initialization: Initializing Neural Networks with only Zeros and Onescited by 28
  2. 2024AdaGrad under Anisotropic Smoothnesscited by 22
  3. 2023Normalized Gradients for Allcited by 20
  4. 2017A Unified Approach to Adaptive Regularization in Online and Stochastic Optimizationcited by 19
  5. 2024Random Scaling and Momentum for Non-smooth Non-convex Optimizationcited by 19
  6. 2024Provable Complexity Improvement of AdaGrad over SGD: Upper and Lower Bounds in Stochastic Non-Convex Optimizationcited by 16
  7. 2026Adaptive Matrix Online Learning through Smoothing with Guarantees for Nonsmooth Nonconvex Optimizationcited by 14
  8. 2025SGD with Adaptive Preconditioning: Unified Analysis and Momentum Accelerationcited by 12
  9. 2024Universality of AdaGrad Stepsizes for Stochastic Optimization: Inexact Oracle, Acceleration and Variance Reductioncited by 9
  10. 2025Non-Euclidean SGD for Structured Optimization: Unified Analysis and Improved Ratescited by 8
  11. 2024On Linear Convergence in Smooth Convex-Concave Bilinearly-Coupled Saddle-Point Optimization: Lower Bounds and Optimal Algorithmscited by 6
  12. 2025A Tale of Two Geometries: Adaptive Optimizers and Non-Euclidean Descentcited by 5

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2026: 1.

1 paper.

  1. 2026Stochastic Non-Smooth Convex Optimization with Unbounded Gradientscited by 0