mlsanityData to 14 Oct 2026

Constrained Stochastic Spectral Preconditioning Converges for Nonconvex Objectives

arXiv 2605.11850 ↗First posted 12 May 2026math.OC cs.LGcited by 0

Not on the map. Why?

Abstract (arXiv)

In this work, we develop proximal preconditioned gradient methods with a focus on spectral gradient methods providing a proximal extension to the Muon and Scion optimizers. We introduce a family of stochastic algorithms that can handle a wide variety of convex and nonconvex constraints and study its convergence under heavy-tailed noise, through a novel analysis tailored to the geometry of the proposed methods. We further propose a variance-reduced version, which achieves faster convergence under standard noise assumptions. Finally, we show that the polynomial iterations used in Muon are more accurately captured by a nonlinear preconditioner than by the ideal matrix sign, leading to a convergence analysis that more faithfully reflects practical implementations.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Builds on 50 of its 68 references that are in the map. First posted on arXiv 12 May 2026.

Builds on

50 of 68 references.

References · 50

  1. 2018Spectral Normalization for Generative Adversarial Networkscited by 1,636
  2. 2022Grokking: Generalization Beyond Overfitting on Small Algorithmic Datasetscited by 629
  3. 2018signSGD: Compressed Optimisation for Non-Convex Problemscited by 571
  4. 2023Symbolic Discovery of Optimization Algorithmscited by 434
  5. 2018Shampoo: Preconditioned Stochastic Tensor Optimizationcited by 426
  6. 2023Analyzing and Improving the Training Dynamics of Diffusion Modelscited by 402
  7. 2019Why gradient clipping accelerates training: A theoretical justification for adaptivitycited by 341
  8. 2019Momentum-Based Variance Reduction in Non-Convex SGDcited by 306
  9. 2025Training Deep Learning Models with Norm-Constrained LMOscited by 192
  10. 2020Momentum Improves Normalized SGDcited by 139
  11. 2025The Polar Express: Optimal Matrix Sign Methods and Their Application to the Muon Algorithmcited by 110
  12. 2025Fantastic Pretraining Optimizers and Where to Find Themcited by 100
38 more
  1. 2025Understanding Gradient Orthogonalization for Deep Learning via Non-Euclidean Trust-Region Optimizationcited by 92
  2. 2020Improved Analysis of Clipping Algorithms for Non-convex Optimizationcited by 89
  3. 2025Muon Optimizes Under Spectral Norm Constraintscited by 86
  4. 2023Revisiting Gradient Clipping: Stochastic bias and tight convergence guaranteescited by 83
  5. 2021High-probability Bounds for Non-Convex Stochastic Optimization with Heavy Tailscited by 80
  6. 2025Gluon: Making Muon & Scion Great Again! (Bridging Theory and Practice of LMO-based Optimizers for LLMs)cited by 72
  7. 2024nGPT: Normalized Transformer with Representation Learning on the Hyperspherecited by 70
  8. 2025Dion: Distributed Orthonormalized Updatescited by 68
  9. 2024MARS: Unleashing the Power of Variance Reduction for Training Large Modelscited by 64
  10. 2023Convergence of AdaGrad for Non-convex Objectives: Simple Proofs and Relaxed Assumptionscited by 57
  11. 2025PolarGrad: A Class of Matrix-Gradient Optimizers from a Unifying Preconditioning Perspectivecited by 52
  12. 2023Lion Secretly Solves Constrained Optimization: As Lyapunov Predictscited by 51
  13. 2015Preconditioned Stochastic Gradient Descentcited by 50
  14. 2024From Gradient Clipping to Normalization for Heavy Tailed SGDcited by 49
  15. 2023Convex and Non-convex Optimization Under Generalized Smoothnesscited by 44
  16. 2024Nonconvex Stochastic Optimization under Heavy-Tailed Noises: Optimal Convergence without Gradient Clippingcited by 42
  17. 2025Lions and Muons: Optimization via Stochastic Frank-Wolfe under Heavy-Tailed Noisecited by 41
  18. 2023Generalized-Smooth Nonconvex Optimization is As Efficient As Smooth Nonconvex Optimizationcited by 35
  19. 2026Controlled LLM Training on Spectral Spherecited by 31
  20. 2025Beyond the Ideal: Analyzing the Inexact Muon Updatecited by 27
  21. 2023High-Probability Convergence for Composite and Distributed Stochastic Minimization and Variational Inequalities with Heavy-Tailed Noisecited by 24
  22. 2025AdaGrad Meets Muon: Adaptive Stepsizes for Orthogonal Updatescited by 23
  23. 2025Training Transformers with Enforced Lipschitz Constantscited by 20
  24. 2024Optimizing (L0, L1)-Smooth Functions by Gradient Methodscited by 19
  25. 2024Communication Efficient Distributed Training with Distributed Lioncited by 18
  26. 2025Generalized Gradient Norm Clipping & Non-Euclidean (L0,L1)-Smoothnesscited by 17
  27. 2025Can SGD Handle Heavy-Tailed Noise?cited by 17
  28. 2022Nonlinear gradient mappings and stochastic optimization: A general framework with applications to heavy-tail noisecited by 16
  29. 2025Sign Operator for Coping with Heavy-Tailed Noise in Non-Convex Optimization: High Probability Bounds Under (L0, L1)-Smoothnesscited by 13
  30. 2025MARS-M: When Variance Reduction Meets Matricescited by 10
  31. 2019Dual Space Preconditioning for Gradient Descentcited by 7
  32. 2025Muon is Provably Faster with Momentum Variance Reductioncited by 7
  33. 2022Anisotropic Proximal Gradientcited by 5
  34. 2025Nonlinearly Preconditioned Gradient Methods under Generalized Smoothnesscited by 4
  35. 2025Escaping saddle points without Lipschitz smoothness: the power of nonlinear preconditioningcited by 3
  36. 2021Dualities for non-Euclidean smoothness and strong convexity under the light of generalized conjugacycited by 2
  37. 2025Nonlinearly Preconditioned Gradient Methods: Momentum and Stochastic Analysiscited by 2
  38. 2019Optimization of Inf-Convolution Regularized Nonconvex Composite Problemscited by 1

Built on by

Not cited yet.