mlsanityData to 14 Oct 2026

Schattor: Schatten-family methods for deep learning optimization

arXiv 2606.15702 ↗First posted 14 Jun 2026math.OC cs.LGcited by 1

Not on the map. Why?

Abstract (arXiv)

Modern deep learning optimization features heterogeneous parameter structures, noisy gradients, and highly nonconvex landscapes, posing significant challenges for both algorithm design and theoretical analysis. Motivated by the limitations of SGD and the success of adaptive optimizers, we propose it Schattor, a family of adaptive first-order methods based on Schatten norms. Schattor unifies SGD and the recently proposed matrix-variate adaptive optimizer Muon within a single Schatten-norm-based framework. We establish dimension-free stationarity guarantees for methods in the Schattor family for stochastic matrix optimization problems via a novel matrix martingale moment bound. We also develop multi-block extensions that adaptively balance block-wise optimization progress and prove dimension-free stationarity guarantees in this more general setting.

Key facts. Cited by 1 paper in the map (data through 14 Oct 2026). Builds on 36 of its 47 references that are in the map. First posted on arXiv 14 Jun 2026.

Builds on

36 of 47 references.

References · 36

  1. 2017Attention Is All You Needcited by 60,361
  2. 2015Deep Residual Learning for Image Recognitioncited by 52,096
  3. 2020Language Models are Few-Shot Learnerscited by 32,419
  4. 2023Mamba: Linear-Time Sequence Modeling with Selective State Spacescited by 4,718
  5. 2017Outrageously Large Neural Networks: The Sparsely-Gated Mixture-of-Experts Layercited by 3,509
  6. 2019On the Convergence of Adam and Beyondcited by 841
  7. 2019On the Variance of the Adaptive Learning Rate and Beyondcited by 772
  8. 2018signSGD: Compressed Optimisation for Non-Convex Problemscited by 571
  9. 2025Muon is Scalable for LLM Trainingcited by 428
  10. 2018Shampoo: Preconditioned Stochastic Tensor Optimizationcited by 426
  11. 2020AdaBelief Optimizer: Adapting Stepsizes by the Belief in Observed Gradientscited by 283
  12. 2024Old Optimizer, New Norm: An Anthologycited by 193
24 more
  1. 2025Training Deep Learning Models with Norm-Constrained LMOscited by 192
  2. 2019Adaptive Gradient Methods with Dynamic Bound of Learning Ratecited by 183
  3. 2018On the Convergence of A Class of Adam-Type Algorithms for Non-Convex Optimizationcited by 174
  4. 2020A Simple Convergence Proof of Adam and Adagradcited by 171
  5. 2024Adam-mini: Use Fewer Learning Rates To Gain Morecited by 137
  6. 2023Prodigy: An Expeditiously Adaptive Parameter-Free Learnercited by 133
  7. 2024Why Transformers Need Adam: A Hessian Perspectivecited by 124
  8. 2022Adam Can Converge Without Any Modification On Update Rulescited by 114
  9. 2025Understanding Gradient Orthogonalization for Deep Learning via Non-Euclidean Trust-Region Optimizationcited by 92
  10. 2025Muon Optimizes Under Spectral Norm Constraintscited by 86
  11. 2025A Note on the Convergence of Muoncited by 83
  12. 2025Gluon: Making Muon & Scion Great Again! (Bridging Theory and Practice of LMO-based Optimizers for LLMs)cited by 72
  13. 2024Scalable Optimization in the Modular Normcited by 68
  14. 2025ASGO: Adaptive Structured Gradient Optimizationcited by 57
  15. 2025PolarGrad: A Class of Matrix-Gradient Optimizers from a Unifying Preconditioning Perspectivecited by 52
  16. 2025Isotropic Curvature Model for Understanding Deep Learning Optimization: Is Gradient Orthogonalization Optimal?cited by 37
  17. 2025When do spectral gradient updates help in deep learning?cited by 31
  18. 2024BAdam: A Memory Efficient Full Parameter Optimization Method for Large Language Modelscited by 27
  19. 2025Beyond the Ideal: Analyzing the Inexact Muon Updatecited by 27
  20. 2025Low-rank Orthogonalization for Large-scale Matrix Optimization with Applications to Foundation Model Trainingcited by 18
  21. 2026The Newton-Muon Optimizercited by 18
  22. 2026Sign-Based Optimizers Are Effective Under Heavy-Tailed Noisecited by 15
  23. 2025DeMuon: A Decentralized Muon for Matrix Optimization over Graphscited by 4
  24. 2025Stochastic Approximation with Block Coordinate Optimal Stepsizescited by 3

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2026: 1.

1 paper.

  1. 2026Optimizing Large Language Models with Chained LMOscited by 0