When to use what Schatten-p norm in deep learning?
Not on the map. Why?
Abstract (arXiv)
Schatten-∞ based optimizers such as Muon have shown promising empirical performance, but there remains seemingly conflicting observations regarding whether they are beneficial. We resolve this conflict by showing that the conclusion is regime dependent. Even when the objective is smooth in the Schatten-∞ geometry, smaller Schatten-p geometries can be optimal, specifically in the low-dimensional regime, which we show includes Chinchilla scaling. This conclusion follows from a new noise-robust acceleration result for the SODA framework for p>2. The same analysis explains why Muon-like methods do not require warmup, why they naturally favor large batches, and yields a batch size scaling rule for arbitrary p.
Key facts. Cited by 2 papers in the map (data through 14 Oct 2026). Builds on 33 of its 53 references that are in the map. First posted on arXiv 13 Jun 2026.
Builds on
33 of 53 references.
References · 33
- 2022Training Compute-Optimal Large Language Modelscited by 3,019
- 2023Symbolic Discovery of Optimization Algorithmscited by 434
- 2019Online Learning: A Modern Introduction Using Convex Optimizationcited by 417
- 2019PowerSGD: Practical Low-Rank Gradient Compression for Distributed Optimizationcited by 238
- 2025Training Deep Learning Models with Norm-Constrained LMOscited by 192
- 2024The Road Less Scheduledcited by 163
- 2020Stochastic Optimization with Heavy-Tailed Noise via Accelerated Gradient Clippingcited by 104
- 2025Fantastic Pretraining Optimizers and Where to Find Themcited by 100
- 2025Understanding Gradient Orthogonalization for Deep Learning via Non-Euclidean Trust-Region Optimizationcited by 92
- 2025Muon Optimizes Under Spectral Norm Constraintscited by 86
- 2021High-probability Bounds for Non-Convex Stochastic Optimization with Heavy Tailscited by 80
- 2025Practical Efficiency of Muon for Pretrainingcited by 80
21 more
- 2024Modular Duality in Deep Learningcited by 77
- 2024Scalable Optimization in the Modular Normcited by 68
- 2025Dion: Distributed Orthonormalized Updatescited by 68
- 2025Benchmarking Optimizers for Large Language Model Pretrainingcited by 68
- 2019UniXGrad: A Universal, Adaptive Algorithm with Optimal Guarantees for Constrained Optimizationcited by 56
- 2018Online Adaptive Methods, Universality and Accelerationcited by 51
- 2024From Gradient Clipping to Normalization for Heavy Tailed SGDcited by 49
- 2018Stochastic Conditional Gradient Methods: From Convex Minimization to Submodular Maximizationcited by 47
- 2024Nonconvex Stochastic Optimization under Heavy-Tailed Noises: Optimal Convergence without Gradient Clippingcited by 42
- 2025Lions and Muons: Optimization via Stochastic Frank-Wolfe under Heavy-Tailed Noisecited by 41
- 2025Small Batch Size Training for Language Models: When Vanilla SGD Works, and Why Gradient Accumulation Is Wastefulcited by 41
- 2017Online to Offline Conversions, Universality and Adaptive Minibatch Sizescited by 39
- 2013On Lower Complexity Bounds for Large-Scale Smooth Convex Optimizationcited by 24
- 2022Mirror Descent Strikes Again: Optimal Stochastic Convex Optimization under Infinite Noise Variancecited by 24
- 202494% on CIFAR-10 in 3.29 Seconds on a Single GPUcited by 16
- 2025Is your batch size the problem? Revisiting the Adam-SGD gap in language modelingcited by 15
- 2026Muon is Not That Special: Random or Inverted Spectra Work Just as Wellcited by 14
- 2026HTMuon: Improving Muon via Heavy-Tailed Spectral Correctioncited by 12
- 2026On the Role of Batch Size in Stochastic Conditional Gradient Methodscited by 7
- 2026Optimistic Dual Averaging Unifies Modern Optimizerscited by 3
- 2025On the necessity of adaptive regularisation:Optimal anytime online learning on boldsymbolℓp-ballscited by 1
Built on by
Citing papers per year: 2026: 2.
2 papers.