mlsanityData to 13 Oct 2026

Clarifying Shampoo: Adapting Spectral Descent to Stochasticity and the Parameter Trajectory

arXiv 2602.09314 ↗First posted 10 Feb 2026cs.LG cs.AI stat.MLcited by 7

Not on the map. Why?

Abstract (arXiv)

Optimizers leveraging the matrix structure in neural networks, such as Shampoo and Muon, are more data-efficient than element-wise algorithms like Adam and Signum. While in specific settings, Shampoo and Muon reduce to spectral descent analogous to how Adam and Signum reduce to sign descent, their general relationship and relative data efficiency under controlled settings remain unclear. Through extensive experiments on language models, we demonstrate that Shampoo achieves higher token efficiency than Muon, mirroring Adam's advantage over Signum. We show that Shampoo's update applied to weight matrices can be decomposed into an adapted Muon update. Consistent with this, Shampoo's benefits can be exclusively attributed to its application to weight matrices, challenging interpretations agnostic to parameter shapes. This admits a new perspective that also avoids shortcomings of related interpretations based on variance adaptation and whitening: rather than enforcing semi-orthogonality as in spectral descent, Shampoo's updates are time-averaged semi-orthogonal in expectation.

Key facts. Cited by 7 papers in the map (data through 13 Oct 2026). Builds on 59 of its 78 references that are in the map. First posted on arXiv 10 Feb 2026.

Builds on

59 of 78 references.

References · 59

  1. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,549
  2. 2015Optimizing Neural Networks with Kronecker-factored Approximate Curvaturecited by 787
  3. 2023ConvNeXt V2: Co-designing and Scaling ConvNets with Masked Autoencoderscited by 705
  4. 2018signSGD: Compressed Optimisation for Non-Convex Problemscited by 567
  5. 2013Generating Sequences With Recurrent Neural Networkscited by 553
  6. 2014New insights and perspectives on the natural gradient methodcited by 438
  7. 2025Muon is Scalable for LLM Trainingcited by 406
  8. 2020AdaBelief Optimizer: Adapting Stepsizes by the Belief in Observed Gradientscited by 283
  9. 2019Limitations of the Empirical Fisher Approximation for Natural Gradient Descentcited by 215
  10. 2024SOAP: Improving and Stabilizing Shampoo using Adamcited by 186
  11. 2024Old Optimizer, New Norm: An Anthologycited by 183
  12. 2025Training Deep Learning Models with Norm-Constrained LMOscited by 183
38 more
  1. 2018Fast Approximate Natural Gradient Descent in a Kronecker-factored Eigenbasiscited by 144
  2. 2016A Kronecker-factored approximate Fisher matrix for convolution layerscited by 141
  3. 2017Dissecting Adam: The Sign, Magnitude and Variance of Stochastic Gradientscited by 138
  4. 2020Descending through a Crowded Valley - Benchmarking Deep Learning Optimizerscited by 118
  5. 2023Noise Is Not the Main Factor Behind the Gap Between SGD and Adam on Transformers, but Sign Descent Might Becited by 114
  6. 2025The Polar Express: Optimal Matrix Sign Methods and Their Application to the Muon Algorithmcited by 106
  7. 2023A Spectral Condition for Feature Learningcited by 104
  8. 2022Better plain ViT baselines for ImageNet-1kcited by 98
  9. 2015Optimal whitening and decorrelationcited by 96
  10. 2025Fantastic Pretraining Optimizers and Where to Find Themcited by 92
  11. 2024TorchTitan: One-stop PyTorch native solution for production ready LLM pre-trainingcited by 83
  12. 2024Heavy-Tailed Class Imbalance and Why Adam Outperforms Gradient Descent on Language Modelscited by 78
  13. 2024Modular Duality in Deep Learningcited by 74
  14. 2023A Distributed Data-Parallel PyTorch Implementation of the Distributed Shampoo Optimizer for Training Neural Networks At-Scalecited by 73
  15. 2024Scalable Optimization in the Modular Normcited by 64
  16. 2024Understanding Optimization in Deep Learning with Central Flowscited by 61
  17. 2024MARS: Unleashing the Power of Variance Reduction for Training Large Modelscited by 61
  18. 2025Benchmarking Optimizers for Large Language Model Pretrainingcited by 59
  19. 2024A New Perspective on Shampoo's Preconditionercited by 57
  20. 2025ASGO: Adaptive Structured Gradient Optimizationcited by 55
  21. 2019Fast and Simple Natural-Gradient Variational Inference with Mixture of Exponential-family Approximationscited by 51
  22. 2024The AdEMAMix Optimizer: Better, Faster, Oldercited by 50
  23. 2023Kronecker-Factored Approximate Curvature for Modern Neural Network Architecturescited by 49
  24. 2025Muon Outperforms Adam in Tail-End Associative Memory Learningcited by 43
  25. 2020Disentangling Adaptive Gradient Methods from Learning Ratescited by 41
  26. 2025In Search of Adam's Secret Saucecited by 40
  27. 2020The Geometry of Sign Gradient Descentcited by 39
  28. 2025Small Batch Size Training for Language Models: When Vanilla SGD Works, and Why Gradient Accumulation Is Wastefulcited by 39
  29. 2025Isotropic Curvature Model for Understanding Deep Learning Optimization: Is Gradient Orthogonalization Optimal?cited by 34
  30. 2022Gradient Descent on Neurons and its Link to Approximate Second-Order Optimizationcited by 30
  31. 2025Accelerating Neural Network Training: An Analysis of the AlgoPerf Competitioncited by 30
  32. 2025Structured Preconditioners in Adaptive Optimization: A Unified Analysiscited by 30
  33. 2025When do spectral gradient updates help in deep learning?cited by 27
  34. 2024Adam Exploits ℓ_∞-geometry of Loss Landscape via Coordinate-wise Adaptivitycited by 21
  35. 2025Purifying Shampoo: Investigating Shampoo's Heuristics by Decomposing its Preconditionercited by 21
  36. 2025Scaling Laws for Gradient Descent and Sign Descent for Linear Bigram Models under Zipf's Lawcited by 20
  37. 2020LaProp: Separating Momentum and Adaptivity in Adamcited by 19
  38. 2013An Almost-Linear-Time Algorithm for Approximate Max Flow in Undirected Graphs, and its Multicommodity Generalizationscited by 18

50 most cited of 59 All 59 (JSON)

Built on by

Cited by year (2026 to 13 Oct)

Citing papers per year: 2026: 7.

7 papers.

  1. 2026Symmetry-Compatible Principle for Optimizer Design: Embeddings, LM Heads, SwiGLU MLPs, and MoE Routerscited by 9
  2. 2026Demystifying Manifold Constraints in LLM Pre-trainingcited by 1
  3. 2026Pro-KLShampoo: Projected KL-Shampoo with Whitening Recovered by Orthogonalizationcited by 1
  4. 2026Runtime-Orchestrated Second-Order Optimization for Scalable LLM Trainingcited by 1
  5. 2026Reparametrizing Shampoo and SOAP for Subspace Basis Updates and BFloat16 Storagecited by 0
  6. 2026Optimizer Geometry Sets the Pace: Spectral Learning Dynamics in Matrix Factorizationcited by 0
  7. 2026How Bregman Divergences Shape Shampoocited by 0