mlsanityData to 7 Oct 2026

Shampoo: Preconditioned Stochastic Tensor Optimization

arXiv 1802.09568 ↗First posted 26 Feb 2018cs.LG math.OC stat.MLcited by 413

Not on the map. Why?

Abstract (arXiv)

Preconditioned gradient methods are among the most general and powerful tools in optimization. However, preconditioning requires storing and manipulating prohibitively large matrices. We describe and analyze a new structure-aware preconditioning algorithm, called Shampoo, for stochastic optimization over tensor spaces. Shampoo maintains a set of preconditioning matrices, each of which operates on a single dimension, contracting over the remaining dimensions. We establish convergence guarantees in the stochastic convex setting, the proof of which builds upon matrix trace inequalities. Our experiments with state-of-the-art deep learning models show that Shampoo is capable of converging considerably faster than commonly used optimizers. Although it involves a more complex update rule, Shampoo's runtime per step is comparable to that of simple gradient methods such as SGD, AdaGrad, and Adam.

Key facts. Cited by 413 papers in the map (data through 7 Oct 2026). First posted on arXiv 26 Feb 2018.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 14; 2022: 18; 2023: 23; 2024: 44; 2025: 126; 2026: 188.

413 papers.

Sort:

By area:

  1. 2023Symbolic Discovery of Optimization Algorithmscited by 423
  2. 2023Sophia: A Scalable Stochastic Second-order Optimizer for Language Model Pre-trainingcited by 260
  3. 2021Does Knowledge Distillation Really Work?cited by 187
  4. 2024SOAP: Improving and Stabilizing Shampoo using Adamcited by 179
  5. 2024Old Optimizer, New Norm: An Anthologycited by 173
  6. 2021Large-Scale Differentially Private BERTcited by 117
  7. 2021Sharpness-Aware Minimization Improves Language Model Generalizationcited by 108
  8. 2025The Polar Express: Optimal Matrix Sign Methods and Their Application to the Muon Algorithmcited by 104
  9. 2021Real-time Neural Radiance Caching for Path Tracingcited by 91
  10. 2023DoG is SGD's Best Friend: A Parameter-Free Dynamic Step Size Schedulecited by 85
  11. 2024Disentangling the Causes of Plasticity Loss in Neural Networkscited by 82
  12. 2022Pathways: Asynchronous Distributed Dataflow for MLcited by 81
38 more
  1. 2025Understanding Gradient Orthogonalization for Deep Learning via Non-Euclidean Trust-Region Optimizationcited by 81
  2. 2025Muon Optimizes Under Spectral Norm Constraintscited by 77
  3. 2025Practical Efficiency of Muon for Pretrainingcited by 74
  4. 2025NorMuon: Making Muon more efficient and scalablecited by 74
  5. 2024Modular Duality in Deep Learningcited by 71
  6. 2023A Distributed Data-Parallel PyTorch Implementation of the Distributed Shampoo Optimizer for Training Neural Networks At-Scalecited by 69
  7. 2025Nested Learning: The Illusion of Deep Learning Architecturescited by 69
  8. 2025Dion: Distributed Orthonormalized Updatescited by 64
  9. 2025AdaMuon: Adaptive Muon Optimizercited by 59
  10. 2022VeLO: Training Versatile Learned Optimizers by Scaling Upcited by 57
  11. 2025Gradient Alignment in Physics-informed Neural Networks: A Second-Order Optimization Perspectivecited by 55
  12. 2025ASGO: Adaptive Structured Gradient Optimizationcited by 55
  13. 2024A New Perspective on Shampoo's Preconditionercited by 54
  14. 2023Deep Transformers without Shortcuts: Modifying Self-attention for Faithful Signal Propagationcited by 44
  15. 2024Cautious Optimizers: Improving Training with One Line of Codecited by 42
  16. 2025Power Lines: Scaling Laws for Weight Decay and Batch Size in LLM Pre-trainingcited by 41
  17. 2024Deconstructing What Makes a Good Optimizer for Language Modelscited by 40
  18. 2025In Search of Adam's Secret Saucecited by 40
  19. 2022Compute-Efficient Deep Learning: Algorithmic Trends and Opportunitiescited by 38
  20. 2025Lions and Muons: Optimization via Stochastic Frank-Wolfe under Heavy-Tailed Noisecited by 38
  21. 2025Small Batch Size Training for Language Models: When Vanilla SGD Works, and Why Gradient Accumulation Is Wastefulcited by 36
  22. 2022Re-parameterizing Your Optimizers rather than Architecturescited by 35
  23. 2022On the Factory Floor: ML Engineering for Industrial-Scale Ads Recommendation Modelscited by 34
  24. 2025Straight to Zero: Why Linearly Decaying the Learning Rate to Zero Works Best for LLMscited by 34
  25. 2025MuonBP: Faster Muon via Block-Periodic Orthogonalizationcited by 34
  26. 2024Training Data Attribution via Approximate Unrolled Differentiationcited by 33
  27. 2025Implicit Bias of Spectral Descent and Muon on Multiclass Separable Datacited by 33
  28. 2021Applications and Techniques for Fast Machine Learning in Sciencecited by 32
  29. 2022Practical tradeoffs between memory, compute, and performance in learned optimizerscited by 32
  30. 2024LoRA Done RITE: Robust Invariant Transformation Equilibration for LoRA Optimizationcited by 31
  31. 2025Structured Preconditioners in Adaptive Optimization: A Unified Analysiscited by 30
  32. 2025Isotropic Curvature Model for Understanding Deep Learning Optimization: Is Gradient Orthogonalization Optimal?cited by 30
  33. 2026Convergence of Muon with Newton-Schulzcited by 30
  34. 2025COSMOS: A Hybrid Adaptive Optimizer for Memory-Efficient Training of LLMscited by 28
  35. 2025On the Convergence of Muon and Beyondcited by 28
  36. 2026Controlled LLM Training on Spectral Spherecited by 28
  37. 2026Preconditioning Benefits of Spectral Orthogonalization in Muoncited by 28
  38. 2023On Efficient Training of Large-Scale Deep Learning Models: A Literature Reviewcited by 26

50 most cited of 413 All 413 (JSON)