mlsanityData to 14 Oct 2026

FOAM: Frequency and Operator Error-Based Adaptive Damping Method for Reducing Staleness-Oriented Error for Shampoo

arXiv 2606.02365 ↗First posted 1 Jun 2026cs.LG cs.AIcited by 2

Not on the map. Why?

Abstract (arXiv)

Shampoo is attracting considerable attention for its superior performance on large-scale optimization benchmarks; yet it faces a significant practical bottleneck: the prohibitive computational overhead of matrix inversion. To mitigate this, practitioners typically rely on stale preconditioner updates, creating a fundamental trade-off between computational efficiency and optimization fidelity. In this work, we provide a theoretical study of staleness through the complementary lenses of convergence and stability. While staleness improves computational efficiency, it inherently degrades performance and introduces numerical instability. Crucially, we identify that damping, acting as a numerical stabilizer, can effectively suppress these negative effects. Guided by this analysis, we propose FOAM, an adaptive algorithm that stabilizes training by dynamically controlling both the damping factor and the eigendecomposition frequency based on an approximation of the staleness-oriented error. Experimental results demonstrate that FOAM reduces wall-clock time compared to standard Shampoo while maintaining robust convergence.

Key facts. Cited by 2 papers in the map (data through 14 Oct 2026). Builds on 35 of its 57 references that are in the map. First posted on arXiv 1 Jun 2026.

Builds on

35 of 57 references.

References · 35

  1. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 30,186
  2. 2023GPT-4 Technical Reportcited by 18,396
  3. 2020Scaling Laws for Neural Language Modelscited by 6,586
  4. 2016Pointer Sentinel Mixture Modelscited by 3,096
  5. 2022Training Compute-Optimal Large Language Modelscited by 3,019
  6. 2020Conformer: Convolution-augmented Transformer for Speech Recognitioncited by 1,863
  7. 2016Optimization Methods for Large-Scale Machine Learningcited by 1,150
  8. 2017Visualizing the Loss Landscape of Neural Netscited by 1,130
  9. 2015Optimizing Neural Networks with Kronecker-factored Approximate Curvaturecited by 796
  10. 2019Large Batch Optimization for Deep Learning: Training BERT in 76 minutescited by 674
  11. 2019Online Learning: A Modern Introduction Using Convex Optimizationcited by 417
  12. 2019PyHessian: Neural Networks Through the Lens of the Hessiancited by 292
23 more
  1. 2019Limitations of the Empirical Fisher Approximation for Natural Gradient Descentcited by 217
  2. 2024Old Optimizer, New Norm: An Anthologycited by 193
  3. 2025Training Deep Learning Models with Norm-Constrained LMOscited by 192
  4. 2023Handbook of Convergence Theorems for (Stochastic) Gradient Methodscited by 163
  5. 2024Adam-mini: Use Fewer Learning Rates To Gain Morecited by 137
  6. 2023A Distributed Data-Parallel PyTorch Implementation of the Distributed Shampoo Optimizer for Training Neural Networks At-Scalecited by 77
  7. 2024A New Perspective on Shampoo's Preconditionercited by 59
  8. 2023Benchmarking Neural Network Training Algorithmscited by 52
  9. 2023Optimal Stochastic Non-smooth Non-convex Optimization through Online-to-Non-convex Conversioncited by 45
  10. 2020Disentangling Adaptive Gradient Methods from Learning Ratescited by 42
  11. 2025Small Batch Size Training for Language Models: When Vanilla SGD Works, and Why Gradient Accumulation Is Wastefulcited by 41
  12. 2025Structured Preconditioners in Adaptive Optimization: A Unified Analysiscited by 33
  13. 2025Accelerating Neural Network Training: An Analysis of the AlgoPerf Competitioncited by 32
  14. 2024Understanding Adam Optimizer via Online Learning of Updates: Adam is FTRL in Disguisecited by 26
  15. 2025Purifying Shampoo: Investigating Shampoo's Heuristics by Decomposing its Preconditionercited by 23
  16. 2017A Unified Approach to Adaptive Regularization in Online and Stochastic Optimizationcited by 19
  17. 2024Random Scaling and Momentum for Non-smooth Non-convex Optimizationcited by 19
  18. 2025A Stable Whitening Optimizer for Efficient Neural Network Trainingcited by 18
  19. 2025Understanding and Improving Shampoo and SOAP via Kullback-Leibler Minimizationcited by 16
  20. 2023On the Parameterization of Second-Order Optimization Effective Towards the Infinite Widthcited by 15
  21. 2024General framework for online-to-nonconvex conversion: Schedule-free SGD is also effective for nonconvex optimizationcited by 15
  22. 2024Online Linear Regression in Dynamic Environments via Discountingcited by 10
  23. 2024Exact, Tractable Gauss-Newton Optimization in Deep Reversible Architectures Reveal Poor Generalizationcited by 4

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2026: 2.

2 papers.

  1. 2026Minimax Optimality of Shampoo for Nonstationary Stochastic Optimizationcited by 0
  2. 2026A Refresh Interval Does Not Fully Specify a Refresh Schedule: Learning-Rate-Dependent Placement in Kronecker-Preconditioned Language Model Trainingcited by 0