mlsanityData to 7 Oct 2026

Spectral-factorized Positive-definite Curvature Learning for NN Training

arXiv 2502.06268 ↗First posted 10 Feb 2025stat.ML cs.LGcited by 0

Not on the map. Why?

Abstract (arXiv)

Many training methods, such as Adam(W) and Shampoo, learn a positive-definite curvature matrix and apply an inverse root before preconditioning. Recently, non-diagonal training methods, such as Shampoo, have gained significant attention; however, they remain computationally inefficient and are limited to specific types of curvature information due to the costly matrix root computation via matrix decomposition. To address this, we propose a Riemannian optimization approach that dynamically adapts spectral-factorized positive-definite curvature estimates, enabling the efficient application of arbitrary matrix roots and generic curvature learning. We demonstrate the efficacy and versatility of our approach in positive-definite matrix optimization and covariance adaptation for gradient-free optimization, as well as its efficiency in curvature learning for neural net training.

Key facts. No citing papers in the map yet (data through 7 Oct 2026). Builds on 39 of its 67 references that are in the map. First posted on arXiv 10 Feb 2025.

Builds on

39 of 67 references.

References · 39

  1. 2017mixup: Beyond Empirical Risk Minimizationcited by 5,155
  2. 2019CutMix: Regularization Strategy to Train Strong Classifiers with Localizable Featurescited by 2,875
  3. 2017Mixed Precision Trainingcited by 1,135
  4. 2015Optimizing Neural Networks with Kronecker-factored Approximate Curvaturecited by 776
  5. 2023Symbolic Discovery of Optimization Algorithmscited by 423
  6. 2019Optimizing Millions of Hyperparameters by Implicit Differentiationcited by 303
  7. 2023Hiera: A Hierarchical Vision Transformer without the Bells-and-Whistlescited by 287
  8. 2023Sophia: A Scalable Stochastic Second-order Optimizer for Language Model Pre-trainingcited by 260
  9. 2023Controlling Text-to-Image Diffusion by Orthogonal Finetuningcited by 207
  10. 2024SOAP: Improving and Stabilizing Shampoo using Adamcited by 179
  11. 2024Old Optimizer, New Norm: An Anthologycited by 173
  12. 2019Practical Deep Learning with Bayesian Principlescited by 158
27 more
  1. 2020Momentum Improves Normalized SGDcited by 136
  2. 2018Fast and Scalable Bayesian Deep Learning by Weight-Perturbation in Adamcited by 131
  3. 2024Why Transformers Need Adam: A Hessian Perspectivecited by 123
  4. 2013Manopt, a Matlab toolbox for optimization on manifoldscited by 112
  5. 2019On Empirical Comparisons of Optimizers for Deep Learningcited by 99
  6. 2022Better plain ViT baselines for ImageNet-1kcited by 97
  7. 2021The Bayesian Learning Rulecited by 95
  8. 2017Noisy Natural Gradient as Variational Inferencecited by 94
  9. 2018Closing the Generalization Gap of Adaptive Gradient Methods in Training Deep Neural Networkscited by 89
  10. 2020Efficient Riemannian Optimization on the Stiefel Manifold via the Cayley Transformcited by 79
  11. 2023A Distributed Data-Parallel PyTorch Implementation of the Distributed Shampoo Optimizer for Training Neural Networks At-Scalecited by 69
  12. 2024Variational Learning is Effective for Large Deep Networkscited by 62
  13. 2019Fast and Simple Natural-Gradient Variational Inference with Mixture of Exponential-family Approximationscited by 51
  14. 2015Preconditioned Stochastic Gradient Descentcited by 48
  15. 2020Orthogonal Over-Parameterized Trainingcited by 44
  16. 2018Natural Gradients in Practice: Non-Conjugate Variational Inference in Gaussian Process Modelscited by 38
  17. 2021On Riemannian Optimization over Positive Definite Matrices with the Bures-Wasserstein Geometrycited by 32
  18. 2020Handling the Positive-Definite Constraint in the Bayesian Learning Rulecited by 31
  19. 2024Can We Remove the Square-Root in Adaptive Gradient Methods? A Second-Order Perspectivecited by 25
  20. 2021Tensor Normal Training for Deep Learning Modelscited by 24
  21. 2021Tractable structured natural gradient descent using local parameterizationscited by 22
  22. 2022Invariance Properties of the Natural Gradient in Overparametrised Systemscited by 11
  23. 2022Momentum Stiefel Optimizer, with Applications to Suitably-Orthogonal Attention, and Optimal Transportcited by 10
  24. 2023Simplifying Momentum-based Positive-definite Submanifold Optimization with Applications to Deep Learningcited by 10
  25. 2019Variational Bayes on Manifoldscited by 9
  26. 2024Efficient, Multimodal, and Derivative-Free Bayesian Inference With Fisher-Rao Gradient Flowscited by 9
  27. 2023Natural Gradient Variational Bayes without Fisher Matrix Analytic Calculation and Its Inversioncited by 3

Built on by

Not cited yet.