mlsanityData to 7 Oct 2026

Simplifying Momentum-based Positive-definite Submanifold Optimization with Applications to Deep Learning

arXiv 2302.09738 ↗First posted 20 Feb 2023stat.ML cs.LGcited by 10

Not on the map. Why?

Abstract (arXiv)

Riemannian submanifold optimization with momentum is computationally challenging because, to ensure that the iterates remain on the submanifold, we often need to solve difficult differential equations. Here, we simplify such difficulties for a class of sparse or structured symmetric positive-definite matrices with the affine-invariant metric. We do so by proposing a generalized version of the Riemannian normal coordinates that dynamically orthonormalizes the metric and locally converts the problem into an unconstrained problem in the Euclidean space. We use our approach to simplify existing approaches for structured covariances and develop matrix-inverse-free 2nd-order optimizers for deep learning with low precision by using only matrix multiplications. Code: https://github.com/yorkerlin/StructuredNGD-DL

Key facts. Cited by 10 papers in the map (data through 7 Oct 2026). Builds on 22 of its 43 references that are in the map. First posted on arXiv 20 Feb 2023.

Builds on

22 of 43 references.

References · 22

  1. 2014Very Deep Convolutional Networks for Large-Scale Image Recognitioncited by 15,267
  2. 2020Designing Network Design Spacescited by 954
  3. 2015Optimizing Neural Networks with Kronecker-factored Approximate Curvaturecited by 776
  4. 2021RepVGG: Making VGG-style ConvNets Great Againcited by 520
  5. 2023Symbolic Discovery of Optimization Algorithmscited by 423
  6. 2017The Marginal Value of Adaptive Gradient Methods in Machine Learningcited by 248
  7. 2022Patches Are All You Need?cited by 201
  8. 2016Deep Pyramidal Residual Networkscited by 163
  9. 2016A Kronecker-factored approximate Fisher matrix for convolution layerscited by 138
  10. 2019Trivializations for Gradient-Based Optimization on Manifoldscited by 78
  11. 2017Conjugate-Computation Variational Inference : Converting Variational Inference in Non-Conjugate Models to Inferences in Conjugate Modelscited by 65
  12. 2021Fast and Accurate Model Scalingcited by 46
10 more
  1. 2020Rethinking Channel Dimensions for Efficient Model Designcited by 41
  2. 2021On Riemannian Optimization over Positive Definite Matrices with the Bures-Wasserstein Geometrycited by 32
  3. 2019A Continuous-time Perspective for Modeling Acceleration in Riemannian Optimizationcited by 25
  4. 2015Riemannian Dictionary Learning and Sparse Coding for Positive Definite Matricescited by 24
  5. 2020From Nesterov's Estimate Sequence to Riemannian Accelerationcited by 24
  6. 2021Tractable structured natural gradient descent using local parameterizationscited by 22
  7. 2021Structured second-order methods via natural gradient descentcited by 7
  8. 2015Regularization-free estimation in trace regression with symmetric positive semidefinite matricescited by 4
  9. 2020Adaptive and Momentum Methods on Manifolds Through Trivializationscited by 4
  10. 2021Vector Transport Free Riemannian LBFGS for Optimization on Symmetric Positive Definite Matrix Manifoldscited by 1

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 2; 2024: 2; 2025: 5; 2026: 1.

10 papers.

  1. 2023Kronecker-Factored Approximate Curvature for Modern Neural Network Architecturescited by 49
  2. 2024Can We Remove the Square-Root in Adaptive Gradient Methods? A Second-Order Perspectivecited by 25
  3. 2025Understanding and Improving Shampoo and SOAP via Kullback-Leibler Minimizationcited by 13
  4. 2023Structured Inverse-Free Natural Gradient: Memory-Efficient & Numerically-Stable KFACcited by 9
  5. 2025Optimization Guarantees for Square-Root Natural-Gradient Variational Inferencecited by 3
  6. 2025MAC: An Efficient Gradient Preconditioning using Mean Activation Approximated Curvaturecited by 1
  7. 2024Variational Stochastic Gradient Descent for Deep Neural Networkscited by 0
  8. 2025Spectral-factorized Positive-definite Curvature Learning for NN Trainingcited by 0
  9. 2025SVRG and Beyond via Posterior Correctioncited by 0
  10. 2026Classifiers in High Dimensional Hilbert Metricscited by 0