mlsanityData to 7 Oct 2026

Structured second-order methods via natural gradient descent

arXiv 2107.10884 ↗First posted 22 Jul 2021stat.ML cs.LGcited by 7

Not on the map. Why?

Abstract (arXiv)

In this paper, we propose new structured second-order methods and structured adaptive-gradient methods obtained by performing natural-gradient descent on structured parameter spaces. Natural-gradient descent is an attractive approach to design new algorithms in many settings such as gradient-free, adaptive-gradient, and second-order methods. Our structured methods not only enjoy a structural invariance but also admit a simple expression. Finally, we test the efficiency of our proposed methods on both deterministic non-convex problems and deep learning problems.

Key facts. Cited by 7 papers in the map (data through 7 Oct 2026). Builds on 10 of its 26 references that are in the map. First posted on arXiv 22 Jul 2021.

Builds on

10 of 26 references.

References · 10

  1. 2016Gaussian Error Linear Units (GELUs)cited by 3,431
  2. 2016Asynchronous Methods for Deep Reinforcement Learningcited by 2,218
  3. 2015Optimizing Neural Networks with Kronecker-factored Approximate Curvaturecited by 776
  4. 2018Shampoo: Preconditioned Stochastic Tensor Optimizationcited by 413
  5. 2019Limitations of the Empirical Fisher Approximation for Natural Gradient Descentcited by 213
  6. 2018Fast and Scalable Bayesian Deep Learning by Weight-Perturbation in Adamcited by 131
  7. 2015On Graduated Optimization for Stochastic Non-Convex Problemscited by 34
  8. 2020Handling the Positive-Definite Constraint in the Bayesian Learning Rulecited by 31
  9. 2021Tractable structured natural gradient descent using local parameterizationscited by 22
  10. 2017Variational Adaptive-Newton Method for Explorative Learningcited by 7

Built on by

Cited by year

Citing papers per year: 2021: 1; 2022: 1; 2023: 2; 2024: 2; 2025: 1.

7 papers.

  1. 2024Can We Remove the Square-Root in Adaptive Gradient Methods? A Second-Order Perspectivecited by 25
  2. 2023The Geometry of Neural Nets' Parameter Spaces Under Reparametrizationcited by 16
  3. 2023Simplifying Momentum-based Positive-definite Submanifold Optimization with Applications to Deep Learningcited by 10
  4. 2021Nys-Newton: Nystr"om-Approximated Curvature for Stochastic Optimizationcited by 4
  5. 2025Optimization Guarantees for Square-Root Natural-Gradient Variational Inferencecited by 3
  6. 2022Manifold Gaussian Variational Bayes on the Precision Matrixcited by 1
  7. 2024Variational Stochastic Gradient Descent for Deep Neural Networkscited by 0