mlsanityData to 7 Oct 2026

Tractable structured natural gradient descent using local parameterizations

arXiv 2102.07405 ↗First posted 15 Feb 2021stat.ML cs.LGcited by 22

Not on the map. Why?

Abstract (arXiv)

Natural-gradient descent (NGD) on structured parameter spaces (e.g., low-rank covariances) is computationally challenging due to difficult Fisher-matrix computations. We address this issue by using local-parameter coordinates to obtain a flexible and efficient NGD method that works well for a wide-variety of structured parameterizations. We show four applications where our method (1) generalizes the exponential natural evolutionary strategy, (2) recovers existing Newton-like algorithms, (3) yields new structured second-order algorithms via matrix groups, and (4) gives new algorithms to learn covariances of Gaussian and Wishart-based distributions. We show results on a range of problems from deep learning, variational inference, and evolution strategies. Our work opens a new direction for scalable structured geometric methods.

Key facts. Cited by 22 papers in the map (data through 7 Oct 2026). Builds on 20 of its 64 references that are in the map. First posted on arXiv 15 Feb 2021.

Builds on

20 of 64 references.

References · 20

  1. 2016Gaussian Error Linear Units (GELUs)cited by 3,431
  2. 2016Asynchronous Methods for Deep Reinforcement Learningcited by 2,218
  3. 2013Black Box Variational Inferencecited by 317
  4. 2019Practical Deep Learning with Bayesian Principlescited by 158
  5. 2018Fast and Scalable Bayesian Deep Learning by Weight-Perturbation in Adamcited by 131
  6. 2017Noisy Natural Gradient as Variational Inferencecited by 94
  7. 2018Implicit Reparameterization Gradientscited by 84
  8. 2019Trivializations for Gradient-Based Optimization on Manifoldscited by 78
  9. 2017Conjugate-Computation Variational Inference : Converting Variational Inference in Non-Conjugate Models to Inferences in Conjugate Modelscited by 65
  10. 2019Fast and Simple Natural-Gradient Variational Inference with Mixture of Exponential-family Approximationscited by 51
  11. 2018Natural Gradients in Practice: Non-Conjugate Variational Inference in Gaussian Process Modelscited by 38
  12. 2018Fast yet Simple Natural-Gradient Descent for Variational Inference in Complex Modelscited by 37
8 more
  1. 2018Large-Scale Distributed Second-Order Optimization Using Kronecker-Factored Approximate Curvature for Deep Convolutional Neural Networkscited by 37
  2. 2015On Graduated Optimization for Stochastic Non-Convex Problemscited by 34
  3. 2018SLANG: Fast Structured Covariance Approximations for Bayesian Deep Learning with Natural Gradientcited by 34
  4. 2020Handling the Positive-Definite Constraint in the Bayesian Learning Rulecited by 31
  5. 2019Stein's Lemma for the Reparameterization Trick with Exponential Family Mixturescited by 18
  6. 2019Modular Block-diagonal Curvature Approximations for Feedforward Architecturescited by 13
  7. 2017Variational Adaptive-Newton Method for Explorative Learningcited by 7
  8. 2018EA-CG: An Approximate Second-Order Method for Training Fully-Connected Neural Networkscited by 1

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 5; 2022: 3; 2023: 6; 2024: 3; 2025: 4; 2026: 1.

22 papers.

Sort:

  1. 2021The Bayesian Learning Rulecited by 95
  2. 2024Can We Remove the Square-Root in Adaptive Gradient Methods? A Second-Order Perspectivecited by 25
  3. 2022A Unified Perspective on Natural Gradient Variational Inference with Gaussian Mixture Modelscited by 22
  4. 2023The Geometry of Neural Nets' Parameter Spaces Under Reparametrizationcited by 16
  5. 2022Invariance Properties of the Natural Gradient in Overparametrised Systemscited by 11
  6. 2023Simplifying Momentum-based Positive-definite Submanifold Optimization with Applications to Deep Learningcited by 10
  7. 2023Structured Inverse-Free Natural Gradient: Memory-Efficient & Numerically-Stable KFACcited by 9
  8. 2024Understanding Stochastic Natural Gradient Variational Inferencecited by 8
  9. 2021Structured second-order methods via natural gradient descentcited by 7
  10. 2021Nys-Newton: Nystr"om-Approximated Curvature for Stochastic Optimizationcited by 4
  11. 2021PredProp: Bidirectional Stochastic Optimization with Precision Weighted Predictive Codingcited by 3
  12. 2023The Lie-Group Bayesian Learning Rulecited by 3
10 more
  1. 2025Optimization Guarantees for Square-Root Natural-Gradient Variational Inferencecited by 3
  2. 2025Information Geometry of Variational Bayescited by 2
  3. 2021Towards Modeling and Resolving Singular Parameter Spaces using Stratifoldscited by 1
  4. 2023Variational Bayes Made Easycited by 1
  5. 2026SOAP-Bubbles: Structured Weight Uncertainty for Neural Networkscited by 1
  6. 2022Characterizing Learning Dynamics under Relative Reparameterization of Singular Modelscited by 0
  7. 2023Natural Gradient Hybrid Variational Inference with Application to Deep Mixed Modelscited by 0
  8. 2024Approximated Orthogonal Projection Unit: Stabilizing Regression Network Training Using Natural Gradientcited by 0
  9. 2025Spectral-factorized Positive-definite Curvature Learning for NN Trainingcited by 0
  10. 2025Deterministic Bounds and Random Estimates of Metric Tensors on Neuromanifoldscited by 0