mlsanityData to 14 Oct 2026

Invariance Properties of the Natural Gradient in Overparametrised Systems

arXiv 2206.15273 ↗First posted 30 Jun 2022cs.LG math.ST stat.THcited by 11

Not on the map. Why?

Abstract (arXiv)

The natural gradient field is a vector field that lives on a model equipped with a distinguished Riemannian metric, e.g. the Fisher-Rao metric, and represents the direction of steepest ascent of an objective function on the model with respect to this metric. In practice, one tries to obtain the corresponding direction on the parameter space by multiplying the ordinary gradient by the inverse of the Gram matrix associated with the metric. We refer to this vector on the parameter space as the natural parameter gradient. In this paper we study when the pushforward of the natural parameter gradient is equal to the natural gradient. Furthermore we investigate the invariance properties of the natural parameter gradient. Both questions are addressed in an overparametrised setting.

Key facts. Cited by 11 papers in the map (data through 14 Oct 2026). Builds on 7 of its 20 references that are in the map. First posted on arXiv 30 Jun 2022.

Builds on

7 of 20 references.

References · 7

  1. 2017Improved Training of Wasserstein GANscited by 2,235
  2. 2014New insights and perspectives on the natural gradient methodcited by 443
  3. 2016A Kronecker-factored approximate Fisher matrix for convolution layerscited by 141
  4. 2019Fast Convergence of Natural Gradient Descent for Overparameterized Neural Networkscited by 76
  5. 2013Riemannian metrics for neural networks I: feedforward networkscited by 30
  6. 2021Tractable structured natural gradient descent using local parameterizationscited by 22
  7. 2018Natural gradient via optimal transportcited by 20

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2022: 1; 2023: 3; 2024: 2; 2025: 3; 2026: 2.

11 papers.

  1. 2023The Geometry of Neural Nets' Parameter Spaces Under Reparametrizationcited by 16
  2. 2022Geometry and convergence of natural policy gradient methodscited by 12
  3. 2026Curvature-Aware Optimization for High-Accuracy Physics-Informed Neural Networkscited by 11
  4. 2023Achieving High Accuracy with PINNs via Energy Natural Gradientscited by 10
  5. 2024Embedding Safety into RL: A New Take on Trust Region Methodscited by 10
  6. 2024Fisher-Rao Gradient Flows of Linear Programs and State-Action Natural Policy Gradientscited by 5
  7. 2023On the Natural Gradient of the Evidence Lower Boundcited by 1
  8. 2025Approximation and learning with compositional tensor trainscited by 1
  9. 2025Spectral-factorized Positive-definite Curvature Learning for NN Trainingcited by 0
  10. 2025Convergence Properties of Natural Gradient Descent for Minimizing KL Divergencecited by 0
  11. 2026Energy Manifold Natural Gradient Descent: Riemannian Optimization for Neural PDE Solverscited by 0