# Accelerating Natural Gradient with Higher-Order Invariance

arXiv: https://arxiv.org/abs/1803.01273 · HTML: https://mlsanity.com/paper/1803.01273/

## Key facts

- Cited by 17 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 4 Mar 2018.

## Built on by (most cited first, 17 of 17)

- [Multi-Agent Generative Adversarial Imitation Learning](https://mlsanity.com/paper/1807.09936/) (2018)
- [Noether's Learning Dynamics: Role of Symmetry Breaking in Neural Networks](https://mlsanity.com/paper/2105.02716/) (2021)
- [Symmetry in Neural Network Parameter Spaces](https://mlsanity.com/paper/2506.13018/) (2025)
- [The Geometry of Neural Nets' Parameter Spaces Under Reparametrization](https://mlsanity.com/paper/2302.07384/) (2023)
- [On a continuous time model of gradient descent dynamics and instability in deep learning](https://mlsanity.com/paper/2302.01952/) (2023)
- [Understanding Stochastic Natural Gradient Variational Inference](https://mlsanity.com/paper/2406.01870/) (2024)
- [Riemannian Laplace Approximation with the Fisher Metric](https://mlsanity.com/paper/2311.02766/) (2023)
- [Dual Natural Gradient Descent for Scalable Training of Physics-Informed Neural Networks](https://mlsanity.com/paper/2505.21404/) (2025)
- [Depth Without the Magic: Inductive Bias of Natural Gradient Descent](https://mlsanity.com/paper/2111.11542/) (2021)
- [RTRA: Rapid Training of Regularization-based Approaches in Continual Learning](https://mlsanity.com/paper/2312.09361/) (2023)
- [Warped geometric information on the optimisation of Euclidean functions](https://mlsanity.com/paper/2308.08305/) (2023)
- [Learning Geometry and Topology via Multi-Chart Flows](https://mlsanity.com/paper/2505.24665/) (2025)
- [A Natural Primal-Dual Hybrid Gradient Method for Adversarial Neural Network Training on Solving Partial Differential Equations](https://mlsanity.com/paper/2411.06278/) (2024)
- [Improving Learning to Optimize Using Parameter Symmetries](https://mlsanity.com/paper/2504.15399/) (2025)
- [An Invariant Information Geometric Method for High-Dimensional Online Optimization](https://mlsanity.com/paper/2401.01579/) (2024)
- [Convergence Properties of Natural Gradient Descent for Minimizing KL Divergence](https://mlsanity.com/paper/2504.19259/) (2025)
- [Higher-Order Geometric Updates for Levenberg-Marquardt Method via Riemann Normal Coordinates](https://mlsanity.com/paper/2607.07623/) (2026)

## Abstract (arXiv)

An appealing property of the natural gradient is that it is invariant to arbitrary differentiable reparameterizations of the model. However, this invariance property requires infinitesimal steps and is lost in practical implementations with small but finite step sizes. In this paper, we study invariance properties from a combined perspective of Riemannian geometry and numerical differential equation solving. We define the order of invariance of a numerical method to be its convergence order to an invariant solution. We propose to use higher-order integrators and geodesic corrections to obtain more invariant optimization trajectories. We prove the numerical convergence properties of geodesic corrected updates and show that they can be as computationally efficient as plain natural gradient. Experimentally, we demonstrate that invariance leads to faster optimization and our techniques improve on traditional natural gradient in deep neural network training and natural policy gradient for reinforcement learning.