Data to 14 Oct 2026

New insights and perspectives on the natural gradient method

arXiv 1412.1193 ↗First posted 3 Dec 2014cs.LG stat.MLcited by 443

Not on the map. Why?

Cited by 33 in LLM safety and security, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

Natural gradient descent is an optimization method traditionally motivated from the perspective of information geometry, and works well for many applications as an alternative to stochastic gradient descent. In this paper we critically analyze this method and its properties, and show how it can be viewed as a type of 2nd-order optimization method, with the Fisher information matrix acting as a substitute for the Hessian. In many important cases, the Fisher information matrix is shown to be equivalent to the Generalized Gauss-Newton matrix, which both approximates the Hessian, but also has certain properties that favor its use over the Hessian. This perspective turns out to have significant implications for the design of a practical and robust natural gradient optimizer, as it motivates the use of techniques like trust regions and Tikhonov regularization. Additionally, we make a series of contributions to the understanding of natural gradient and 2nd-order methods, including: a thorough analysis of the convergence speed of stochastic natural gradient descent (and more general stochastic 2nd-order methods) as applied to convex quadratics, a critical examination of the oft-used "empirical" approximation of the Fisher matrix, and an analysis of the (approximate) parameterization invariance property possessed by natural gradient methods (which we show also holds for certain other curvature, but notably not the Hessian).

Key facts. Cited by 443 papers in the map (data through 14 Oct 2026). First posted on arXiv 3 Dec 2014.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2021: 51; 2022: 44; 2023: 61; 2024: 87; 2025: 80; 2026: 120.

443 papers.

Sort:

By area:

2021Laplace Redux -- Effortless Bayesian Deep Learningcited by 365
2022A Survey of Machine Unlearningcited by 312
2023TRAK: Attributing Model Behavior at Scalecited by 283
2023Sophia: A Scalable Stochastic Second-order Optimizer for Language Model Pre-trainingcited by 265
2021Training Neural Networks with Fixed Sparse Maskscited by 214
2021Fishr: Invariant Gradient Variances for Out-of-Distribution Generalizationcited by 186
2022If Influence Functions are the Answer, Then What is the Question?cited by 144
2021Federated Unlearning via Class-Discriminative Pruningcited by 115
2021The Bayesian Learning Rulecited by 99
2023Machine Unlearning: Solutions and Challengescited by 93
2022Fisher SAM: Information Geometry and Sharpness Aware Minimisationcited by 84
2023A Distributed Data-Parallel PyTorch Implementation of the Distributed Shampoo Optimizer for Training Neural Networks At-Scalecited by 77

38 more

2023Model Merging by Uncertainty-Based Gradient Matchingcited by 77
2022An Improved Analysis of (Variance-Reduced) Policy Gradient and Natural Policy Gradient Methodscited by 68
2022On the Identifiability and Estimation of Causal Location-Scale Noise Modelscited by 56
2021M-FAC: Efficient Matrix-Free Approximations of Second-Order Informationcited by 54
2021Noether's Learning Dynamics: Role of Symmetry Breaking in Neural Networkscited by 50
2023Kronecker-Factored Approximate Curvature for Modern Neural Network Architecturescited by 49
2023Intriguing Properties of Data Attribution on Diffusion Modelscited by 47
2024Categorical Flow Matching on Statistical Manifoldscited by 47
2021Natural continual learning: success is a journey, not (just) a destinationcited by 46
2022Linear Convergence of Natural Policy Gradient Methods with Log-Linear Policiescited by 46
2021Posterior Meta-Replay for Continual Learningcited by 45
2024Unified Gradient-Based Machine Unlearning with Remain Geometry Enhancementcited by 45
2021Certifiable Machine Unlearning for Linear Modelscited by 42
2022Invariance Learning in Deep Neural Networks with Differentiable Laplace Approximationscited by 42
2024Differentiation and Specialization of Attention Heads via the Refined Local Learning Coefficientcited by 40
2021Strength of Minibatch Noise in SGDcited by 39
2022Evaluating Machine Unlearning via Epistemic Uncertaintycited by 39
2021The Information Geometry of Unsupervised Reinforcement Learningcited by 37
2024Elastic Feature Consolidation for Cold Start Exemplar-Free Incremental Learningcited by 37
2023Merging by Matching Models in Task Parameter Subspacescited by 36
2024Training Data Attribution via Approximate Unrolled Differentiationcited by 34
2024Influence Functions for Scalable Data Attribution in Diffusion Modelscited by 33
2024A Review on Machine Unlearningcited by 32
2021Local Adaptivity in Federated Learning: Convergence and Consistencycited by 31
2022Gradient Descent on Neurons and its Link to Approximate Second-Order Optimizationcited by 30
2023Elastic Weight Removal for Faithful and Abstractive Dialogue Generationcited by 29
2024Corrective Machine Unlearningcited by 29
2021PAC-Bayes Information Bottleneckcited by 28
2022Particle algorithms for maximum likelihood training of latent variable modelscited by 26
2024Can We Remove the Square-Root in Adaptive Gradient Methods? A Second-Order Perspectivecited by 25
2021Tensor Normal Training for Deep Learning Modelscited by 24
2022PipeFisher: Efficient Training of Large Language Models Using Pipelining and Fisher Information Matricescited by 24
2021Pulling back information geometrycited by 23
2021Avoiding Inference Heuristics in Few-shot Prompt-based Finetuningcited by 23
2023Learning Layer-wise Equivariances Automatically using Gradientscited by 23
2024Super Consistency of Neural Network Landscapes and Learning Rate Transfercited by 23
2025Purifying Shampoo: Investigating Shampoo's Heuristics by Decomposing its Preconditionercited by 23
2025Stable Gradients for Stable Learning at Scale in Deep Reinforcement Learningcited by 23

50 most cited of 443 All 443 (JSON)