# Is All Learning (Natural) Gradient Descent?

arXiv: https://arxiv.org/abs/2409.16422 · HTML: https://mlsanity.com/paper/2409.16422/

## Key facts

- Cited by 2 papers in the map (data through 14 Oct 2026).
- Builds on 14 of its 31 references that are in the map.
- First posted on arXiv 24 Sep 2024.

## Builds on (most cited first, 14 of 14)

- [Introduction to Online Convex Optimization](https://mlsanity.com/paper/1909.05207/) (2019)
- [Optimizing Neural Networks with Kronecker-factored Approximate Curvature](https://mlsanity.com/paper/1503.05671/) (2015)
- [New insights and perspectives on the natural gradient method](https://mlsanity.com/paper/1412.1193/) (2014)
- [Direct Feedback Alignment Provides Learning in Deep Neural Networks](https://mlsanity.com/paper/1609.01596/) (2016)
- [Revisiting Natural Gradient for Deep Networks](https://mlsanity.com/paper/1301.3584/) (2013)
- [Direct Feedback Alignment Scales to Modern Deep Learning Tasks and Architectures](https://mlsanity.com/paper/2006.12878/) (2020)
- [A Dynamical Systems Perspective on Nesterov Acceleration](https://mlsanity.com/paper/1905.07436/) (2019)
- [Kronecker-Factored Approximate Curvature for Physics-Informed Neural Networks](https://mlsanity.com/paper/2405.15603/) (2024)
- [Beyond Convexity -- Contraction and Global Convergence of Gradient Descent](https://mlsanity.com/paper/1806.06655/) (2018)
- [On the choice of metric in gradient-based theories of brain function](https://mlsanity.com/paper/1805.11851/) (2018)
- [Implicit Regularization and Momentum Algorithms in Nonlinearly Parameterized Adaptive Control and Prediction](https://mlsanity.com/paper/1912.13154/) (2019)
- [Variational Principles for Mirror Descent and Mirror Langevin Dynamics](https://mlsanity.com/paper/2303.09532/) (2023)
- [Synaptic Weight Distributions Depend on the Geometry of Plasticity](https://mlsanity.com/paper/2305.19394/) (2023)
- [Desiderata for normative models of synaptic plasticity](https://mlsanity.com/paper/2308.04988/) (2023)

## Built on by (most cited first, 2 of 2)

- [Gradient Descent as Loss Landscape Navigation: a Normative Framework for Deriving Learning Rules](https://mlsanity.com/paper/2510.26997/) (2025)
- [Curl Descent: Non-Gradient Learning Dynamics with Sign-Diverse Plasticity](https://mlsanity.com/paper/2510.02765/) (2025)

## Abstract (arXiv)

This paper shows that a wide class of effective learning rules -- those that improve a scalar performance measure over a given time window -- can be rewritten as natural gradient descent with respect to a suitably defined loss function and metric. Specifically, we show that parameter updates within this class of learning rules can be expressed as the product of a symmetric positive definite matrix (i.e., a metric) and the negative gradient of a loss function. We also demonstrate that these metrics have a canonical form and identify several optimal ones, including the metric that achieves the minimum possible condition number. The proofs of the main results are straightforward, relying only on elementary linear algebra and calculus, and are applicable to continuous-time, discrete-time, stochastic, and higher-order learning rules, as well as loss functions that explicitly depend on time.