# True Asymptotic Natural Gradient Optimization

arXiv: https://arxiv.org/abs/1712.08449 · HTML: https://mlsanity.com/paper/1712.08449/

## Key facts

- Cited by 5 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 22 Dec 2017.

## Built on by (most cited first, 5 of 5)

- [Gradient Descent on Neurons and its Link to Approximate Second-Order Optimization](https://mlsanity.com/paper/2201.12250/) (2022)
- [FISMO: Fisher-Structured Momentum-Orthogonalized Optimizer](https://mlsanity.com/paper/2601.21750/) (2026)
- [Growing Tiny Networks: Spotting Expressivity Bottlenecks and Fixing Them Optimally](https://mlsanity.com/paper/2405.19816/) (2024)
- [ANaGRAM: A Natural Gradient Relative to Adapted Model for efficient PINNs learning](https://mlsanity.com/paper/2412.10782/) (2024)
- [Distribution-Aware Tensor Decomposition for Compression of Convolutional Neural Networks](https://mlsanity.com/paper/2511.04494/) (2025)

## Abstract (arXiv)

We introduce a simple algorithm, True Asymptotic Natural Gradient Optimization (TANGO), that converges to a true natural gradient descent in the limit of small learning rates, without explicit Fisher matrix estimation. For quadratic models the algorithm is also an instance of averaged stochastic gradient, where the parameter is a moving average of a "fast", constant-rate gradient descent. TANGO appears as a particular de-linearization of averaged SGD, and is sometimes quite different on non-quadratic models. This further connects averaged SGD and natural gradient, both of which are arguably optimal asymptotically. In large dimension, small learning rates will be required to approximate the natural gradient well. Still, this shows it is possible to get arbitrarily close to exact natural gradient descent with a lightweight algorithm.