# On the Natural Gradient of the Evidence Lower Bound

arXiv: https://arxiv.org/abs/2307.11249 · HTML: https://mlsanity.com/paper/2307.11249/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Builds on 8 of its 29 references that are in the map.
- First posted on arXiv 20 Jul 2023.

## Builds on (most cited first, 8 of 8)

- [New insights and perspectives on the natural gradient method](https://mlsanity.com/paper/1412.1193/) (2014)
- [R'enyi Divergence Variational Inference](https://mlsanity.com/paper/1602.02311/) (2016)
- [Tighter Variational Bounds are Not Necessarily Better](https://mlsanity.com/paper/1802.04537/) (2018)
- [Riemannian metrics for neural networks I: feedforward networks](https://mlsanity.com/paper/1303.0818/) (2013)
- [Faithful Inversion of Generative Models for Effective Amortized Inference](https://mlsanity.com/paper/1712.00287/) (2017)
- [Mind the Gap when Conditioning Amortised Inference in Sequential Latent-Variable Models](https://mlsanity.com/paper/2101.07046/) (2021)
- [Invariance Properties of the Natural Gradient in Overparametrised Systems](https://mlsanity.com/paper/2206.15273/) (2022)
- [Inversion of Bayesian Networks](https://mlsanity.com/paper/2212.10649/) (2022)

## Built on by (most cited first, 1 of 1)

- [Convergence Properties of Natural Gradient Descent for Minimizing KL Divergence](https://mlsanity.com/paper/2504.19259/) (2025)

## Abstract (arXiv)

This article studies the Fisher-Rao gradient, also referred to as the natural gradient, of the evidence lower bound (ELBO) which plays a central role in generative machine learning. It reveals that the gap between the evidence and its lower bound, the ELBO, has essentially a vanishing natural gradient within unconstrained optimization. As a result, maximization of the ELBO is equivalent to minimization of the Kullback-Leibler divergence from a target distribution, the primary objective function of learning. Building on this insight, we derive a condition under which this equivalence persists even when optimization is constrained to a model. This condition yields a geometric characterization, which we formalize through the notion of a cylindrical model.