# Gradient Flows for Sampling: Mean-Field Models, Gaussian Approximations and Affine Invariance

arXiv: https://arxiv.org/abs/2302.11024 · HTML: https://mlsanity.com/paper/2302.11024/

## Key facts

- Cited by 16 papers in the map (data through 14 Oct 2026).
- Builds on 39 of its 119 references that are in the map.
- First posted on arXiv 21 Feb 2023.

## Builds on (most cited first, 30 of 39)

- [Variational Inference: A Review for Statisticians](https://mlsanity.com/paper/1601.00670/) (2016)
- [Stein Variational Gradient Descent: A General Purpose Bayesian Inference Algorithm](https://mlsanity.com/paper/1608.04471/) (2016)
- [New insights and perspectives on the natural gradient method](https://mlsanity.com/paper/1412.1193/) (2014)
- [On the Global Convergence of Gradient Descent for Over-parameterized Models using Optimal Transport](https://mlsanity.com/paper/1805.09545/) (2018)
- [Sliced Score Matching: A Scalable Approach to Density and Score Estimation](https://mlsanity.com/paper/1905.07088/) (2019)
- [Stein Variational Gradient Descent as Gradient Flow](https://mlsanity.com/paper/1704.07520/) (2017)
- [Fast and Scalable Bayesian Deep Learning by Weight-Perturbation in Adam](https://mlsanity.com/paper/1806.04854/) (2018)
- [Solving and Learning Nonlinear PDEs with Gaussian Processes](https://mlsanity.com/paper/2103.12959/) (2021)
- [Interacting particle solutions of Fokker-Planck equations through gradient-log-density estimation](https://mlsanity.com/paper/2006.00702/) (2020)
- [On the geometry of Stein variational gradient descent](https://mlsanity.com/paper/1912.00894/) (2019)
- [Fast Convergence of Natural Gradient Descent for Overparameterized Neural Networks](https://mlsanity.com/paper/1905.10961/) (2019)
- [Conjugate-Computation Variational Inference : Converting Variational Inference in Non-Conjugate Models to Inferences in Conjugate Models](https://mlsanity.com/paper/1703.04265/) (2017)
- [Mirrored Langevin Dynamics](https://mlsanity.com/paper/1802.10174/) (2018)
- [A Non-Asymptotic Analysis for Stein Variational Gradient Descent](https://mlsanity.com/paper/2006.09797/) (2020)
- [Wasserstein Control of Mirror Langevin Monte Carlo](https://mlsanity.com/paper/2002.04363/) (2020)
- [Earth System Modeling 2.0: A Blueprint for Models That Learn From Observations and Targeted High-Resolution Simulations](https://mlsanity.com/paper/1709.00037/) (2017)
- [Fast and Simple Natural-Gradient Variational Inference with Mixture of Exponential-family Approximations](https://mlsanity.com/paper/1906.02914/) (2019)
- [A Stein variational Newton method](https://mlsanity.com/paper/1806.03085/) (2018)
- [Probability flow solution of the Fokker-Planck equation](https://mlsanity.com/paper/2206.04642/) (2022)
- [Accelerated Information Gradient flow](https://mlsanity.com/paper/1909.02102/) (2019)
- [Dimension-independent likelihood-informed MCMC](https://mlsanity.com/paper/1411.3688/) (2014)
- [Exponential ergodicity of mirror-Langevin diffusions](https://mlsanity.com/paper/2005.09669/) (2020)
- [Scaling limit of the Stein variational gradient descent: the mean field regime](https://mlsanity.com/paper/1805.04035/) (2018)
- [Projected Stein Variational Gradient Descent](https://mlsanity.com/paper/2002.03469/) (2020)
- [Online Natural Gradient as a Kalman Filter](https://mlsanity.com/paper/1703.00209/) (2017)
- [Information Newton's flow: second-order optimization method in probability space](https://mlsanity.com/paper/2001.04341/) (2020)
- [Projected Wasserstein gradient descent for high-dimensional Bayesian inference](https://mlsanity.com/paper/2102.06350/) (2021)
- [Ensemble preconditioning for Markov chain Monte Carlo simulation](https://mlsanity.com/paper/1607.03954/) (2016)
- [Optimal transport natural gradient for statistical manifolds with continuous sample space](https://mlsanity.com/paper/1805.08380/) (2018)
- [Scalable and efficient algorithms for the propagation of uncertainty from data through inference to prediction for large-scale problems, with application to flow of the Antarctic ice sheet](https://mlsanity.com/paper/1410.1221/) (2014)

## Built on by (most cited first, 16 of 16)

- [Sampling in Unit Time with Kernel Fisher-Rao Flow](https://mlsanity.com/paper/2401.03892/) (2024)
- [Sampling via Gradient Flows in the Space of Probability Measures](https://mlsanity.com/paper/2310.03597/) (2023)
- [Repulsive Latent Score Distillation for Solving Inverse Problems](https://mlsanity.com/paper/2406.16683/) (2024)
- [Towards Understanding the Dynamics of Gaussian-Stein Variational Gradient Descent](https://mlsanity.com/paper/2305.14076/) (2023)
- [Stein transport for Bayesian inference](https://mlsanity.com/paper/2409.01464/) (2024)
- [Efficient, Multimodal, and Derivative-Free Bayesian Inference With Fisher-Rao Gradient Flows](https://mlsanity.com/paper/2406.17263/) (2024)
- [Kernel Approximation of Fisher-Rao Gradient Flows](https://mlsanity.com/paper/2410.20622/) (2024)
- [Sequential Monte Carlo approximations of Wasserstein--Fisher--Rao gradient flows](https://mlsanity.com/paper/2506.05905/) (2025)
- [Inclusive KL Gradient Flows: Otto-Wasserstein, Fisher-Rao-Gaussian, and Local-Estimator Dynamics](https://mlsanity.com/paper/2411.00214/) (2024)
- [Evolution of Gaussians in the Hellinger-Kantorovich-Boltzmann gradient flow](https://mlsanity.com/paper/2504.20400/) (2025)
- [Optimization Guarantees for Square-Root Natural-Gradient Variational Inference](https://mlsanity.com/paper/2507.07853/) (2025)
- [Towards understanding Accelerated Stein Variational Gradient Flow -- Analysis of Generalized Bilinear Kernels for Gaussian target distributions](https://mlsanity.com/paper/2509.04008/) (2025)
- [To discretize continually: Mean shift interacting particle systems for Bayesian inference](https://mlsanity.com/paper/2605.14142/) (2026)
- [Kuramoto Oscillators and Swarms on Manifolds for Geometry Informed Machine Learning](https://mlsanity.com/paper/2405.09453/) (2024)
- [Learning Paths for Dynamic Measure Transport: A Control Perspective](https://mlsanity.com/paper/2511.03797/) (2025)
- [Radon--Wasserstein Gradient Flows for Interacting-Particle Sampling in High Dimensions](https://mlsanity.com/paper/2602.05227/) (2026)

## Abstract (arXiv)

Sampling a probability distribution with an unknown normalization constant is a fundamental problem in computational science and engineering. This task may be cast as an optimization problem over all probability measures, and an initial distribution can be evolved to the desired minimizer dynamically via gradient flows. Mean-field models, whose law is governed by the gradient flow in the space of probability measures, may also be identified; particle approximations of these mean-field models form the basis of algorithms. The gradient flow approach is also the basis of algorithms for variational inference, in which the optimization is performed over a parameterized family of probability distributions such as Gaussians, and the underlying gradient flow is restricted to the parameterized family. By choosing different energy functionals and metrics for the gradient flow, different algorithms with different convergence properties arise. In this paper, we concentrate on the Kullback-Leibler divergence after showing that, up to scaling, it has the unique property that the gradient flows resulting from this choice of energy do not depend on the normalization constant. For the metrics, we focus on variants of the Fisher-Rao, Wasserstein, and Stein metrics; we introduce the affine invariance property for gradient flows, and their corresponding mean-field models, determine whether a given metric leads to affine invariance, and modify it to make it affine invariant if it does not. We study the resulting gradient flows in both probability density space and Gaussian space. The flow in the Gaussian space may be understood as a Gaussian approximation of the flow. We demonstrate that the Gaussian approximation based on the metric and through moment closure coincide, establish connections between them, and study their long-time convergence properties showing the advantages of affine invariance.