# KALE Flow: A Relaxed KL Gradient Flow for Probabilities with Disjoint Support

arXiv: https://arxiv.org/abs/2106.08929 · HTML: https://mlsanity.com/paper/2106.08929/

## Key facts

- Cited by 37 papers in the map (data through 14 Oct 2026).
- Builds on 34 of its 68 references that are in the map.
- First posted on arXiv 16 Jun 2021.

## Builds on (most cited first, 30 of 34)

- [Improved Training of Wasserstein GANs](https://mlsanity.com/paper/1704.00028/) (2017)
- [Variational Inference: A Review for Statisticians](https://mlsanity.com/paper/1601.00670/) (2016)
- [Demystifying MMD GANs](https://mlsanity.com/paper/1801.01401/) (2018)
- [f-GAN: Training Generative Neural Samplers using Variational Divergence Minimization](https://mlsanity.com/paper/1606.00709/) (2016)
- [A Mean Field View of the Landscape of Two-Layers Neural Networks](https://mlsanity.com/paper/1804.06561/) (2018)
- [Towards Principled Methods for Training Generative Adversarial Networks](https://mlsanity.com/paper/1701.04862/) (2017)
- [On the Global Convergence of Gradient Descent for Over-parameterized Models using Optimal Transport](https://mlsanity.com/paper/1805.09545/) (2018)
- [Learning Generative Models with Sinkhorn Divergences](https://mlsanity.com/paper/1706.00292/) (2017)
- [Generative Moment Matching Networks](https://mlsanity.com/paper/1502.02761/) (2015)
- [MMD GAN: Towards Deeper Understanding of Moment Matching Network](https://mlsanity.com/paper/1705.08584/) (2017)
- [Non-asymptotic convergence analysis for the Unadjusted Langevin Algorithm](https://mlsanity.com/paper/1507.05021/) (2015)
- [Stein Variational Gradient Descent as Gradient Flow](https://mlsanity.com/paper/1704.07520/) (2017)
- [Training generative neural networks via Maximum Mean Discrepancy optimization](https://mlsanity.com/paper/1505.03906/) (2015)
- [Analysis of Langevin Monte Carlo via convex optimization](https://mlsanity.com/paper/1802.09188/) (2018)
- [Maximum Mean Discrepancy Gradient Flow](https://mlsanity.com/paper/1906.04370/) (2019)
- [The Cramer Distance as a Solution to Biased Wasserstein Gradients](https://mlsanity.com/paper/1705.10743/) (2017)
- [User-friendly guarantees for the Langevin Monte Carlo with inaccurate gradient](https://mlsanity.com/paper/1710.00095/) (2017)
- [Adversarial Variational Bayes: Unifying Variational Autoencoders and Generative Adversarial Networks](https://mlsanity.com/paper/1701.04722/) (2017)
- [Mean Field Analysis of Neural Networks: A Central Limit Theorem](https://mlsanity.com/paper/1808.09372/) (2018)
- [Generalized Energy Based Models](https://mlsanity.com/paper/2003.05033/) (2020)
- [Mirrored Langevin Dynamics](https://mlsanity.com/paper/1802.10174/) (2018)
- [Efficient constrained sampling via the mirror-Langevin algorithm](https://mlsanity.com/paper/2010.16212/) (2020)
- [Sampling from a log-concave distribution with Projected Langevin Monte Carlo](https://mlsanity.com/paper/1507.02564/) (2015)
- [On the Discrimination-Generalization Tradeoff in GANs](https://mlsanity.com/paper/1711.02771/) (2017)
- [Kernel Distribution Embeddings: Universal Kernels, Characteristic Kernels and Kernel Metrics on Distributions](https://mlsanity.com/paper/1604.05251/) (2016)
- [Primal Dual Interpretation of the Proximal Stochastic Gradient Langevin Algorithm](https://mlsanity.com/paper/2006.09270/) (2020)
- [Sobolev GAN](https://mlsanity.com/paper/1711.04894/) (2017)
- [Approximation and Convergence Properties of Generative Adversarial Learning](https://mlsanity.com/paper/1705.08991/) (2017)
- [On gradient regularizers for MMD GANs](https://mlsanity.com/paper/1805.11565/) (2018)
- [Synchronizing Probability Measures on Rotations via Optimal Transport](https://mlsanity.com/paper/2004.00663/) (2020)

## Built on by (most cited first, 30 of 37)

- [Generative Sliced MMD Flows with Riesz Kernels](https://mlsanity.com/paper/2305.11463/) (2023)
- [A Rigorous Link between Deep Ensembles and (Variational) Bayesian Methods](https://mlsanity.com/paper/2305.15027/) (2023)
- [A mean-field games laboratory for generative modeling](https://mlsanity.com/paper/2304.13534/) (2023)
- [Scalable Wasserstein Gradient Flow for Generative Modeling through Unbalanced Optimal Transport](https://mlsanity.com/paper/2402.05443/) (2024)
- [Efficient Gradient Flows in Sliced-Wasserstein Space](https://mlsanity.com/paper/2110.10972/) (2021)
- [Unifying GANs and Score-Based Diffusion as Generative Particle Models](https://mlsanity.com/paper/2305.16150/) (2023)
- [Deep MMD Gradient Flow without adversarial training](https://mlsanity.com/paper/2405.06780/) (2024)
- [Structure-preserving GANs](https://mlsanity.com/paper/2202.01129/) (2022)
- [Wasserstein Gradient Flows for Moreau Envelopes of f-Divergences in Reproducing Kernel Hilbert Spaces](https://mlsanity.com/paper/2402.04613/) (2024)
- [Weighted quantization using MMD: From mean field to mean shift via gradient flows](https://mlsanity.com/paper/2502.10600/) (2025)
- [On the Wasserstein Gradient Flow Interpretation of Drifting Models](https://mlsanity.com/paper/2605.05118/) (2026)
- [What makes an image realistic?](https://mlsanity.com/paper/2403.04493/) (2024)
- [Neural Wasserstein Gradient Flows for Maximum Mean Discrepancies with Riesz Kernels](https://mlsanity.com/paper/2301.11624/) (2023)
- [Interaction-Force Transport Gradient Flows](https://mlsanity.com/paper/2405.17075/) (2024)
- [Lipschitz-regularized gradient flows and generative particle algorithms for high-dimensional scarce data](https://mlsanity.com/paper/2210.17230/) (2022)
- [Bridging the Gap Between Variational Inference and Wasserstein Gradient Flows](https://mlsanity.com/paper/2310.20090/) (2023)
- [Kernel Approximation of Fisher-Rao Gradient Flows](https://mlsanity.com/paper/2410.20622/) (2024)
- [Outcome Assumptions and Duality Theory for Balancing Weights](https://mlsanity.com/paper/2203.09557/) (2022)
- [Ultra-fast feature learning for the training of two-layer neural networks in the two-timescale regime](https://mlsanity.com/paper/2504.18208/) (2025)
- [DDEQs: Distributional Deep Equilibrium Models through Wasserstein Gradient Flows](https://mlsanity.com/paper/2503.01140/) (2025)
- [Statistical and Geometrical properties of regularized Kernel Kullback-Leibler divergence](https://mlsanity.com/paper/2408.16543/) (2024)
- [Proximal optimal transport divergences](https://mlsanity.com/paper/2505.12097/) (2025)
- [Model Uncertainty and Correctability for Directed Graphical Models](https://mlsanity.com/paper/2107.08179/) (2021)
- [Data Interpolants -- That's What Discriminators in Higher-order Gradient-regularized GANs Are](https://mlsanity.com/paper/2306.00785/) (2023)
- [Statistical Error Bounds for GANs with Nonlinear Objective Functionals](https://mlsanity.com/paper/2406.16834/) (2024)
- [Smoothed Distance Kernels for MMDs and Applications in Wasserstein Gradient Flows](https://mlsanity.com/paper/2504.07820/) (2025)
- [Stationary MMD Points](https://mlsanity.com/paper/2505.20754/) (2025)
- [Regularized f-Divergence Kernel Tests](https://mlsanity.com/paper/2601.19755/) (2026)
- [Busemann Functions in the Wasserstein Space: Existence, Closed-Forms, and Applications to Slicing](https://mlsanity.com/paper/2510.04579/) (2025)
- [A Gradient Flow Perspective on Minimum MMD Estimation](https://mlsanity.com/paper/2607.03871/) (2026)

## Abstract (arXiv)

We study the gradient flow for a relaxed approximation to the Kullback-Leibler (KL) divergence between a moving source and a fixed target distribution. This approximation, termed the KALE (KL approximate lower-bound estimator), solves a regularized version of the Fenchel dual problem defining the KL over a restricted class of functions. When using a Reproducing Kernel Hilbert Space (RKHS) to define the function class, we show that the KALE continuously interpolates between the KL and the Maximum Mean Discrepancy (MMD). Like the MMD and other Integral Probability Metrics, the KALE remains well defined for mutually singular distributions. Nonetheless, the KALE inherits from the limiting KL a greater sensitivity to mismatch in the support of the distributions, compared with the MMD. These two properties make the KALE gradient flow particularly well suited when the target distribution is supported on a low-dimensional manifold. Under an assumption of sufficient smoothness of the trajectories, we show the global convergence of the KALE flow. We propose a particle implementation of the flow given initial samples from the source and the target distribution, which we use to empirically confirm the KALE's properties.