mlsanityData to 14 Oct 2026

KALE Flow: A Relaxed KL Gradient Flow for Probabilities with Disjoint Support

arXiv 2106.08929 ↗First posted 16 Jun 2021stat.ML cs.LGcited by 37

Not on the map. Why?

Abstract (arXiv)

We study the gradient flow for a relaxed approximation to the Kullback-Leibler (KL) divergence between a moving source and a fixed target distribution. This approximation, termed the KALE (KL approximate lower-bound estimator), solves a regularized version of the Fenchel dual problem defining the KL over a restricted class of functions. When using a Reproducing Kernel Hilbert Space (RKHS) to define the function class, we show that the KALE continuously interpolates between the KL and the Maximum Mean Discrepancy (MMD). Like the MMD and other Integral Probability Metrics, the KALE remains well defined for mutually singular distributions. Nonetheless, the KALE inherits from the limiting KL a greater sensitivity to mismatch in the support of the distributions, compared with the MMD. These two properties make the KALE gradient flow particularly well suited when the target distribution is supported on a low-dimensional manifold. Under an assumption of sufficient smoothness of the trajectories, we show the global convergence of the KALE flow. We propose a particle implementation of the flow given initial samples from the source and the target distribution, which we use to empirically confirm the KALE's properties.

Key facts. Cited by 37 papers in the map (data through 14 Oct 2026). Builds on 34 of its 68 references that are in the map. First posted on arXiv 16 Jun 2021.

Builds on

34 of 68 references.

References · 34

  1. 2017Improved Training of Wasserstein GANscited by 2,235
  2. 2016Variational Inference: A Review for Statisticianscited by 1,472
  3. 2018Demystifying MMD GANscited by 1,293
  4. 2016f-GAN: Training Generative Neural Samplers using Variational Divergence Minimizationcited by 506
  5. 2018A Mean Field View of the Landscape of Two-Layers Neural Networkscited by 466
  6. 2017Towards Principled Methods for Training Generative Adversarial Networkscited by 464
  7. 2018On the Global Convergence of Gradient Descent for Over-parameterized Models using Optimal Transportcited by 426
  8. 2017Learning Generative Models with Sinkhorn Divergencescited by 314
  9. 2015Generative Moment Matching Networkscited by 257
  10. 2017MMD GAN: Towards Deeper Understanding of Moment Matching Networkcited by 254
  11. 2015Non-asymptotic convergence analysis for the Unadjusted Langevin Algorithmcited by 197
  12. 2017Stein Variational Gradient Descent as Gradient Flowcited by 152
22 more
  1. 2015Training generative neural networks via Maximum Mean Discrepancy optimizationcited by 151
  2. 2018Analysis of Langevin Monte Carlo via convex optimizationcited by 127
  3. 2019Maximum Mean Discrepancy Gradient Flowcited by 127
  4. 2017The Cramer Distance as a Solution to Biased Wasserstein Gradientscited by 119
  5. 2017User-friendly guarantees for the Langevin Monte Carlo with inaccurate gradientcited by 116
  6. 2017Adversarial Variational Bayes: Unifying Variational Autoencoders and Generative Adversarial Networkscited by 107
  7. 2018Mean Field Analysis of Neural Networks: A Central Limit Theoremcited by 107
  8. 2020Generalized Energy Based Modelscited by 73
  9. 2018Mirrored Langevin Dynamicscited by 61
  10. 2020Efficient constrained sampling via the mirror-Langevin algorithmcited by 55
  11. 2015Sampling from a log-concave distribution with Projected Langevin Monte Carlocited by 52
  12. 2017On the Discrimination-Generalization Tradeoff in GANscited by 48
  13. 2016Kernel Distribution Embeddings: Universal Kernels, Characteristic Kernels and Kernel Metrics on Distributionscited by 40
  14. 2020Primal Dual Interpretation of the Proximal Stochastic Gradient Langevin Algorithmcited by 35
  15. 2017Sobolev GANcited by 29
  16. 2017Approximation and Convergence Properties of Generative Adversarial Learningcited by 28
  17. 2018On gradient regularizers for MMD GANscited by 28
  18. 2020Synchronizing Probability Measures on Rotations via Optimal Transportcited by 19
  19. 2020Unbalanced Sobolev Descentcited by 11
  20. 2020Sinkhorn Barycenter via Functional Gradient Descentcited by 9
  21. 2017Geometrical Insights for Implicit Generative Modelingcited by 7
  22. 2019Statistical Estimation of the Poincar'e constant and Application to Sampling Multimodal Distributionscited by 5

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2021: 2; 2022: 4; 2023: 7; 2024: 9; 2025: 9; 2026: 6.

37 papers.

Sort:

By area:

  1. 2023Generative Sliced MMD Flows with Riesz Kernelscited by 37
  2. 2023A Rigorous Link between Deep Ensembles and (Variational) Bayesian Methodscited by 32
  3. 2023A mean-field games laboratory for generative modelingcited by 28
  4. 2024Scalable Wasserstein Gradient Flow for Generative Modeling through Unbalanced Optimal Transportcited by 23
  5. 2021Efficient Gradient Flows in Sliced-Wasserstein Spacecited by 21
  6. 2023Unifying GANs and Score-Based Diffusion as Generative Particle Modelscited by 21
  7. 2024Deep MMD Gradient Flow without adversarial trainingcited by 19
  8. 2022Structure-preserving GANscited by 17
  9. 2024Wasserstein Gradient Flows for Moreau Envelopes of f-Divergences in Reproducing Kernel Hilbert Spacescited by 15
  10. 2025Weighted quantization using MMD: From mean field to mean shift via gradient flowscited by 12
  11. 2026On the Wasserstein Gradient Flow Interpretation of Drifting Modelscited by 11
  12. 2024What makes an image realistic?cited by 10
25 more
  1. 2023Neural Wasserstein Gradient Flows for Maximum Mean Discrepancies with Riesz Kernelscited by 9
  2. 2024Interaction-Force Transport Gradient Flowscited by 9
  3. 2022Lipschitz-regularized gradient flows and generative particle algorithms for high-dimensional scarce datacited by 8
  4. 2023Bridging the Gap Between Variational Inference and Wasserstein Gradient Flowscited by 8
  5. 2024Kernel Approximation of Fisher-Rao Gradient Flowscited by 7
  6. 2022Outcome Assumptions and Duality Theory for Balancing Weightscited by 5
  7. 2025Ultra-fast feature learning for the training of two-layer neural networks in the two-timescale regimecited by 5
  8. 2025DDEQs: Distributional Deep Equilibrium Models through Wasserstein Gradient Flowscited by 4
  9. 2024Statistical and Geometrical properties of regularized Kernel Kullback-Leibler divergencecited by 3
  10. 2025Proximal optimal transport divergencescited by 3
  11. 2021Model Uncertainty and Correctability for Directed Graphical Modelscited by 2
  12. 2023Data Interpolants -- That's What Discriminators in Higher-order Gradient-regularized GANs Arecited by 2
  13. 2024Statistical Error Bounds for GANs with Nonlinear Objective Functionalscited by 2
  14. 2025Smoothed Distance Kernels for MMDs and Applications in Wasserstein Gradient Flowscited by 2
  15. 2025Stationary MMD Pointscited by 2
  16. 2026Regularized f-Divergence Kernel Testscited by 2
  17. 2025Busemann Functions in the Wasserstein Space: Existence, Closed-Forms, and Applications to Slicingcited by 1
  18. 2026A Gradient Flow Perspective on Minimum MMD Estimationcited by 1
  19. 2022Unified Perspective on Probability Divergence via Maximum Likelihood Density Ratio Estimation: Bridging KL-Divergence and Integral Probability Metricscited by 0
  20. 2024Revised Regularization for Efficient Continual Learning through Correlation-Based Parameter Update in Bayesian Neural Networkscited by 0
  21. 2025Learning Energy-Based Models by Self-normalising the Likelihoodcited by 0
  22. 2025Variational Inference for Quantum HyperNetworkscited by 0
  23. 2026Rethinking the Flow-Based Gradual Domain Adaptation: A Semi-Dual Optimal Transport Perspectivecited by 0
  24. 2026Well-Posed KL-Regularized Control via Wasserstein and Kalman-Wasserstein KL Divergencescited by 0
  25. 2026One-Step Generative Modeling via Unbalanced Optimal Transportcited by 0