# Statistical and Geometrical properties of regularized Kernel Kullback-Leibler divergence

arXiv: https://arxiv.org/abs/2408.16543 · HTML: https://mlsanity.com/paper/2408.16543/

## Key facts

- Cited by 3 papers in the map (data through 14 Oct 2026).
- Builds on 30 of its 52 references that are in the map.
- First posted on arXiv 29 Aug 2024.

## Builds on (most cited first, 30 of 30)

- [Denoising Diffusion Probabilistic Models](https://mlsanity.com/paper/2006.11239/) (2020)
- [Score-Based Generative Modeling through Stochastic Differential Equations](https://mlsanity.com/paper/2011.13456/) (2020)
- [Large Scale GAN Training for High Fidelity Natural Image Synthesis](https://mlsanity.com/paper/1809.11096/) (2018)
- [Variational Inference: A Review for Statisticians](https://mlsanity.com/paper/1601.00670/) (2016)
- [Demystifying MMD GANs](https://mlsanity.com/paper/1801.01401/) (2018)
- [Stein Variational Gradient Descent: A General Purpose Bayesian Inference Algorithm](https://mlsanity.com/paper/1608.04471/) (2016)
- [Black Box Variational Inference](https://mlsanity.com/paper/1401.0118/) (2013)
- [Learning Generative Models with Sinkhorn Divergences](https://mlsanity.com/paper/1706.00292/) (2017)
- [Generalization Properties of Learning with Random Features](https://mlsanity.com/paper/1602.04474/) (2016)
- [Maximum Mean Discrepancy Gradient Flow](https://mlsanity.com/paper/1906.04370/) (2019)
- [Sampling as optimization in the space of measures: The Langevin dynamics as a composite optimization problem](https://mlsanity.com/paper/1802.08089/) (2018)
- [Sliced-Wasserstein Flows: Nonparametric Generative Modeling via Optimal Transport and Diffusions](https://mlsanity.com/paper/1806.08141/) (2018)
- [SVGD as a kernelized Wasserstein gradient flow of the chi-squared divergence](https://mlsanity.com/paper/2006.02509/) (2020)
- [Kernel Stein Discrepancy Descent](https://mlsanity.com/paper/2105.09994/) (2021)
- [Refining Deep Generative Models via Discriminator Gradient Flow](https://mlsanity.com/paper/2012.00780/) (2020)
- [KALE Flow: A Relaxed KL Gradient Flow for Probabilities with Disjoint Support](https://mlsanity.com/paper/2106.08929/) (2021)
- [Information Theory with Kernel Methods](https://mlsanity.com/paper/2202.08545/) (2022)
- [Generative Sliced MMD Flows with Riesz Kernels](https://mlsanity.com/paper/2305.11463/) (2023)
- [Deep Generative Learning via Variational Gradient Flow](https://mlsanity.com/paper/1901.08469/) (2019)
- [Unifying GANs and Score-Based Diffusion as Generative Particle Models](https://mlsanity.com/paper/2305.16150/) (2023)
- [Spectral Regularized Kernel Two-Sample Tests](https://mlsanity.com/paper/2212.09201/) (2022)
- [On the Optimality of Kernel-Embedding Based Goodness-of-Fit Tests](https://mlsanity.com/paper/1709.08148/) (2017)
- [Deep MMD Gradient Flow without adversarial training](https://mlsanity.com/paper/2405.06780/) (2024)
- [Wasserstein Steepest Descent Flows of Discrepancies with Riesz Kernels](https://mlsanity.com/paper/2211.01804/) (2022)
- [Optimizing Variational Representations of Divergences and Accelerating their Statistical Estimation](https://mlsanity.com/paper/2006.08781/) (2020)
- [Wasserstein Gradient Flows for Moreau Envelopes of f-Divergences in Reproducing Kernel Hilbert Spaces](https://mlsanity.com/paper/2402.04613/) (2024)
- [Properties of Discrete Sliced Wasserstein Losses](https://mlsanity.com/paper/2307.10352/) (2023)
- [(f,Gamma)-Divergences: Interpolating between f-Divergences and Integral Probability Metrics](https://mlsanity.com/paper/2011.05953/) (2020)
- [Minimizing f-Divergences by Interpolating Velocity Fields](https://mlsanity.com/paper/2305.15577/) (2023)
- [α-Geodesical Skew Divergence](https://mlsanity.com/paper/2103.17060/) (2021)

## Built on by (most cited first, 3 of 3)

- [DDEQs: Distributional Deep Equilibrium Models through Wasserstein Gradient Flows](https://mlsanity.com/paper/2503.01140/) (2025)
- [Variational Inference on the Boolean Hypercube with the Quantum Entropy](https://mlsanity.com/paper/2411.03759/) (2024)
- [Kernel Trace Distance: Quantum Statistical Metric between Measures through RKHS Density Operators](https://mlsanity.com/paper/2507.06055/) (2025)

## Abstract (arXiv)

In this paper, we study the statistical and geometrical properties of the Kullback-Leibler divergence with kernel covariance operators (KKL) introduced by Bach [2022]. Unlike the classical Kullback-Leibler (KL) divergence that involves density ratios, the KKL compares probability distributions through covariance operators (embeddings) in a reproducible kernel Hilbert space (RKHS), and compute the Kullback-Leibler quantum divergence. This novel divergence hence shares parallel but different aspects with both the standard Kullback-Leibler between probability distributions and kernel embeddings metrics such as the maximum mean discrepancy. A limitation faced with the original KKL divergence is its inability to be defined for distributions with disjoint supports. To solve this problem, we propose in this paper a regularised variant that guarantees that the divergence is well defined for all distributions. We derive bounds that quantify the deviation of the regularised KKL to the original one, as well as finite-sample bounds. In addition, we provide a closed-form expression for the regularised KKL, specifically applicable when the distributions consist of finite sets of points, which makes it implementable. Furthermore, we derive a Wasserstein gradient descent scheme of the KKL divergence in the case of discrete distributions, and study empirically its properties to transport a set of points to a target distribution.