mlsanityData to 7 Oct 2026

Statistical and Geometrical properties of regularized Kernel Kullback-Leibler divergence

arXiv 2408.16543 ↗First posted 29 Aug 2024stat.ML cs.LG math.FAcited by 3

Not on the map. Why?

Abstract (arXiv)

In this paper, we study the statistical and geometrical properties of the Kullback-Leibler divergence with kernel covariance operators (KKL) introduced by Bach [2022]. Unlike the classical Kullback-Leibler (KL) divergence that involves density ratios, the KKL compares probability distributions through covariance operators (embeddings) in a reproducible kernel Hilbert space (RKHS), and compute the Kullback-Leibler quantum divergence. This novel divergence hence shares parallel but different aspects with both the standard Kullback-Leibler between probability distributions and kernel embeddings metrics such as the maximum mean discrepancy. A limitation faced with the original KKL divergence is its inability to be defined for distributions with disjoint supports. To solve this problem, we propose in this paper a regularised variant that guarantees that the divergence is well defined for all distributions. We derive bounds that quantify the deviation of the regularised KKL to the original one, as well as finite-sample bounds. In addition, we provide a closed-form expression for the regularised KKL, specifically applicable when the distributions consist of finite sets of points, which makes it implementable. Furthermore, we derive a Wasserstein gradient descent scheme of the KKL divergence in the case of discrete distributions, and study empirically its properties to transport a set of points to a target distribution.

Key facts. Cited by 3 papers in the map (data through 7 Oct 2026). Builds on 30 of its 52 references that are in the map. First posted on arXiv 29 Aug 2024.

Builds on

30 of 52 references.

References · 30

  1. 2020Denoising Diffusion Probabilistic Modelscited by 20,851
  2. 2020Score-Based Generative Modeling through Stochastic Differential Equationscited by 8,964
  3. 2018Large Scale GAN Training for High Fidelity Natural Image Synthesiscited by 2,609
  4. 2016Variational Inference: A Review for Statisticianscited by 1,466
  5. 2018Demystifying MMD GANscited by 1,268
  6. 2016Stein Variational Gradient Descent: A General Purpose Bayesian Inference Algorithmcited by 498
  7. 2013Black Box Variational Inferencecited by 317
  8. 2017Learning Generative Models with Sinkhorn Divergencescited by 310
  9. 2016Generalization Properties of Learning with Random Featurescited by 163
  10. 2018Sampling as optimization in the space of measures: The Langevin dynamics as a composite optimization problemcited by 123
  11. 2019Maximum Mean Discrepancy Gradient Flowcited by 123
  12. 2018Sliced-Wasserstein Flows: Nonparametric Generative Modeling via Optimal Transport and Diffusionscited by 77
18 more
  1. 2020SVGD as a kernelized Wasserstein gradient flow of the chi-squared divergencecited by 64
  2. 2021Kernel Stein Discrepancy Descentcited by 57
  3. 2020Refining Deep Generative Models via Discriminator Gradient Flowcited by 45
  4. 2021KALE Flow: A Relaxed KL Gradient Flow for Probabilities with Disjoint Supportcited by 37
  5. 2022Information Theory with Kernel Methodscited by 37
  6. 2023Generative Sliced MMD Flows with Riesz Kernelscited by 36
  7. 2019Deep Generative Learning via Variational Gradient Flowcited by 32
  8. 2023Unifying GANs and Score-Based Diffusion as Generative Particle Modelscited by 21
  9. 2017On the Optimality of Kernel-Embedding Based Goodness-of-Fit Testscited by 19
  10. 2022Spectral Regularized Kernel Two-Sample Testscited by 19
  11. 2024Deep MMD Gradient Flow without adversarial trainingcited by 18
  12. 2022Wasserstein Steepest Descent Flows of Discrepancies with Riesz Kernelscited by 16
  13. 2020Optimizing Variational Representations of Divergences and Accelerating their Statistical Estimationcited by 15
  14. 2024Wasserstein Gradient Flows for Moreau Envelopes of f-Divergences in Reproducing Kernel Hilbert Spacescited by 15
  15. 2023Properties of Discrete Sliced Wasserstein Lossescited by 12
  16. 2020(f,Gamma)-Divergences: Interpolating between f-Divergences and Integral Probability Metricscited by 10
  17. 2023Minimizing f-Divergences by Interpolating Velocity Fieldscited by 8
  18. 2021α-Geodesical Skew Divergencecited by 6

Built on by

Cited by year

Citing papers per year: 2024: 1; 2025: 2.

3 papers.

  1. 2025DDEQs: Distributional Deep Equilibrium Models through Wasserstein Gradient Flowscited by 4
  2. 2024Variational Inference on the Boolean Hypercube with the Quantum Entropycited by 0
  3. 2025Kernel Trace Distance: Quantum Statistical Metric between Measures through RKHS Density Operatorscited by 0