# Kullback-Leibler and Renyi divergences in reproducing kernel Hilbert space and Gaussian process settings

arXiv: https://arxiv.org/abs/2207.08406 · HTML: https://mlsanity.com/paper/2207.08406/

## Key facts

- Cited by 4 papers in the map (data through 14 Oct 2026).
- Builds on 4 of its 91 references that are in the map.
- First posted on arXiv 18 Jul 2022.

## Builds on (most cited first, 4 of 4)

- [Understanding Variational Inference in Function-Space](https://mlsanity.com/paper/2011.09421/) (2020)
- [Convergence and finite sample approximations of entropic regularized Wasserstein distances in Gaussian and RKHS settings](https://mlsanity.com/paper/2101.01429/) (2021)
- [Estimation of Riemannian distances between covariance operators and Gaussian processes](https://mlsanity.com/paper/2108.11683/) (2021)
- [Estimating R'enyi's α-Cross-Entropies in a Matrix-Based Way](https://mlsanity.com/paper/2109.11737/) (2021)

## Built on by (most cited first, 4 of 4)

- [The Representation Jensen-Shannon Divergence](https://mlsanity.com/paper/2305.16446/) (2023)
- [Regularized KL-Divergence for Well-Defined Function-Space Variational Inference in Bayesian neural networks](https://mlsanity.com/paper/2406.04317/) (2024)
- [Geometric Jensen-Shannon Divergence Between Gaussian Measures On Hilbert Space](https://mlsanity.com/paper/2506.10494/) (2025)
- [Likelihood Ratio Tests by Kernel Gaussian Embedding](https://mlsanity.com/paper/2508.07982/) (2025)

## Abstract (arXiv)

In this work, we present formulations for regularized Kullback-Leibler and R'enyi divergences via the Alpha Log-Determinant (Log-Det) divergences between positive Hilbert-Schmidt operators on Hilbert spaces in two different settings, namely (i) covariance operators and Gaussian measures defined on reproducing kernel Hilbert spaces (RKHS); and (ii) Gaussian processes with squared integrable sample paths. For characteristic kernels, the first setting leads to divergences between arbitrary Borel probability measures on a complete, separable metric space. We show that the Alpha Log-Det divergences are continuous in the Hilbert-Schmidt norm, which enables us to apply laws of large numbers for Hilbert space-valued random variables. As a consequence of this, we show that, in both settings, the infinite-dimensional divergences can be consistently and efficiently estimated from their finite-dimensional versions, using finite-dimensional Gram matrices/Gaussian measures and finite sample data, with it dimension-independent sample complexities in all cases. RKHS methodology plays a central role in the theoretical analysis in both settings. The mathematical formulation is illustrated by numerical experiments.