# Estimating Kullback-Leibler Divergence Using Kernel Machines

arXiv: https://arxiv.org/abs/1905.00586 · HTML: https://mlsanity.com/paper/1905.00586/

## Key facts

- Cited by 3 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 2 May 2019.

## Built on by (most cited first, 3 of 3)

- [Reliable Estimation of KL Divergence using a Discriminator in Reproducing Kernel Hilbert Space](https://mlsanity.com/paper/2109.14688/) (2021)
- [The Conditional Cauchy-Schwarz Divergence with Applications to Time-Series Data and Sequential Decision Making](https://mlsanity.com/paper/2301.08970/) (2023)
- [OBSER: Object-Based Sub-Environment Recognition for Zero-Shot Environmental Inference](https://mlsanity.com/paper/2507.02929/) (2025)

## Abstract (arXiv)

Recently, a method called the Mutual Information Neural Estimator (MINE) that uses neural networks has been proposed to estimate mutual information and more generally the Kullback-Leibler (KL) divergence between two distributions. The method uses the Donsker-Varadhan representation to arrive at the estimate of the KL divergence and is better than the existing estimators in terms of scalability and flexibility. The output of MINE algorithm is not guaranteed to be a consistent estimator. We propose a new estimator that instead of searching among functions characterized by neural networks searches the functions in a Reproducing Kernel Hilbert Space. We prove that the proposed estimator is consistent. We carry out simulations and show that when the datasets are small the proposed estimator is more reliable than the MINE estimator and when the datasets are large the performance of the two methods are close.