# A Neural Network Algorithm for KL Divergence Estimation with Quantitative Error Bounds

arXiv: https://arxiv.org/abs/2510.05386 · HTML: https://mlsanity.com/paper/2510.05386/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 18 of its 42 references that are in the map.
- First posted on arXiv 6 Oct 2025.

## Builds on (most cited first, 18 of 18)

- [f-GAN: Training Generative Neural Samplers using Variational Divergence Minimization](https://mlsanity.com/paper/1606.00709/) (2016)
- [Understanding the Limitations of Variational Mutual Information Estimators](https://mlsanity.com/paper/1910.06222/) (2019)
- [Approximation Bounds for Random Neural Networks and Reservoir Systems](https://mlsanity.com/paper/2002.05933/) (2020)
- [Random feature neural networks learn Black-Scholes type PDEs without curse of dimensionality](https://mlsanity.com/paper/2106.08900/) (2021)
- [On the Approximation Power of Two-Layer Networks of Random ReLUs](https://mlsanity.com/paper/2102.02336/) (2021)
- [Risk Bounds for High-dimensional Ridge Function Combinations Including Neural Networks](https://mlsanity.com/paper/1607.01434/) (2016)
- [Neural Estimation of Statistical Divergences](https://mlsanity.com/paper/2110.03652/) (2021)
- [Universal approximation property of Banach space-valued random feature models including random neural networks](https://mlsanity.com/paper/2312.08410/) (2023)
- [Neural Entropic Estimation: A faster path to mutual information estimation](https://mlsanity.com/paper/1905.12957/) (2019)
- [Conditional Mutual Information Neural Estimator](https://mlsanity.com/paper/1911.02277/) (2019)
- [Neural Estimators for Conditional Mutual Information Using Nearest Neighbors Sampling](https://mlsanity.com/paper/2006.07225/) (2020)
- [Data-Efficient Mutual Information Neural Estimator](https://mlsanity.com/paper/1905.03319/) (2019)
- [Combating the Instability of Mutual Information-based Losses via Regularization](https://mlsanity.com/paper/2011.07932/) (2020)
- [Neural Capacity Estimators: How Reliable Are They?](https://mlsanity.com/paper/2111.07401/) (2021)
- [Approximation with Random Shallow ReLU Networks with Applications to Model Reference Adaptive Control](https://mlsanity.com/paper/2403.17142/) (2024)
- [Neural Estimation for Scaling Entropic Multimarginal Optimal Transport](https://mlsanity.com/paper/2506.00573/) (2025)
- [Efficient uniform approximation using Random Vector Functional Link networks](https://mlsanity.com/paper/2306.17501/) (2023)
- [Function Gradient Approximation with Random Shallow ReLU Networks with Control Applications](https://mlsanity.com/paper/2410.05071/) (2024)

## Abstract (arXiv)

Estimating the Kullback-Leibler (KL) divergence between random variables is a fundamental problem in statistical analysis. For continuous random variables, traditional information-theoretic estimators scale poorly with dimension and/or sample size. To mitigate this challenge, a variety of methods have been proposed to estimate KL divergences and related quantities, such as mutual information, using neural networks. The existing theoretical analyses show that neural network parameters achieving low error exist. However, since they rely on non-constructive neural network approximation theorems, they do not guarantee that the existing algorithms actually achieve low error. In this paper, we propose a KL divergence estimation algorithm using a shallow neural network with randomized hidden weights and biases (i.e. a random feature method). We show that with high probability, the algorithm achieves a KL divergence estimation error of O(m-1/2+T-1/3), where m is the number of neurons and T is both the number of steps of the algorithm and the number of samples.