mlsanityData to 14 Oct 2026

A Neural Network Algorithm for KL Divergence Estimation with Quantitative Error Bounds

arXiv 2510.05386 ↗First posted 6 Oct 2025cs.LG cs.IT math.ITcited by 0

Not on the map. Why?

Abstract (arXiv)

Estimating the Kullback-Leibler (KL) divergence between random variables is a fundamental problem in statistical analysis. For continuous random variables, traditional information-theoretic estimators scale poorly with dimension and/or sample size. To mitigate this challenge, a variety of methods have been proposed to estimate KL divergences and related quantities, such as mutual information, using neural networks. The existing theoretical analyses show that neural network parameters achieving low error exist. However, since they rely on non-constructive neural network approximation theorems, they do not guarantee that the existing algorithms actually achieve low error. In this paper, we propose a KL divergence estimation algorithm using a shallow neural network with randomized hidden weights and biases (i.e. a random feature method). We show that with high probability, the algorithm achieves a KL divergence estimation error of O(m-1/2+T-1/3), where m is the number of neurons and T is both the number of steps of the algorithm and the number of samples.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Builds on 18 of its 42 references that are in the map. First posted on arXiv 6 Oct 2025.

Builds on

18 of 42 references.

References · 18

  1. 2016f-GAN: Training Generative Neural Samplers using Variational Divergence Minimizationcited by 506
  2. 2019Understanding the Limitations of Variational Mutual Information Estimatorscited by 140
  3. 2020Approximation Bounds for Random Neural Networks and Reservoir Systemscited by 52
  4. 2021Random feature neural networks learn Black-Scholes type PDEs without curse of dimensionalitycited by 34
  5. 2021On the Approximation Power of Two-Layer Networks of Random ReLUscited by 29
  6. 2016Risk Bounds for High-dimensional Ridge Function Combinations Including Neural Networkscited by 26
  7. 2021Neural Estimation of Statistical Divergencescited by 25
  8. 2023Universal approximation property of Banach space-valued random feature models including random neural networkscited by 13
  9. 2019Neural Entropic Estimation: A faster path to mutual information estimationcited by 9
  10. 2019Conditional Mutual Information Neural Estimatorcited by 7
  11. 2020Neural Estimators for Conditional Mutual Information Using Nearest Neighbors Samplingcited by 7
  12. 2019Data-Efficient Mutual Information Neural Estimatorcited by 6
6 more
  1. 2020Combating the Instability of Mutual Information-based Losses via Regularizationcited by 5
  2. 2021Neural Capacity Estimators: How Reliable Are They?cited by 4
  3. 2024Approximation with Random Shallow ReLU Networks with Applications to Model Reference Adaptive Controlcited by 4
  4. 2025Neural Estimation for Scaling Entropic Multimarginal Optimal Transportcited by 2
  5. 2023Efficient uniform approximation using Random Vector Functional Link networkscited by 1
  6. 2024Function Gradient Approximation with Random Shallow ReLU Networks with Control Applicationscited by 1

Built on by

Not cited yet.