# Neural Estimation of Statistical Divergences

arXiv: https://arxiv.org/abs/2110.03652 · HTML: https://mlsanity.com/paper/2110.03652/

## Key facts

- Cited by 25 papers in the map (data through 14 Oct 2026).
- Builds on 4 of its 51 references that are in the map.
- First posted on arXiv 7 Oct 2021.

## Builds on (most cited first, 4 of 4)

- [Breaking the Curse of Dimensionality with Convex Neural Networks](https://mlsanity.com/paper/1412.8690/) (2014)
- [A Short Note on Concentration Inequalities for Random Vectors with SubGaussian Norm](https://mlsanity.com/paper/1902.03736/) (2019)
- [Optimal rates of entropy estimation over Lipschitz balls](https://mlsanity.com/paper/1711.02141/) (2017)
- [Estimating Certain Integral Probability Metric (IPM) is as Hard as Estimating under the IPM](https://mlsanity.com/paper/1911.00730/) (2019)

## Built on by (most cited first, 25 of 25)

- [MAUVE Scores for Generative Models: Theory and Practice](https://mlsanity.com/paper/2212.14578/) (2022)
- [Convergence of flow-based generative models via proximal gradient descent in Wasserstein space](https://mlsanity.com/paper/2310.17582/) (2023)
- [Mean-field Variational Inference via Wasserstein Gradient Flow](https://mlsanity.com/paper/2207.08074/) (2022)
- [Understanding Multimodal LLMs Under Distribution Shifts: An Information-Theoretic Approach](https://mlsanity.com/paper/2502.00577/) (2025)
- [Sample Complexity of Probability Divergences under Group Symmetry](https://mlsanity.com/paper/2302.01915/) (2023)
- [Max-Sliced Mutual Information](https://mlsanity.com/paper/2309.16200/) (2023)
- [Discriminative Estimation of Total Variation Distance: A Fidelity Auditor for Generative Data](https://mlsanity.com/paper/2405.15337/) (2024)
- [The Representation Jensen-Shannon Divergence](https://mlsanity.com/paper/2305.16446/) (2023)
- [Quantum Neural Estimation of Entropies](https://mlsanity.com/paper/2307.01171/) (2023)
- [Neural network-based CUSUM for online change-point detection](https://mlsanity.com/paper/2210.17312/) (2022)
- [Weight Clipping for Robust Conformal Inference under Unbounded Covariate Shifts](https://mlsanity.com/paper/2605.02072/) (2026)
- [Information Processing Equalities and the Information-Risk Bridge](https://mlsanity.com/paper/2207.11987/) (2022)
- [Performance Guarantees for Quantum Neural Estimation of Entropies](https://mlsanity.com/paper/2511.19289/) (2025)
- [Neural Stein critics with staged L2-regularization](https://mlsanity.com/paper/2207.03406/) (2022)
- [Data-Driven Optimization of Directed Information over Discrete Alphabets](https://mlsanity.com/paper/2301.00621/) (2023)
- [Neural Estimation for Scaling Entropic Multimarginal Optimal Transport](https://mlsanity.com/paper/2506.00573/) (2025)
- [Universality of reservoir systems with recurrent neural networks](https://mlsanity.com/paper/2403.01900/) (2024)
- [A Perspective on Neural Capacity Estimation: Viability and Reliability](https://mlsanity.com/paper/2203.11793/) (2022)
- [k-Sliced Mutual Information: A Quantitative Study of Scalability with Dimension](https://mlsanity.com/paper/2206.08526/) (2022)
- [Stabilizing Subject Transfer in EEG Classification with Divergence Estimation](https://mlsanity.com/paper/2310.08762/) (2023)
- [On approximating the f-divergence between two Ising models](https://mlsanity.com/paper/2509.05016/) (2025)
- [A Neural Network Algorithm for KL Divergence Estimation with Quantitative Error Bounds](https://mlsanity.com/paper/2510.05386/) (2025)
- [Nonparametric Estimation of Joint Entropy via Partitioned Sample-Spacing](https://mlsanity.com/paper/2511.13602/) (2025)
- [Generative modeling using evolved quantum Boltzmann machines](https://mlsanity.com/paper/2512.02721/) (2025)
- [Blind Recovery of Latent Domains via Unsupervised Symmetry Discovery](https://mlsanity.com/paper/2606.17782/) (2026)

## Abstract (arXiv)

Statistical divergences (SDs), which quantify the dissimilarity between probability distributions, are a basic constituent of statistical inference and machine learning. A modern method for estimating those divergences relies on parametrizing an empirical variational form by a neural network (NN) and optimizing over parameter space. Such neural estimators are abundantly used in practice, but corresponding performance guarantees are partial and call for further exploration. We establish non-asymptotic absolute error bounds for a neural estimator realized by a shallow NN, focusing on four popular mathsff-divergences -- Kullback-Leibler, chi-squared, squared Hellinger, and total variation. Our analysis relies on non-asymptotic function approximation theorems and tools from empirical process theory to bound the two sources of error involved: function approximation and empirical estimation. The bounds characterize the effective error in terms of NN size and the number of samples, and reveal scaling rates that ensure consistency. For compactly supported distributions, we further show that neural estimators of the first three divergences above with appropriate NN growth-rate are minimax rate-optimal, achieving the parametric convergence rate.