# Non-Asymptotic Performance Guarantees for Neural Estimation of mathsff-Divergences

arXiv: https://arxiv.org/abs/2103.06923 · HTML: https://mlsanity.com/paper/2103.06923/

## Key facts

- Cited by 15 papers in the map (data through 14 Oct 2026).
- Builds on 14 of its 36 references that are in the map.
- First posted on arXiv 11 Mar 2021.

## Builds on (most cited first, 14 of 14)

- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [Auto-Encoding Variational Bayes](https://mlsanity.com/paper/1312.6114/) (2013)
- [MINE: Mutual Information Neural Estimation](https://mlsanity.com/paper/1801.04062/) (2018)
- [f-GAN: Training Generative Neural Samplers using Variational Divergence Minimization](https://mlsanity.com/paper/1606.00709/) (2016)
- [Wasserstein Auto-Encoders](https://mlsanity.com/paper/1711.01558/) (2017)
- [The Expressive Power of Neural Networks: A View from the Width](https://mlsanity.com/paper/1709.02540/) (2017)
- [Generalization and Equilibrium in Generative Adversarial Nets (GANs)](https://mlsanity.com/paper/1703.00573/) (2017)
- [On the Discrimination-Generalization Tradeoff in GANs](https://mlsanity.com/paper/1711.02771/) (2017)
- [Efficient multivariate entropy estimation via k-nearest neighbour distances](https://mlsanity.com/paper/1606.00304/) (2016)
- [Fast Optimal Transport Averaging of Neuroimaging Data](https://mlsanity.com/paper/1503.08596/) (2015)
- [Optimal rates of entropy estimation over Lipschitz balls](https://mlsanity.com/paper/1711.02141/) (2017)
- [Estimating Certain Integral Probability Metric (IPM) is as Hard as Estimating under the IPM](https://mlsanity.com/paper/1911.00730/) (2019)
- [Wasserstein Barycenter Model Ensembling](https://mlsanity.com/paper/1902.04999/) (2019)
- [Nonparametric Estimation of Renyi Divergence and Friends](https://mlsanity.com/paper/1402.2966/) (2014)

## Built on by (most cited first, 15 of 15)

- [Energy-guided Entropic Neural Optimal Transport](https://mlsanity.com/paper/2304.06094/) (2023)
- [Divergence Frontiers for Generative Models: Sample Complexity, Quantization Effects, and Frontier Integrals](https://mlsanity.com/paper/2106.07898/) (2021)
- [Max-Sliced Mutual Information](https://mlsanity.com/paper/2309.16200/) (2023)
- [Certifying Out-of-Domain Generalization for Blackbox Functions](https://mlsanity.com/paper/2202.01679/) (2022)
- [Quantum Neural Estimation of Entropies](https://mlsanity.com/paper/2307.01171/) (2023)
- [Robust Generative Learning with Lipschitz-Regularized α-Divergences Allows Minimal Assumptions on Target Distributions](https://mlsanity.com/paper/2405.13962/) (2024)
- [Fast approximations of the Jeffreys divergence between univariate Gaussian mixture models via exponential polynomial densities](https://mlsanity.com/paper/2107.05901/) (2021)
- [Performance Guarantees for Quantum Neural Estimation of Entropies](https://mlsanity.com/paper/2511.19289/) (2025)
- [Data-Driven Optimization of Directed Information over Discrete Alphabets](https://mlsanity.com/paper/2301.00621/) (2023)
- [Neural Estimation for Scaling Entropic Multimarginal Optimal Transport](https://mlsanity.com/paper/2506.00573/) (2025)
- [Universality of reservoir systems with recurrent neural networks](https://mlsanity.com/paper/2403.01900/) (2024)
- [k-Sliced Mutual Information: A Quantitative Study of Scalability with Dimension](https://mlsanity.com/paper/2206.08526/) (2022)
- [Lower Bounds on the MMSE of Adversarially Inferring Sensitive Features](https://mlsanity.com/paper/2505.09004/) (2025)
- [Approximating f-Divergences with Rank Statistics](https://mlsanity.com/paper/2601.22784/) (2026)
- [Scalable Minimum-Volume Simplex Estimation with Non-asymptotic Analysis](https://mlsanity.com/paper/2609.25576/) (2026)

## Abstract (arXiv)

Statistical distances (SDs), which quantify the dissimilarity between probability distributions, are central to machine learning and statistics. A modern method for estimating such distances from data relies on parametrizing a variational form by a neural network (NN) and optimizing it. These estimators are abundantly used in practice, but corresponding performance guarantees are partial and call for further exploration. In particular, there seems to be a fundamental tradeoff between the two sources of error involved: approximation and estimation. While the former needs the NN class to be rich and expressive, the latter relies on controlling complexity. This paper explores this tradeoff by means of non-asymptotic error bounds, focusing on three popular choices of SDs -- Kullback-Leibler divergence, chi-squared divergence, and squared Hellinger distance. Our analysis relies on non-asymptotic function approximation theorems and tools from empirical process theory. Numerical results validating the theory are also provided.