# Variational Representations and Neural Network Estimation of R'enyi Divergences

arXiv: https://arxiv.org/abs/2007.03814 · HTML: https://mlsanity.com/paper/2007.03814/

## Key facts

- Cited by 20 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 7 Jul 2020.

## Built on by (most cited first, 20 of 20)

- [Towards a Complete Analysis of Langevin Monte Carlo: Beyond Poincar'e Inequality](https://mlsanity.com/paper/2303.03589/) (2023)
- [Information Theoretic Guarantees For Policy Alignment In Large Language Models](https://mlsanity.com/paper/2406.05883/) (2024)
- [Structure-preserving GANs](https://mlsanity.com/paper/2202.01129/) (2022)
- [Sample Complexity of Probability Divergences under Group Symmetry](https://mlsanity.com/paper/2302.01915/) (2023)
- [RenyiCL: Contrastive Representation Learning with Skew Renyi Divergence](https://mlsanity.com/paper/2208.06270/) (2022)
- [Function-space regularized R'enyi divergences](https://mlsanity.com/paper/2210.04974/) (2022)
- [Quantum Neural Estimation of Entropies](https://mlsanity.com/paper/2307.01171/) (2023)
- [Sequentially Auditing Differential Privacy](https://mlsanity.com/paper/2509.07055/) (2025)
- [Adaptive importance sampling for heavy-tailed distributions via α-divergence minimization](https://mlsanity.com/paper/2310.16653/) (2023)
- [Diffusion Secant Alignment for Score-Based Density Ratio Estimation](https://mlsanity.com/paper/2509.04852/) (2025)
- [Performance Guarantees for Quantum Neural Estimation of Entropies](https://mlsanity.com/paper/2511.19289/) (2025)
- [Model Uncertainty and Correctability for Directed Graphical Models](https://mlsanity.com/paper/2107.08179/) (2021)
- [Lower Bounds on the Bayesian Risk via Information Measures](https://mlsanity.com/paper/2303.12497/) (2023)
- [Classification Utility, Fairness, and Compactness via Tunable Information Bottleneck and R'enyi Measures](https://mlsanity.com/paper/2206.10043/) (2022)
- [Alpha-divergence loss function for neural density ratio estimation](https://mlsanity.com/paper/2402.02041/) (2024)
- [Generalization Bounds for Quantum Learning via R'enyi Divergences](https://mlsanity.com/paper/2505.11025/) (2025)
- [Bounds on the Excess Minimum Risk via Generalized Information Divergence Measures](https://mlsanity.com/paper/2505.24117/) (2025)
- [IRIS: Interpolative R'enyi Iterative Self-play for Large Language Model Fine-Tuning](https://mlsanity.com/paper/2604.20933/) (2026)
- [Unlearning with Asymmetric Sources: Improved Unlearning-Utility Trade-off with Public Data](https://mlsanity.com/paper/2605.11170/) (2026)
- [Optimal Guarantees for Auditing R'enyi Differentially Private Machine Learning](https://mlsanity.com/paper/2605.21938/) (2026)

## Abstract (arXiv)

We derive a new variational formula for the R'enyi family of divergences, Rα(Q|P), between probability measures Q and P. Our result generalizes the classical Donsker-Varadhan variational formula for the Kullback-Leibler divergence. We further show that this R'enyi variational formula holds over a range of function spaces; this leads to a formula for the optimizer under very weak assumptions and is also key in our development of a consistency theory for R'enyi divergence estimators. By applying this theory to neural-network estimators, we show that if a neural network family satisfies one of several strengthened versions of the universal approximation property then the corresponding R'enyi divergence estimator is consistent. In contrast to density-estimator based methods, our estimators involve only expectations under Q and P and hence are more effective in high dimensional systems. We illustrate this via several numerical examples of neural network estimation in systems of up to 5000 dimensions.