# Optimizing Variational Representations of Divergences and Accelerating their Statistical Estimation

arXiv: https://arxiv.org/abs/2006.08781 · HTML: https://mlsanity.com/paper/2006.08781/

## Key facts

- Cited by 15 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 15 Jun 2020.

## Built on by (most cited first, 15 of 15)

- [Sample Complexity of Probability Divergences under Group Symmetry](https://mlsanity.com/paper/2302.01915/) (2023)
- [Lipschitz-regularized gradient flows and generative particle algorithms for high-dimensional scarce data](https://mlsanity.com/paper/2210.17230/) (2022)
- [Function-space regularized R'enyi divergences](https://mlsanity.com/paper/2210.04974/) (2022)
- [On f-Divergence Principled Domain Adaptation: An Improved Framework](https://mlsanity.com/paper/2402.01887/) (2024)
- [Outcome Assumptions and Duality Theory for Balancing Weights](https://mlsanity.com/paper/2203.09557/) (2022)
- [Generalization Bounds via Conditional f-Information](https://mlsanity.com/paper/2410.22887/) (2024)
- [Information Processing Equalities and the Information-Risk Bridge](https://mlsanity.com/paper/2207.11987/) (2022)
- [Statistical and Geometrical properties of regularized Kernel Kullback-Leibler divergence](https://mlsanity.com/paper/2408.16543/) (2024)
- [Statistical Error Bounds for GANs with Nonlinear Objective Functionals](https://mlsanity.com/paper/2406.16834/) (2024)
- [Generalized Balancing Weights via Deep Neural Networks](https://mlsanity.com/paper/2211.07533/) (2022)
- [Alpha-divergence loss function for neural density ratio estimation](https://mlsanity.com/paper/2402.02041/) (2024)
- [Neural Measures for learning distributions of Random PDEs](https://mlsanity.com/paper/2507.01687/) (2025)
- [A Unifying Framework for Some Directed Distances in Statistics](https://mlsanity.com/paper/2203.00863/) (2022)
- [Seeking Flat Minima over Diverse Surrogates for Improved Adversarial Transferability: A Theoretical Framework and Algorithmic Instantiation](https://mlsanity.com/paper/2504.16474/) (2025)
- [Two Approaches to Direct Estimation of Riesz Representers](https://mlsanity.com/paper/2603.20936/) (2026)

## Abstract (arXiv)

Variational representations of divergences and distances between high-dimensional probability distributions offer significant theoretical insights and practical advantages in numerous research areas. Recently, they have gained popularity in machine learning as a tractable and scalable approach for training probabilistic models and for statistically differentiating between data distributions. Their advantages include: 1) They can be estimated from data as statistical averages. 2) Such representations can leverage the ability of neural networks to efficiently approximate optimal solutions in function spaces. However, a systematic and practical approach to improving the tightness of such variational formulas, and accordingly accelerate statistical learning and estimation from data, is currently lacking. Here we develop such a methodology for building new, tighter variational representations of divergences. Our approach relies on improved objective functionals constructed via an auxiliary optimization problem. Furthermore, the calculation of the functional Hessian of objective functionals unveils the local curvature differences around the common optimal variational solution; this quantifies and orders the tightness gains between different variational representations. Finally, numerical simulations utilizing neural network optimization demonstrate that tighter representations can result in significantly faster learning and more accurate estimation of divergences in both synthetic and real datasets (of more than 1000 dimensions), often accelerated by nearly an order of magnitude.