# Function-space regularized R'enyi divergences

arXiv: https://arxiv.org/abs/2210.04974 · HTML: https://mlsanity.com/paper/2210.04974/

## Key facts

- Cited by 7 papers in the map (data through 14 Oct 2026).
- Builds on 17 of its 40 references that are in the map.
- First posted on arXiv 10 Oct 2022.

## Builds on (most cited first, 17 of 17)

- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [Representation Learning with Contrastive Predictive Coding](https://mlsanity.com/paper/1807.03748/) (2018)
- [Improved Training of Wasserstein GANs](https://mlsanity.com/paper/1704.00028/) (2017)
- [MINE: Mutual Information Neural Estimation](https://mlsanity.com/paper/1801.04062/) (2018)
- [Renyi Differential Privacy](https://mlsanity.com/paper/1702.07476/) (2017)
- [CLUB: A Contrastive Log-ratio Upper Bound of Mutual Information](https://mlsanity.com/paper/2006.12013/) (2020)
- [Formal Limitations on the Measurement of Mutual Information](https://mlsanity.com/paper/1811.04251/) (2018)
- [Understanding the Limitations of Variational Mutual Information Estimators](https://mlsanity.com/paper/1910.06222/) (2019)
- [R'enyi Divergence Variational Inference](https://mlsanity.com/paper/1602.02311/) (2016)
- [Variational Representations and Neural Network Estimation of R'enyi Divergences](https://mlsanity.com/paper/2007.03814/) (2020)
- [Group Equivariant Generative Adversarial Networks](https://mlsanity.com/paper/2005.01683/) (2020)
- [Structure-preserving GANs](https://mlsanity.com/paper/2202.01129/) (2022)
- [Optimizing Variational Representations of Divergences and Accelerating their Statistical Estimation](https://mlsanity.com/paper/2006.08781/) (2020)
- [Formulation and properties of a divergence used to compare probability measures without absolute continuity](https://mlsanity.com/paper/1911.07422/) (2019)
- [Cumulant GAN](https://mlsanity.com/paper/2006.06625/) (2020)
- [RenyiCL: Contrastive Representation Learning with Skew Renyi Divergence](https://mlsanity.com/paper/2208.06270/) (2022)
- [Bayesian brains and the R'enyi divergence](https://mlsanity.com/paper/2107.05438/) (2021)

## Built on by (most cited first, 7 of 7)

- [Sample Complexity of Probability Divergences under Group Symmetry](https://mlsanity.com/paper/2302.01915/) (2023)
- [Combining Wasserstein-1 and Wasserstein-2 proximals: robust manifold learning via well-posed generative flows](https://mlsanity.com/paper/2407.11901/) (2024)
- [Optimal Transport Regularized Divergences: Application to Adversarial Robustness](https://mlsanity.com/paper/2309.03791/) (2023)
- [Orlicz regrets to consistently bound statistics of random variables with an application to environmental indicators](https://mlsanity.com/paper/2310.05168/) (2023)
- [Concentration Inequalities for Stochastic Optimization of Unbounded Objective Functions with Application to Denoising Score Matching](https://mlsanity.com/paper/2502.08628/) (2025)
- [Privacy Amplification Persists under Unlimited Synthetic Data Release](https://mlsanity.com/paper/2602.04895/) (2026)
- [Optimal Guarantees for Auditing R'enyi Differentially Private Machine Learning](https://mlsanity.com/paper/2605.21938/) (2026)

## Abstract (arXiv)

We propose a new family of regularized R'enyi divergences parametrized not only by the order α but also by a variational function space. These new objects are defined by taking the infimal convolution of the standard R'enyi divergence with the integral probability metric (IPM) associated with the chosen function space. We derive a novel dual variational representation that can be used to construct numerically tractable divergence estimators. This representation avoids risk-sensitive terms and therefore exhibits lower variance, making it well-behaved when α>1; this addresses a notable weakness of prior approaches. We prove several properties of these new divergences, showing that they interpolate between the classical R'enyi divergences and IPMs. We also study the α→∞ limit, which leads to a regularized worst-case-regret and a new variational representation in the classical case. Moreover, we show that the proposed regularized R'enyi divergences inherit features from IPMs such as the ability to compare distributions that are not absolutely continuous, e.g., empirical measures and distributions with low-dimensional support. We present numerical results on both synthetic and real datasets, showing the utility of these new divergences in both estimation and GAN training applications; in particular, we demonstrate significantly reduced variance and improved training performance.