# R'enyi Divergence Variational Inference

arXiv: https://arxiv.org/abs/1602.02311 · HTML: https://mlsanity.com/paper/1602.02311/

## Key facts

- Cited by 104 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 6 Feb 2016.

## Built on by (most cited first, 30 of 104)

- [Generative Time Series Forecasting with Diffusion, Denoise, and Disentanglement](https://mlsanity.com/paper/2301.03028/) (2023)
- [Beyond ELBOs: A Large-Scale Evaluation of Variational Methods for Sampling](https://mlsanity.com/paper/2406.07423/) (2024)
- [Variational methods for simulation-based inference](https://mlsanity.com/paper/2203.04176/) (2022)
- [Challenges and Opportunities in High-dimensional Variational Inference](https://mlsanity.com/paper/2103.01085/) (2021)
- [InfoLM: A New Metric to Evaluate Summarization & Data2Text Generation](https://mlsanity.com/paper/2112.01589/) (2021)
- [Variational Refinement for Importance Sampling Using the Forward Kullback-Leibler Divergence](https://mlsanity.com/paper/2106.15980/) (2021)
- [Transport meets Variational Inference: Controlled Monte Carlo Diffusions](https://mlsanity.com/paper/2307.01050/) (2023)
- [Sliced Wasserstein Variational Inference](https://mlsanity.com/paper/2207.13177/) (2022)
- [On the Convergence of Coordinate Ascent Variational Inference](https://mlsanity.com/paper/2306.01122/) (2023)
- [t3-Variational Autoencoder: Learning Heavy-tailed Data with Student's t and Power Divergence](https://mlsanity.com/paper/2312.01133/) (2023)
- [Flow Density Control: Generative Optimization Beyond Entropy-Regularized Fine-Tuning](https://mlsanity.com/paper/2511.22640/) (2025)
- [Amortized Variational Inference: A Systematic Review](https://mlsanity.com/paper/2209.10888/) (2022)
- [Understanding Multimodal LLMs Under Distribution Shifts: An Information-Theoretic Approach](https://mlsanity.com/paper/2502.00577/) (2025)
- [Alpha-divergence Variational Inference Meets Importance Weighted Auto-Encoders: Methodology and Asymptotics](https://mlsanity.com/paper/2210.06226/) (2022)
- [Mind the Gap when Conditioning Amortised Inference in Sequential Latent-Variable Models](https://mlsanity.com/paper/2101.07046/) (2021)
- [Multiple Importance Sampling ELBO and Deep Ensembles of Variational Approximations](https://mlsanity.com/paper/2202.10951/) (2022)
- [Variational Inference MPC using Tsallis Divergence](https://mlsanity.com/paper/2104.00241/) (2021)
- [PriorVAE: Encoding spatial priors with VAEs for small-area estimation](https://mlsanity.com/paper/2110.10422/) (2021)
- [Fat-Tailed Variational Inference with Anisotropic Tail Adaptive Flows](https://mlsanity.com/paper/2205.07918/) (2022)
- [Variational Inference for Uncertainty Quantification: an Analysis of Trade-offs](https://mlsanity.com/paper/2403.13748/) (2024)
- [Beyond Squared Error: Exploring Loss Design for Enhanced Training of Generative Flow Networks](https://mlsanity.com/paper/2410.02596/) (2024)
- [Empirical Evaluation of Biased Methods for Alpha Divergence Minimization](https://mlsanity.com/paper/2105.06587/) (2021)
- [Loss Functions and Operators Generated by f-Divergences](https://mlsanity.com/paper/2501.18537/) (2025)
- [The Representation Jensen-Shannon Divergence](https://mlsanity.com/paper/2305.16446/) (2023)
- [Variational Inference in Location-Scale Families: Exact Recovery of the Mean and Correlation Matrix](https://mlsanity.com/paper/2410.11067/) (2024)
- [Spiking Generative Adversarial Networks With a Neural Network Discriminator: Local Training, Bayesian Models, and Continual Meta-Learning](https://mlsanity.com/paper/2111.01750/) (2021)
- [RenyiCL: Contrastive Representation Learning with Skew Renyi Divergence](https://mlsanity.com/paper/2208.06270/) (2022)
- [Bridging the Gap Between Variational Inference and Wasserstein Gradient Flows](https://mlsanity.com/paper/2310.20090/) (2023)
- [Pseudo-Spherical Contrastive Divergence](https://mlsanity.com/paper/2111.00780/) (2021)
- [Function-space regularized R'enyi divergences](https://mlsanity.com/paper/2210.04974/) (2022)

## Abstract (arXiv)

This paper introduces the variational R'enyi bound (VR) that extends traditional variational inference to R'enyi's alpha-divergences. This new family of variational methods unifies a number of existing approaches, and enables a smooth interpolation from the evidence lower-bound to the log (marginal) likelihood that is controlled by the value of alpha that parametrises the divergence. The reparameterization trick, Monte Carlo approximation and stochastic optimisation methods are deployed to obtain a tractable and unified framework for optimisation. We further consider negative alpha values and propose a novel variational inference method as a new special case in the proposed framework. Experiments on Bayesian neural networks and variational auto-encoders demonstrate the wide applicability of the VR bound.