# An Introduction to Variational Inference

arXiv: https://arxiv.org/abs/2108.13083 · HTML: https://mlsanity.com/paper/2108.13083/

## Key facts

- Cited by 12 papers in the map (data through 14 Oct 2026).
- Builds on 22 of its 51 references that are in the map.
- First posted on arXiv 30 Aug 2021.

## Builds on (most cited first, 22 of 22)

- [Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift](https://mlsanity.com/paper/1502.03167/) (2015)
- [A Style-Based Generator Architecture for Generative Adversarial Networks](https://mlsanity.com/paper/1812.04948/) (2018)
- [Deep Learning Face Attributes in the Wild](https://mlsanity.com/paper/1411.7766/) (2014)
- [Deep Inside Convolutional Networks: Visualising Image Classification Models and Saliency Maps](https://mlsanity.com/paper/1312.6034/) (2013)
- [Unsupervised Representation Learning with Deep Convolutional Generative Adversarial Networks](https://mlsanity.com/paper/1511.06434/) (2015)
- [Variational Inference: A Review for Statisticians](https://mlsanity.com/paper/1601.00670/) (2016)
- [InfoGAN: Interpretable Representation Learning by Information Maximizing Generative Adversarial Nets](https://mlsanity.com/paper/1606.03657/) (2016)
- [NVAE: A Deep Hierarchical Variational Autoencoder](https://mlsanity.com/paper/2007.03898/) (2020)
- [Tutorial on Variational Autoencoders](https://mlsanity.com/paper/1606.05908/) (2016)
- [Autoencoding beyond pixels using a learned similarity metric](https://mlsanity.com/paper/1512.09300/) (2015)
- [GraphVAE: Towards Generation of Small Graphs Using Variational Autoencoders](https://mlsanity.com/paper/1802.03480/) (2018)
- [Understanding disentangling in β-VAE](https://mlsanity.com/paper/1804.03599/) (2018)
- [Black Box Variational Inference](https://mlsanity.com/paper/1401.0118/) (2013)
- [Monte Carlo Gradient Estimation in Machine Learning](https://mlsanity.com/paper/1906.10652/) (2019)
- [Advances in Variational Inference](https://mlsanity.com/paper/1711.05597/) (2017)
- [Adversarial Latent Autoencoders](https://mlsanity.com/paper/2004.04467/) (2020)
- [Improved Variational Autoencoders for Text Modeling using Dilated Convolutions](https://mlsanity.com/paper/1702.08139/) (2017)
- [Anomaly Detection With Conditional Variational Autoencoders](https://mlsanity.com/paper/2010.05531/) (2020)
- [Amortized Inference Regularization](https://mlsanity.com/paper/1805.08913/) (2018)
- [The capacity of feedforward neural networks](https://mlsanity.com/paper/1901.00434/) (2019)
- [Celeste: Variational inference for a generative model of astronomical images](https://mlsanity.com/paper/1506.01351/) (2015)
- [Variational Inference for Stochastic Block Models from Sampled Data](https://mlsanity.com/paper/1707.04141/) (2017)

## Built on by (most cited first, 12 of 12)

- [InfoPrompt: Information-Theoretic Soft Prompt Tuning for Natural Language Understanding](https://mlsanity.com/paper/2306.04933/) (2023)
- [Amortized Variational Inference: A Systematic Review](https://mlsanity.com/paper/2209.10888/) (2022)
- [BioDiffusion: A Versatile Diffusion Model for Biomedical Signal Synthesis](https://mlsanity.com/paper/2401.10282/) (2024)
- [Quasi Black-Box Variational Inference with Natural Gradients for Bayesian Learning](https://mlsanity.com/paper/2205.11568/) (2022)
- [Disposable Transfer Learning for Selective Source Task Unlearning](https://mlsanity.com/paper/2308.09971/) (2023)
- [Manifold Gaussian Variational Bayes on the Precision Matrix](https://mlsanity.com/paper/2210.14598/) (2022)
- [BanglaFake: Constructing and Evaluating a Specialized Bengali Deepfake Audio Dataset](https://mlsanity.com/paper/2505.10885/) (2025)
- [Guided Data Generation for Understanding Model Behavior](https://mlsanity.com/paper/2502.06658/) (2025)
- [Monte Carlo Stochastic Depth for Uncertainty Estimation in Deep Learning](https://mlsanity.com/paper/2604.12719/) (2026)
- [Revisiting Neural Activation Coverage for Uncertainty Estimation](https://mlsanity.com/paper/2604.22360/) (2026)
- [From Static Personal Values to Contextualized Personalization: Bayesian Personalized Value Alignment for LLMs](https://mlsanity.com/paper/2609.28942/) (2026)
- [Bridging Risk Approximation Gaps in Model Predictive Task Sampling via In-Context Modeling](https://mlsanity.com/paper/2610.14913/) (2026)

## Abstract (arXiv)

Approximating complex probability densities is a core problem in modern statistics. In this paper, we introduce the concept of Variational Inference (VI), a popular method in machine learning that uses optimization techniques to estimate complex probability densities. This property allows VI to converge faster than classical methods, such as, Markov Chain Monte Carlo sampling. Conceptually, VI works by choosing a family of probability density functions and then finding the one closest to the actual probability density -- often using the Kullback-Leibler (KL) divergence as the optimization metric. We introduce the Evidence Lower Bound to tractably compute the approximated probability density and we review the ideas behind mean-field variational inference. Finally, we discuss the applications of VI to variational auto-encoders (VAE) and VAE-Generative Adversarial Network (VAE-GAN). With this paper, we aim to explain the concept of VI and assist in future research with this approach.