# VAE Approximation Error: ELBO and Exponential Families

arXiv: https://arxiv.org/abs/2102.09310 · HTML: https://mlsanity.com/paper/2102.09310/

## Key facts

- Cited by 12 papers in the map (data through 7 Oct 2026).
- Builds on 22 of its 38 references that are in the map.
- First posted on arXiv 18 Feb 2021.

## Builds on (most cited first, 22 of 22)

- [Deep Learning Face Attributes in the Wild](https://mlsanity.com/paper/1411.7766/) (2014)
- [Variational Inference with Normalizing Flows](https://mlsanity.com/paper/1505.05770/) (2015)
- [Improving Variational Inference with Inverse Autoregressive Flow](https://mlsanity.com/paper/1606.04934/) (2016)
- [NVAE: A Deep Hierarchical Variational Autoencoder](https://mlsanity.com/paper/2007.03898/) (2020)
- [Semi-Supervised Learning with Deep Generative Models](https://mlsanity.com/paper/1406.5298/) (2014)
- [Importance Weighted Autoencoders](https://mlsanity.com/paper/1509.00519/) (2015)
- [Ladder Variational Autoencoders](https://mlsanity.com/paper/1602.02282/) (2016)
- [Diagnosing and Enhancing VAE Models](https://mlsanity.com/paper/1903.05789/) (2019)
- [Inference Suboptimality in Variational Autoencoders](https://mlsanity.com/paper/1801.03558/) (2018)
- [Lagging Inference Networks and Posterior Collapse in Variational Autoencoders](https://mlsanity.com/paper/1901.05534/) (2019)
- [Don't Blame the ELBO! A Linear VAE Perspective on Posterior Collapse](https://mlsanity.com/paper/1911.02469/) (2019)
- [BIVA: A Very Deep Hierarchy of Latent Variables for Generative Modeling](https://mlsanity.com/paper/1902.02102/) (2019)
- [Hierarchical Variational Models](https://mlsanity.com/paper/1511.02386/) (2015)
- [Semi-Amortized Variational Autoencoders](https://mlsanity.com/paper/1802.02550/) (2018)
- [The Usual Suspects? Reassessing Blame for VAE Posterior Collapse](https://mlsanity.com/paper/1912.10702/) (2019)
- [Amortized Inference Regularization](https://mlsanity.com/paper/1805.08913/) (2018)
- [Accurate and Conservative Estimates of MRF Log-likelihood using Reverse Annealing](https://mlsanity.com/paper/1412.8566/) (2014)
- [Variational Deep Semantic Hashing for Text Documents](https://mlsanity.com/paper/1708.03436/) (2017)
- [NASH: Toward End-to-End Neural Architecture for Generative Semantic Hashing](https://mlsanity.com/paper/1805.05361/) (2018)
- [A Generalised Linear Model Framework for β-Variational Autoencoders based on Exponential Dispersion Families](https://mlsanity.com/paper/2006.06267/) (2020)
- [Pairwise Supervised Hashing with Bernoulli Variational Auto-Encoder and Self-Control Gradient Estimator](https://mlsanity.com/paper/2005.10477/) (2020)
- [Self-Supervised Bernoulli Autoencoders for Semi-Supervised Hashing](https://mlsanity.com/paper/2007.08799/) (2020)

## Built on by (most cited first, 12 of 12)

- [Posterior Collapse of a Linear Latent Variable Model](https://mlsanity.com/paper/2205.04009/) (2022)
- [Bi-fidelity Variational Auto-encoder for Uncertainty Quantification](https://mlsanity.com/paper/2305.16530/) (2023)
- [Symmetric Equilibrium Learning of VAEs](https://mlsanity.com/paper/2307.09883/) (2023)
- [Generalization in VAE and Diffusion Models: A Unified Information-Theoretic Analysis](https://mlsanity.com/paper/2506.00849/) (2025)
- [Inverse Design of Metamaterials with Manufacturing-Guiding Spectrum-to-Structure Conditional Diffusion Model](https://mlsanity.com/paper/2506.07083/) (2025)
- [On the Equality of the ELBO to a Sum of Entropies at Stationary Points of Learning](https://mlsanity.com/paper/2209.03077/) (2022)
- [Break The Spell Of Total Correlation In betaTCVAE](https://mlsanity.com/paper/2210.08794/) (2022)
- [Energy-Based Model for Accurate Estimation of Shapley Values in Feature Attribution](https://mlsanity.com/paper/2404.01078/) (2024)
- [A Unified Theory of Exact Inference and Learning in Exponential Family Latent Variable Models](https://mlsanity.com/paper/2404.19501/) (2024)
- [Beta-Sigma VAE: Separating beta and decoder variance in Gaussian variational autoencoder](https://mlsanity.com/paper/2409.09361/) (2024)
- [Game-theoretic distributed learning of generative models for heterogeneous data collections](https://mlsanity.com/paper/2511.01740/) (2025)
- [Deep Multivariate Models with Parametric Conditionals](https://mlsanity.com/paper/2602.01953/) (2026)

## Abstract (arXiv)

The importance of Variational Autoencoders reaches far beyond standalone generative models -- the approach is also used for learning latent representations and can be generalized to semi-supervised learning. This requires a thorough analysis of their commonly known shortcomings: posterior collapse and approximation errors. This paper analyzes VAE approximation errors caused by the combination of the ELBO objective and encoder models from conditional exponential families, including, but not limited to, commonly used conditionally independent discrete and continuous models. We characterize subclasses of generative models consistent with these encoder families. We show that the ELBO optimizer is pulled away from the likelihood optimizer towards the consistent subset and study this effect experimentally. Importantly, this subset can not be enlarged, and the respective error cannot be decreased, by considering deeper encoder/decoder networks.