# Fixing a Broken ELBO

arXiv: https://arxiv.org/abs/1711.00464 · HTML: https://mlsanity.com/paper/1711.00464/

## Key facts

- Cited by 12 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 1 Nov 2017.

## Built on by (most cited first, 12 of 12)

- [Neither hype nor gloom do DNNs justice](https://mlsanity.com/paper/2312.05355/) (2023)
- [Don't Throw it Away! The Utility of Unlabeled Data in Fair Decision Making](https://mlsanity.com/paper/2205.04790/) (2022)
- [Transfer Learning Based Multi-Objective Genetic Algorithm for Dynamic Community Detection](https://mlsanity.com/paper/2109.15136/) (2021)
- [Expressive Communication: A Common Framework for Evaluating Developments in Generative Models and Steering Interfaces](https://mlsanity.com/paper/2111.14951/) (2021)
- [Deep Music Information Dynamics](https://mlsanity.com/paper/2102.01133/) (2021)
- [Beyond the limitations of any imaginable mechanism: large language models and psycholinguistics](https://mlsanity.com/paper/2303.00077/) (2023)
- [Fenchel-Young Variational Learning](https://mlsanity.com/paper/2502.10295/) (2025)
- [IB-GAN: Disentangled Representation Learning with Information Bottleneck Generative Adversarial Networks](https://mlsanity.com/paper/2510.20165/) (2025)
- [Speech representation learning: Learning bidirectional encoders with single-view, multi-view, and multi-task methods](https://mlsanity.com/paper/2308.00129/) (2023)
- [Fixing the problems of deep neural networks will require better training data and learning algorithms](https://mlsanity.com/paper/2311.12819/) (2023)
- [Taming Audio VAEs via Target-KL Regularization](https://mlsanity.com/paper/2605.17085/) (2026)
- [SPIBER: Reconstructing Free Energy Landscapes from Short, Unconverged Trajectories with Generative Flow Networks](https://mlsanity.com/paper/2609.22663/) (2026)

## Abstract (arXiv)

Recent work in unsupervised representation learning has focused on learning deep directed latent-variable models. Fitting these models by maximizing the marginal likelihood or evidence is typically intractable, thus a common approximation is to maximize the evidence lower bound (ELBO) instead. However, maximum likelihood training (whether exact or approximate) does not necessarily result in a good latent representation, as we demonstrate both theoretically and empirically. In particular, we derive variational lower and upper bounds on the mutual information between the input and the latent variable, and use these bounds to derive a rate-distortion curve that characterizes the tradeoff between compression and reconstruction accuracy. Using this framework, we demonstrate that there is a family of models with identical ELBO, but different quantitative and qualitative characteristics. Our framework also suggests a simple new method to ensure that latent variable models with powerful stochastic decoders do not ignore their latent code.