# Don't Blame the ELBO! A Linear VAE Perspective on Posterior Collapse

arXiv: https://arxiv.org/abs/1911.02469 · HTML: https://mlsanity.com/paper/1911.02469/

## Key facts

- Cited by 121 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 6 Nov 2019.

## Built on by (most cited first, 30 of 121)

- [HuMoR: 3D Human Motion Model for Robust Pose Estimation](https://mlsanity.com/paper/2105.04668/) (2021)
- [Autoregressive Diffusion Model for Graph Generation](https://mlsanity.com/paper/2307.08849/) (2023)
- [Posterior Collapse and Latent Variable Non-identifiability](https://mlsanity.com/paper/2301.00537/) (2023)
- [Embrace the Gap: VAEs Perform Independent Mechanism Analysis](https://mlsanity.com/paper/2206.02416/) (2022)
- [Symmetry Induces Structure and Constraint of Learning](https://mlsanity.com/paper/2309.16932/) (2023)
- [Latent Thought Models with Variational Bayes Inference-Time Computation](https://mlsanity.com/paper/2502.01567/) (2025)
- [Solving Inverse Problems by Joint Posterior Maximization with Autoencoding Prior](https://mlsanity.com/paper/2103.01648/) (2021)
- [Exact Solutions of a Deep Linear Network](https://mlsanity.com/paper/2202.04777/) (2022)
- [Bi-level Feature Alignment for Versatile Image Translation and Manipulation](https://mlsanity.com/paper/2107.03021/) (2021)
- [Marginal Contrastive Correspondence for Guided Image Generation](https://mlsanity.com/paper/2204.00442/) (2022)
- [ContrastVAE: Contrastive Variational AutoEncoder for Sequential Recommendation](https://mlsanity.com/paper/2209.00456/) (2022)
- [What shapes the loss landscape of self-supervised learning?](https://mlsanity.com/paper/2210.00638/) (2022)
- [Complexity Matters: Rethinking the Latent Space for Generative Modeling](https://mlsanity.com/paper/2307.08283/) (2023)
- [Posterior Collapse of a Linear Latent Variable Model](https://mlsanity.com/paper/2205.04009/) (2022)
- [Multi-Rate VAE: Train Once, Get the Full Rate-Distortion Curve](https://mlsanity.com/paper/2212.03905/) (2022)
- [Euro-PVI: Pedestrian Vehicle Interactions in Dense Urban Centers](https://mlsanity.com/paper/2106.12442/) (2021)
- [Preventing Oversmoothing in VAE via Generalized Variance Parameterization](https://mlsanity.com/paper/2102.08663/) (2021)
- [Learning Energy-Based Prior Model with Diffusion-Amortized MCMC](https://mlsanity.com/paper/2310.03218/) (2023)
- [VAE Approximation Error: ELBO and Exponential Families](https://mlsanity.com/paper/2102.09310/) (2021)
- [Learning Dynamics in Linear VAE: Posterior Collapse Threshold, Superfluous Latent Space Pitfalls, and Speedup with KL Annealing](https://mlsanity.com/paper/2310.15440/) (2023)
- [Variational autoencoders in the presence of low-dimensional data: landscape and implicit bias](https://mlsanity.com/paper/2112.06868/) (2021)
- [Revisiting Structured Variational Autoencoders](https://mlsanity.com/paper/2305.16543/) (2023)
- [LLM Reading Tea Leaves: Automatically Evaluating Topic Models with Large Language Models](https://mlsanity.com/paper/2406.09008/) (2024)
- [How do Variational Autoencoders Learn? Insights from Representational Similarity](https://mlsanity.com/paper/2205.08399/) (2022)
- [Event Temporal Relation Extraction with Bayesian Translational Model](https://mlsanity.com/paper/2302.04985/) (2023)
- [Auto-regressive Image Synthesis with Integrated Quantization](https://mlsanity.com/paper/2207.10776/) (2022)
- [Distributional Learning of Variational AutoEncoder: Application to Synthetic Data Generation](https://mlsanity.com/paper/2302.11294/) (2023)
- [Remove Symmetries to Control Model Expressivity and Improve Optimization](https://mlsanity.com/paper/2408.15495/) (2024)
- [Augmented Deep Contexts for Spatially Embedded Video Coding](https://mlsanity.com/paper/2505.05309/) (2025)
- [Be More Active! Understanding the Differences between Mean and Sampled Representations of Variational Autoencoders](https://mlsanity.com/paper/2109.12679/) (2021)

## Abstract (arXiv)

Posterior collapse in Variational Autoencoders (VAEs) arises when the variational posterior distribution closely matches the prior for a subset of latent variables. This paper presents a simple and intuitive explanation for posterior collapse through the analysis of linear VAEs and their direct correspondence with Probabilistic PCA (pPCA). We explain how posterior collapse may occur in pPCA due to local maxima in the log marginal likelihood. Unexpectedly, we prove that the ELBO objective for the linear VAE does not introduce additional spurious local maxima relative to log marginal likelihood. We show further that training a linear VAE with exact variational inference recovers an identifiable global maximum corresponding to the principal component directions. Empirically, we find that our linear analysis is predictive even for high-capacity, non-linear VAEs and helps explain the relationship between the observation noise, local maxima, and posterior collapse in deep Gaussian VAEs.