mlsanityData to 7 Oct 2026

VAE Approximation Error: ELBO and Exponential Families

arXiv 2102.09310 ↗First posted 18 Feb 2021cs.LG cs.CV stat.MLcited by 12

Not on the map. Why?

Abstract (arXiv)

The importance of Variational Autoencoders reaches far beyond standalone generative models -- the approach is also used for learning latent representations and can be generalized to semi-supervised learning. This requires a thorough analysis of their commonly known shortcomings: posterior collapse and approximation errors. This paper analyzes VAE approximation errors caused by the combination of the ELBO objective and encoder models from conditional exponential families, including, but not limited to, commonly used conditionally independent discrete and continuous models. We characterize subclasses of generative models consistent with these encoder families. We show that the ELBO optimizer is pulled away from the likelihood optimizer towards the consistent subset and study this effect experimentally. Importantly, this subset can not be enlarged, and the respective error cannot be decreased, by considering deeper encoder/decoder networks.

Key facts. Cited by 12 papers in the map (data through 7 Oct 2026). Builds on 22 of its 38 references that are in the map. First posted on arXiv 18 Feb 2021.

Builds on

22 of 38 references.

References · 22

  1. 2014Deep Learning Face Attributes in the Wildcited by 3,679
  2. 2015Variational Inference with Normalizing Flowscited by 2,161
  3. 2016Improving Variational Inference with Inverse Autoregressive Flowcited by 636
  4. 2020NVAE: A Deep Hierarchical Variational Autoencodercited by 635
  5. 2014Semi-Supervised Learning with Deep Generative Modelscited by 494
  6. 2015Importance Weighted Autoencoderscited by 454
  7. 2016Ladder Variational Autoencoderscited by 315
  8. 2019Diagnosing and Enhancing VAE Modelscited by 192
  9. 2018Inference Suboptimality in Variational Autoencoderscited by 131
  10. 2019Lagging Inference Networks and Posterior Collapse in Variational Autoencoderscited by 124
  11. 2019Don't Blame the ELBO! A Linear VAE Perspective on Posterior Collapsecited by 120
  12. 2019BIVA: A Very Deep Hierarchy of Latent Variables for Generative Modelingcited by 109
10 more
  1. 2015Hierarchical Variational Modelscited by 89
  2. 2018Semi-Amortized Variational Autoencoderscited by 83
  3. 2019The Usual Suspects? Reassessing Blame for VAE Posterior Collapsecited by 52
  4. 2018Amortized Inference Regularizationcited by 30
  5. 2014Accurate and Conservative Estimates of MRF Log-likelihood using Reverse Annealingcited by 18
  6. 2017Variational Deep Semantic Hashing for Text Documentscited by 12
  7. 2018NASH: Toward End-to-End Neural Architecture for Generative Semantic Hashingcited by 11
  8. 2020A Generalised Linear Model Framework for β-Variational Autoencoders based on Exponential Dispersion Familiescited by 7
  9. 2020Pairwise Supervised Hashing with Bernoulli Variational Auto-Encoder and Self-Control Gradient Estimatorcited by 6
  10. 2020Self-Supervised Bernoulli Autoencoders for Semi-Supervised Hashingcited by 2

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2022: 3; 2023: 2; 2024: 3; 2025: 3; 2026: 1.

12 papers.

  1. 2022Posterior Collapse of a Linear Latent Variable Modelcited by 18
  2. 2023Bi-fidelity Variational Auto-encoder for Uncertainty Quantificationcited by 8
  3. 2023Symmetric Equilibrium Learning of VAEscited by 3
  4. 2025Generalization in VAE and Diffusion Models: A Unified Information-Theoretic Analysiscited by 2
  5. 2025Inverse Design of Metamaterials with Manufacturing-Guiding Spectrum-to-Structure Conditional Diffusion Modelcited by 2
  6. 2022On the Equality of the ELBO to a Sum of Entropies at Stationary Points of Learningcited by 1
  7. 2022Break The Spell Of Total Correlation In betaTCVAEcited by 1
  8. 2024Energy-Based Model for Accurate Estimation of Shapley Values in Feature Attributioncited by 1
  9. 2024A Unified Theory of Exact Inference and Learning in Exponential Family Latent Variable Modelscited by 0
  10. 2024Beta-Sigma VAE: Separating beta and decoder variance in Gaussian variational autoencodercited by 0
  11. 2025Game-theoretic distributed learning of generative models for heterogeneous data collectionscited by 0
  12. 2026Deep Multivariate Models with Parametric Conditionalscited by 0