mlsanityData to 7 Oct 2026

High-dimensional Asymptotics of VAEs: Threshold of Posterior Collapse and Dataset-Size Dependence of Rate-Distortion Curve

arXiv 2309.07663 ↗First posted 14 Sep 2023stat.ML cs.LGcited by 5

Not on the map. Why?

Abstract (arXiv)

In variational autoencoders (VAEs), the variational posterior often collapses to the prior, known as posterior collapse, which leads to poor representation learning quality. An adjustable hyperparameter beta has been introduced in VAEs to address this issue. This study sharply evaluates the conditions under which the posterior collapse occurs with respect to beta and dataset size by analyzing a minimal VAE in a high-dimensional limit. Additionally, this setting enables the evaluation of the rate-distortion curve of the VAE. Our results show that, unlike typical regularization parameters, VAEs face "inevitable posterior collapse" beyond a certain beta threshold, regardless of dataset size. Moreover, the dataset-size dependence of the derived rate-distortion curve suggests that relatively large datasets are required to achieve a rate-distortion curve with high rates. These findings robustly explain generalization behavior observed in various real datasets with highly non-linear VAEs.

Key facts. Cited by 5 papers in the map (data through 7 Oct 2026). Builds on 39 of its 82 references that are in the map. First posted on arXiv 14 Sep 2023.

Builds on

39 of 82 references.

References · 39

  1. 2014Adam: A Method for Stochastic Optimizationcited by 36,960
  2. 2015Rethinking the Inception Architecture for Computer Visioncited by 5,451
  3. 2017Fashion-MNIST: a Novel Image Dataset for Benchmarking Machine Learning Algorithmscited by 4,283
  4. 2020NVAE: A Deep Hierarchical Variational Autoencodercited by 635
  5. 2019Surprises in High-Dimensional Ridgeless Least Squares Interpolationcited by 508
  6. 2019The generalization error of random features regression: Precise asymptotics and double descent curvecited by 410
  7. 2017Causal Effect Inference with Deep Latent-Variable Modelscited by 408
  8. 2018A Probabilistic U-Net for Segmentation of Ambiguous Imagescited by 281
  9. 2020Very Deep VAEs Generalize Autoregressive Models and Can Outperform Them on Imagescited by 261
  10. 2016Variational Deep Embedding: An Unsupervised and Generative Approach to Clusteringcited by 219
  11. 2020Spectrum Dependent Learning Curves in Kernel Regression and Wide Neural Networkscited by 200
  12. 2020Generalisation error in learning with random features and the hidden manifold modelcited by 126
27 more
  1. 2021Learning curves of generic features maps for realistic datasets with a teacher-student modelcited by 126
  2. 2019Don't Blame the ELBO! A Linear VAE Perspective on Posterior Collapsecited by 120
  3. 2017Optimal Errors and Phase Transitions in High-Dimensional Generalized Linear Modelscited by 115
  4. 2020The Gaussian equivalence of generative models for learning with shallow neural networkscited by 113
  5. 2019Improved Conditional VRNNs for Video Predictioncited by 104
  6. 2022Lossy Compression with Gaussian Diffusioncited by 87
  7. 2022Universality of empirical risk minimizationcited by 73
  8. 2020The role of regularization in classification of high-dimensional noisy Gaussian mixturecited by 63
  9. 2018The committee machine: Computational to statistical gaps in learning a two-layers neural networkcited by 58
  10. 2020Simple and Effective VAE Training with Calibrated Decoderscited by 57
  11. 2020Generalization error in high-dimensional perceptrons: Approaching Bayes error with convex optimizationcited by 51
  12. 2018On the Spectrum of Random Features Maps of High Dimensional Datacited by 27
  13. 2022The dynamics of representation learning in shallow, non-linear autoencoderscited by 27
  14. 2018Thermodynamics of Restricted Boltzmann Machines and related learning dynamicscited by 25
  15. 2021An Information-Theoretic Justification for Model Pruningcited by 23
  16. 2023High-dimensional Asymptotics of Denoising Autoencoderscited by 19
  17. 2022Posterior Collapse of a Linear Latent Variable Modelcited by 18
  18. 2022Multi-Rate VAE: Train Once, Get the Full Rate-Distortion Curvecited by 18
  19. 2020Evaluating Lossy Compression Rates of Deep Generative Modelscited by 16
  20. 2015Phase Transitions in Sparse PCAcited by 15
  21. 2018Rate Distortion For Model Compression: From Theory To Practicecited by 14
  22. 2021Analysis of feature learning in weight-tied autoencoders via the mean field lenscited by 13
  23. 2020Interpreting Rate-Distortion of Variational Autoencoder and Using Model Uncertainty for Anomaly Detectioncited by 10
  24. 2020A Generalised Linear Model Framework for β-Variational Autoencoders based on Exponential Dispersion Familiescited by 7
  25. 2020Quantitative Understanding of VAE as a Non-linearly Scaled Isometric Embeddingcited by 7
  26. 2019Exact Rate-Distortion in Autoencoders via Echo Noisecited by 5
  27. 2019Phase transition in PCA with missing data: Reduced signal-to-noise ratio, not sample size!cited by 4

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 1; 2024: 1; 2025: 1; 2026: 2.

5 papers.

  1. 2023Learning Dynamics in Linear VAE: Posterior Collapse Threshold, Superfluous Latent Space Pitfalls, and Speedup with KL Annealingcited by 12
  2. 2025Posterior Collapse as a Phase Transition in Variational Autoencoderscited by 4
  3. 2024Statistical Mechanics of Min-Max Problemscited by 1
  4. 2026Latent Spectroscopy: Posterior Collapse as a Featurecited by 1
  5. 2026Mutual Information Collapse Explains Disentanglement Failure in β-VAEscited by 0