mlsanityData to 7 Oct 2026

Learning Dynamics in Linear VAE: Posterior Collapse Threshold, Superfluous Latent Space Pitfalls, and Speedup with KL Annealing

arXiv 2310.15440 ↗First posted 24 Oct 2023stat.ML cond-mat.dis-nn cs.LGcited by 12

Not on the map. Why?

Abstract (arXiv)

Variational autoencoders (VAEs) face a notorious problem wherein the variational posterior often aligns closely with the prior, a phenomenon known as posterior collapse, which hinders the quality of representation learning. To mitigate this problem, an adjustable hyperparameter β and a strategy for annealing this parameter, called KL annealing, are proposed. This study presents a theoretical analysis of the learning dynamics in a minimal VAE. It is rigorously proved that the dynamics converge to a deterministic process within the limit of large input dimensions, thereby enabling a detailed dynamical analysis of the generalization error. Furthermore, the analysis shows that the VAE initially learns entangled representations and gradually acquires disentangled representations. A fixed-point analysis of the deterministic process reveals that when β exceeds a certain threshold, posterior collapse becomes inevitable regardless of the learning period. Additionally, the superfluous latent variables for the data-generative factors lead to overfitting of the background noise; this adversely affects both generalization and learning convergence. The analysis further unveiled that appropriately tuned KL annealing can accelerate convergence.

Key facts. Cited by 12 papers in the map (data through 7 Oct 2026). Builds on 27 of its 69 references that are in the map. First posted on arXiv 24 Oct 2023.

Builds on

27 of 69 references.

References · 27

  1. 2020NVAE: A Deep Hierarchical Variational Autoencodercited by 635
  2. 2015Generating Sentences from a Continuous Spacecited by 607
  3. 2019The generalization error of random features regression: Precise asymptotics and double descent curvecited by 410
  4. 2017Causal Effect Inference with Deep Latent-Variable Modelscited by 408
  5. 2020Very Deep VAEs Generalize Autoregressive Models and Can Outperform Them on Imagescited by 261
  6. 2017Learning Discourse-level Diversity for Neural Dialog Models using Conditional Variational Autoencoderscited by 222
  7. 2016Variational Deep Embedding: An Unsupervised and Generative Approach to Clusteringcited by 219
  8. 2019Cyclical Annealing Schedule: A Simple Approach to Mitigating KL Vanishingcited by 191
  9. 2018A Hierarchical Latent Vector Model for Learning Long-Term Structure in Musiccited by 131
  10. 2019Don't Blame the ELBO! A Linear VAE Perspective on Posterior Collapsecited by 120
  11. 2020The Gaussian equivalence of generative models for learning with shallow neural networkscited by 113
  12. 2019Dynamics of stochastic gradient descent for two-layer neural networks in the teacher-student setupcited by 112
15 more
  1. 2018Semi-Amortized Variational Autoencoderscited by 83
  2. 2017Improved Variational Autoencoders for Text Modeling using Dilated Convolutionscited by 81
  3. 2022Phase diagram of Stochastic Gradient Descent in high-dimensional two-layer neural networkscited by 50
  4. 2018Avoiding Latent Variable Collapse With Generative Skip Modelscited by 47
  5. 2018On the Spectrum of Random Features Maps of High Dimensional Datacited by 27
  6. 2022The dynamics of representation learning in shallow, non-linear autoencoderscited by 27
  7. 2022Posterior Collapse of a Linear Latent Variable Modelcited by 18
  8. 2022Multi-Rate VAE: Train Once, Get the Full Rate-Distortion Curvecited by 18
  9. 2015Phase Transitions in Sparse PCAcited by 15
  10. 2018A Solvable High-Dimensional Model of GANcited by 13
  11. 2020Interpreting Rate-Distortion of Variational Autoencoder and Using Model Uncertainty for Anomaly Detectioncited by 10
  12. 2020A Generalised Linear Model Framework for β-Variational Autoencoders based on Exponential Dispersion Familiescited by 7
  13. 2018Subspace Estimation from Incomplete Observations: A High-Dimensional Analysiscited by 6
  14. 2023High-dimensional Asymptotics of VAEs: Threshold of Posterior Collapse and Dataset-Size Dependence of Rate-Distortion Curvecited by 5
  15. 2019Phase transition in PCA with missing data: Reduced signal-to-noise ratio, not sample size!cited by 4

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 2; 2025: 3; 2026: 7.

12 papers.

  1. 2025High-Dimensional Learning Dynamics of Quantized Models with Straight-Through Estimatorcited by 4
  2. 2024Exploring the Precise Dynamics of Single-Layer GAN Models: Leveraging Multi-Feature Discriminators for High-Dimensional Subspace Learningcited by 2
  3. 2025Generalization in VAE and Diffusion Models: A Unified Information-Theoretic Analysiscited by 2
  4. 2024Statistical Mechanics of Min-Max Problemscited by 1
  5. 2025AI-Driven Multi-Stage Computer Vision System for Defect Detection in Laser-Engraved Industrial Nameplatescited by 1
  6. 2026Latent Spectroscopy: Posterior Collapse as a Featurecited by 1
  7. 2026Mutual Information Collapse Explains Disentanglement Failure in β-VAEscited by 0
  8. 2026RST: certifying and constructing prescribed information in variational autoencoderscited by 0
  9. 2026Multi-Teacher Contrastive Distillation for Edge-Efficient Pathology Foundation Modelscited by 0
  10. 2026Orthogonal Dendritic Intrinsic Networks: An Architecture for Significance-Ordered, Orthogonal Latent Spacescited by 0
  11. 2026CLDRoute: Conditional Latent Diffusion for Routability Map Generation in Physical Designcited by 0
  12. 2026Earth observation embeddings are effective sub-grid descriptors for probabilistic weather downscalingcited by 0