# Avoiding Latent Variable Collapse With Generative Skip Models

arXiv: https://arxiv.org/abs/1807.04863 · HTML: https://mlsanity.com/paper/1807.04863/

## Key facts

- Cited by 47 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 12 Jul 2018.

## Built on by (most cited first, 30 of 47)

- [Hierarchical VAEs Know What They Don't Know](https://mlsanity.com/paper/2102.08248/) (2021)
- [Posterior Collapse and Latent Variable Non-identifiability](https://mlsanity.com/paper/2301.00537/) (2023)
- [Consistency Regularization for Variational Auto-Encoders](https://mlsanity.com/paper/2105.14859/) (2021)
- [MuseMorphose: Full-Song and Fine-Grained Piano Music Style Transfer with One Transformer VAE](https://mlsanity.com/paper/2105.04090/) (2021)
- [Latent Feature-Guided Diffusion Models for Shadow Removal](https://mlsanity.com/paper/2312.02156/) (2023)
- [AI-Generated Content (AIGC) for Various Data Modalities: A Survey](https://mlsanity.com/paper/2308.14177/) (2023)
- [SKID RAW: Skill Discovery from Raw Trajectories](https://mlsanity.com/paper/2103.14610/) (2021)
- [Fuse It More Deeply! A Variational Transformer with Layer-Wise Latent Variable Inference for Text Generation](https://mlsanity.com/paper/2207.06130/) (2022)
- [Geometry-Complete Perceptron Networks for 3D Molecular Graphs](https://mlsanity.com/paper/2211.02504/) (2022)
- [Variational Autoencoding Discrete Diffusion with Enhanced Dimensional Correlations Modeling](https://mlsanity.com/paper/2505.17384/) (2025)
- [Diverse Text Generation via Variational Encoder-Decoder Models with Gaussian Process Priors](https://mlsanity.com/paper/2204.01227/) (2022)
- [Learning Dynamics in Linear VAE: Posterior Collapse Threshold, Superfluous Latent Space Pitfalls, and Speedup with KL Annealing](https://mlsanity.com/paper/2310.15440/) (2023)
- [Multi-modal Latent Diffusion](https://mlsanity.com/paper/2306.04445/) (2023)
- [Transport Score Climbing: Variational Inference Using Forward KL and Adaptive Neural Transport](https://mlsanity.com/paper/2202.01841/) (2022)
- [Hierarchical VAE with a Diffusion-based VampPrior](https://mlsanity.com/paper/2412.01373/) (2024)
- [Controlling Posterior Collapse by an Inverse Lipschitz Constraint on the Decoder Network](https://mlsanity.com/paper/2304.12770/) (2023)
- [Beyond Vanilla Variational Autoencoders: Detecting Posterior Collapse in Conditional and Hierarchical Variational Autoencoders](https://mlsanity.com/paper/2306.05023/) (2023)
- [Forget-me-not! Contrastive Critics for Mitigating Posterior Collapse](https://mlsanity.com/paper/2207.09535/) (2022)
- [Cooperation in the Latent Space: The Benefits of Adding Mixture Components in Variational Autoencoders](https://mlsanity.com/paper/2209.15514/) (2022)
- [Counterfactual Explanations via Latent Space Projection and Interpolation](https://mlsanity.com/paper/2112.00890/) (2021)
- [Conformalized Generative Bayesian Imaging: An Uncertainty Quantification Framework for Computational Imaging](https://mlsanity.com/paper/2504.07696/) (2025)
- [Discriminative Multimodal Learning via Conditional Priors in Generative Models](https://mlsanity.com/paper/2110.04616/) (2021)
- [CR-VAE: Contrastive Regularization on Variational Autoencoders for Preventing Posterior Collapse](https://mlsanity.com/paper/2309.02968/) (2023)
- [A Non-negative VAE:the Generalized Gamma Belief Network](https://mlsanity.com/paper/2408.03388/) (2024)
- [PerTok: Expressive Encoding and Modeling of Symbolic Musical Ideas and Variations](https://mlsanity.com/paper/2410.02060/) (2024)
- [Covariate-informed Representation Learning to Prevent Posterior Collapse of iVAE](https://mlsanity.com/paper/2202.04206/) (2022)
- [Learning to Drop Out: An Adversarial Approach to Training Sequence VAEs](https://mlsanity.com/paper/2209.12590/) (2022)
- [Improving Variational Autoencoders with Density Gap-based Regularization](https://mlsanity.com/paper/2211.00321/) (2022)
- [Enhancing audio quality for expressive Neural Text-to-Speech](https://mlsanity.com/paper/2108.06270/) (2021)
- [Preventing posterior collapse in variational autoencoders for text generation via decoder regularization](https://mlsanity.com/paper/2110.14945/) (2021)

## Abstract (arXiv)

Variational autoencoders learn distributions of high-dimensional data. They model data with a deep latent-variable model and then fit the model by maximizing a lower bound of the log marginal likelihood. VAEs can capture complex distributions, but they can also suffer from an issue known as "latent variable collapse," especially if the likelihood model is powerful. Specifically, the lower bound involves an approximate posterior of the latent variables; this posterior "collapses" when it is set equal to the prior, i.e., when the approximate posterior is independent of the data. While VAEs learn good generative models, latent variable collapse prevents them from learning useful representations. In this paper, we propose a simple new way to avoid latent variable collapse by including skip connections in our generative model; these connections enforce strong links between the latent variables and the likelihood function. We study generative skip models both theoretically and empirically. Theoretically, we prove that skip models increase the mutual information between the observations and the inferred latent variables. Empirically, we study images (MNIST and Omniglot) and text (Yahoo). Compared to existing VAE architectures, we show that generative skip models maintain similar predictive performance but lead to less collapse and provide more meaningful representations of the data.