# Preventing Posterior Collapse with delta-VAEs

arXiv: https://arxiv.org/abs/1901.03416 · HTML: https://mlsanity.com/paper/1901.03416/

## Key facts

- Cited by 86 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 10 Jan 2019.

## Built on by (most cited first, 30 of 86)

- [Maximum Likelihood Training of Score-Based Diffusion Models](https://mlsanity.com/paper/2101.09258/) (2021)
- [Score-based Generative Modeling in Latent Space](https://mlsanity.com/paper/2106.05931/) (2021)
- [Step-unrolled Denoising Autoencoders for Text Generation](https://mlsanity.com/paper/2112.06749/) (2021)
- [Posterior Collapse and Latent Variable Non-identifiability](https://mlsanity.com/paper/2301.00537/) (2023)
- [FLAG: Flow-based 3D Avatar Generation from Sparse Observations](https://mlsanity.com/paper/2203.05789/) (2022)
- [Densely connected normalizing flows](https://mlsanity.com/paper/2106.04627/) (2021)
- [Maximum Likelihood Training of Implicit Nonlinear Diffusion Models](https://mlsanity.com/paper/2205.13699/) (2022)
- [Dataset Condensation with Latent Space Knowledge Factorization and Sharing](https://mlsanity.com/paper/2208.10494/) (2022)
- [A Critical Look at the Consistency of Causal Estimation With Deep Latent Variable Models](https://mlsanity.com/paper/2102.06648/) (2021)
- [Data-Driven Offline Optimization For Architecting Hardware Accelerators](https://mlsanity.com/paper/2110.11346/) (2021)
- [Efficient-VDVAE: Less is more](https://mlsanity.com/paper/2203.13751/) (2022)
- [AI-Generated Content (AIGC) for Various Data Modalities: A Survey](https://mlsanity.com/paper/2308.14177/) (2023)
- [Variable-rate discrete representation learning](https://mlsanity.com/paper/2103.06089/) (2021)
- [The Free Energy Principle for Perception and Action: A Deep Learning Perspective](https://mlsanity.com/paper/2207.06415/) (2022)
- [Sawtooth Factorial Topic Embeddings Guided Gamma Belief Network](https://mlsanity.com/paper/2107.02757/) (2021)
- [A survey on Variational Autoencoders from a GreenAI perspective](https://mlsanity.com/paper/2103.01071/) (2021)
- [Fuse It More Deeply! A Variational Transformer with Layer-Wise Latent Variable Inference for Text Generation](https://mlsanity.com/paper/2207.06130/) (2022)
- [Unified Latents (UL): How to train your latents](https://mlsanity.com/paper/2602.17270/) (2026)
- [Missing Data Imputation and Acquisition with Deep Hierarchical Models and Hamiltonian Monte Carlo](https://mlsanity.com/paper/2202.04599/) (2022)
- [Normalizing Flow-based Neural Process for Few-Shot Knowledge Graph Completion](https://mlsanity.com/paper/2304.08183/) (2023)
- [Complexity Matters: Rethinking the Latent Space for Generative Modeling](https://mlsanity.com/paper/2307.08283/) (2023)
- [Posterior Collapse of a Linear Latent Variable Model](https://mlsanity.com/paper/2205.04009/) (2022)
- [Multi-Rate VAE: Train Once, Get the Full Rate-Distortion Curve](https://mlsanity.com/paper/2212.03905/) (2022)
- [Amortized Variational Inference: A Systematic Review](https://mlsanity.com/paper/2209.10888/) (2022)
- [Preventing Oversmoothing in VAE via Generalized Variance Parameterization](https://mlsanity.com/paper/2102.08663/) (2021)
- [Hierarchical Latent Structure for Multi-Modal Vehicle Trajectory Forecasting](https://mlsanity.com/paper/2207.04624/) (2022)
- [Modeling Multimodal Aleatoric Uncertainty in Segmentation with Mixture of Stochastic Experts](https://mlsanity.com/paper/2212.07328/) (2022)
- [Learning Energy-Based Prior Model with Diffusion-Amortized MCMC](https://mlsanity.com/paper/2310.03218/) (2023)
- [A Gray-box Attack against Latent Diffusion Model-based Image Editing by Posterior Collapse](https://mlsanity.com/paper/2408.10901/) (2024)
- [Comparing the latent space of generative models](https://mlsanity.com/paper/2207.06812/) (2022)

## Abstract (arXiv)

Due to the phenomenon of "posterior collapse," current latent variable generative models pose a challenging design choice that either weakens the capacity of the decoder or requires augmenting the objective so it does not only maximize the likelihood of the data. In this paper, we propose an alternative that utilizes the most powerful generative models as decoders, whilst optimising the variational lower bound all while ensuring that the latent variables preserve and encode useful information. Our proposed δ-VAEs achieve this by constraining the variational family for the posterior to have a minimum distance to the prior. For sequential latent variable models, our approach resembles the classic representation learning approach of slow feature analysis. We demonstrate the efficacy of our approach at modeling text on LM1B and modeling images: learning representations, improving sample quality, and achieving state of the art log-likelihood on CIFAR-10 and ImageNet 32× 32.