# Improving Variational Autoencoders with Density Gap-based Regularization

arXiv: https://arxiv.org/abs/2211.00321 · HTML: https://mlsanity.com/paper/2211.00321/

## Key facts

- Cited by 3 papers in the map (data through 14 Oct 2026).
- Builds on 35 of its 57 references that are in the map.
- First posted on arXiv 1 Nov 2022.

## Builds on (most cited first, 30 of 35)

- [Character-level Convolutional Networks for Text Classification](https://mlsanity.com/paper/1509.01626/) (2015)
- [A large annotated corpus for learning natural language inference](https://mlsanity.com/paper/1508.05326/) (2015)
- [Unsupervised Feature Learning via Non-Parametric Instance-level Discrimination](https://mlsanity.com/paper/1805.01978/) (2018)
- [Generating Sentences from a Continuous Space](https://mlsanity.com/paper/1511.06349/) (2015)
- [Importance Weighted Autoencoders](https://mlsanity.com/paper/1509.00519/) (2015)
- [Wasserstein Auto-Encoders](https://mlsanity.com/paper/1711.01558/) (2017)
- [Adversarial Autoencoders](https://mlsanity.com/paper/1511.05644/) (2015)
- [Deep Generative Modelling: A Comparative Review of VAEs, GANs, Normalizing Flows, Energy-Based and Autoregressive Models](https://mlsanity.com/paper/2103.04922/) (2021)
- [VAE with a VampPrior](https://mlsanity.com/paper/1705.07120/) (2017)
- [Style Transfer from Non-Parallel Text by Cross-Alignment](https://mlsanity.com/paper/1705.09655/) (2017)
- [Learning Discourse-level Diversity for Neural Dialog Models using Conditional Variational Autoencoders](https://mlsanity.com/paper/1703.10960/) (2017)
- [Cyclical Annealing Schedule: A Simple Approach to Mitigating KL Vanishing](https://mlsanity.com/paper/1903.10145/) (2019)
- [Preventing Posterior Collapse with delta-VAEs](https://mlsanity.com/paper/1901.03416/) (2019)
- [Taming VAEs](https://mlsanity.com/paper/1810.00597/) (2018)
- [Improved Variational Autoencoders for Text Modeling using Dilated Convolutions](https://mlsanity.com/paper/1702.08139/) (2017)
- [PixelVAE: A Latent Variable Model for Natural Images](https://mlsanity.com/paper/1611.05013/) (2016)
- [Spherical Latent Spaces for Stable Variational Autoencoders](https://mlsanity.com/paper/1808.10805/) (2018)
- [A Contrastive Learning Approach for Training Variational Autoencoder Priors](https://mlsanity.com/paper/2010.02917/) (2020)
- [Resampled Priors for Variational Autoencoders](https://mlsanity.com/paper/1810.11428/) (2018)
- [Learning Hierarchical Priors in VAEs](https://mlsanity.com/paper/1905.04982/) (2019)
- [Avoiding Latent Variable Collapse With Generative Skip Models](https://mlsanity.com/paper/1807.04863/) (2018)
- [A Surprisingly Effective Fix for Deep Latent Variable Modeling of Text](https://mlsanity.com/paper/1909.00868/) (2019)
- [Implicit Deep Latent Variable Models for Text Generation](https://mlsanity.com/paper/1908.11527/) (2019)
- [Neural Machine Translation with Phrase-Level Universal Visual Representations](https://mlsanity.com/paper/2203.10299/) (2022)
- [Data Augmentation for Spoken Language Understanding via Joint Variational Generation](https://mlsanity.com/paper/1809.02305/) (2018)
- [A Batch Normalized Inference Network Keeps the KL Vanishing Away](https://mlsanity.com/paper/2004.12585/) (2020)
- [There Are a Thousand Hamlets in a Thousand People's Eyes: Enhancing Knowledge-grounded Dialogue with Personal Memory](https://mlsanity.com/paper/2204.02624/) (2022)
- [Preventing Posterior Collapse with Levenshtein Variational Autoencoder](https://mlsanity.com/paper/2004.14758/) (2020)
- [Pseudo Siamese Network for Few-shot Intent Generation](https://mlsanity.com/paper/2105.00896/) (2021)
- [Improving Variational Autoencoder for Text Modelling with Timestep-Wise Regularisation](https://mlsanity.com/paper/2011.01136/) (2020)

## Built on by (most cited first, 3 of 3)

- [Evaluating Open-Domain Dialogues in Latent Space with Next Sentence Prediction and Mutual Information](https://mlsanity.com/paper/2305.16967/) (2023)
- [Disentangling Preference Representation and Text Generation for Efficient Individual Preference Alignment](https://mlsanity.com/paper/2412.20834/) (2024)
- [Disentangled representations via score-based variational autoencoders](https://mlsanity.com/paper/2512.17127/) (2025)

## Abstract (arXiv)

Variational autoencoders (VAEs) are one of the powerful unsupervised learning frameworks in NLP for latent representation learning and latent-directed generation. The classic optimization goal of VAEs is to maximize the Evidence Lower Bound (ELBo), which consists of a conditional likelihood for generation and a negative Kullback-Leibler (KL) divergence for regularization. In practice, optimizing ELBo often leads the posterior distribution of all samples converge to the same degenerated local optimum, namely posterior collapse or KL vanishing. There are effective ways proposed to prevent posterior collapse in VAEs, but we observe that they in essence make trade-offs between posterior collapse and hole problem, i.e., mismatch between the aggregated posterior distribution and the prior distribution. To this end, we introduce new training objectives to tackle both two problems through a novel regularization based on the probabilistic density gap between the aggregated posterior distribution and the prior distribution. Through experiments on language modeling, latent space visualization and interpolation, we show that our proposed method can solve both problems effectively and thus outperforms the existing methods in latent-directed generation. To the best of our knowledge, we are the first to jointly solve the hole problem and the posterior collapse.