# On the Latent Holes of VAEs for Text Generation

arXiv: https://arxiv.org/abs/2110.03318 · HTML: https://mlsanity.com/paper/2110.03318/

## Key facts

- Cited by 4 papers in the map (data through 14 Oct 2026).
- Builds on 23 of its 51 references that are in the map.
- First posted on arXiv 7 Oct 2021.

## Builds on (most cited first, 23 of 23)

- [Character-level Convolutional Networks for Text Classification](https://mlsanity.com/paper/1509.01626/) (2015)
- [Sinkhorn Distances: Lightspeed Computation of Optimal Transportation Distances](https://mlsanity.com/paper/1306.0895/) (2013)
- [A large annotated corpus for learning natural language inference](https://mlsanity.com/paper/1508.05326/) (2015)
- [Generating Diverse High-Fidelity Images with VQ-VAE-2](https://mlsanity.com/paper/1906.00446/) (2019)
- [Plug and Play Language Models: A Simple Approach to Controlled Text Generation](https://mlsanity.com/paper/1912.02164/) (2019)
- [Generating Sentences from a Continuous Space](https://mlsanity.com/paper/1511.06349/) (2015)
- [Symmetric Cross Entropy for Robust Learning with Noisy Labels](https://mlsanity.com/paper/1908.06112/) (2019)
- [Wasserstein Auto-Encoders](https://mlsanity.com/paper/1711.01558/) (2017)
- [Cyclical Annealing Schedule: A Simple Approach to Mitigating KL Vanishing](https://mlsanity.com/paper/1903.10145/) (2019)
- [Hyperspherical Variational Auto-Encoders](https://mlsanity.com/paper/1804.00891/) (2018)
- [DARLA: Improving Zero-Shot Transfer in Reinforcement Learning](https://mlsanity.com/paper/1707.08475/) (2017)
- [A Hierarchical Latent Vector Model for Learning Long-Term Structure in Music](https://mlsanity.com/paper/1803.05428/) (2018)
- [Lagging Inference Networks and Posterior Collapse in Variational Autoencoders](https://mlsanity.com/paper/1901.05534/) (2019)
- [IntroVAE: Introspective Variational Autoencoders for Photographic Image Synthesis](https://mlsanity.com/paper/1807.06358/) (2018)
- [Improved Variational Autoencoders for Text Modeling using Dilated Convolutions](https://mlsanity.com/paper/1702.08139/) (2017)
- [Explorations in Homeomorphic Variational Auto-Encoding](https://mlsanity.com/paper/1807.04689/) (2018)
- [Sliced Wasserstein Generative Models](https://mlsanity.com/paper/1904.05408/) (2019)
- [Variational Autoencoders with Riemannian Brownian Motion Priors](https://mlsanity.com/paper/2002.05227/) (2020)
- [A Surprisingly Effective Fix for Deep Latent Variable Modeling of Text](https://mlsanity.com/paper/1909.00868/) (2019)
- [Implicit Deep Latent Variable Models for Text Generation](https://mlsanity.com/paper/1908.11527/) (2019)
- [Fast and Smooth Interpolation on Wasserstein Space](https://mlsanity.com/paper/2010.12101/) (2020)
- [A Batch Normalized Inference Network Keeps the KL Vanishing Away](https://mlsanity.com/paper/2004.12585/) (2020)
- [Topological Constraints on Homeomorphic Auto-Encoding](https://mlsanity.com/paper/1812.10783/) (2018)

## Built on by (most cited first, 4 of 4)

- [Improving Variational Autoencoders with Density Gap-based Regularization](https://mlsanity.com/paper/2211.00321/) (2022)
- [Learning from Multiple Sources for Data-to-Text and Text-to-Data](https://mlsanity.com/paper/2302.11269/) (2023)
- [Vacant Holes for Unsupervised Detection of the Outliers in Compact Latent Representation](https://mlsanity.com/paper/2306.09646/) (2023)
- [Disentangled representations via score-based variational autoencoders](https://mlsanity.com/paper/2512.17127/) (2025)

## Abstract (arXiv)

In this paper, we provide the first focused study on the discontinuities (aka. holes) in the latent space of Variational Auto-Encoders (VAEs), a phenomenon which has been shown to have a detrimental effect on model capacity. When investigating latent holes, existing works are exclusively centred around the encoder network and they merely explore the existence of holes. We tackle these limitations by proposing a highly efficient Tree-based Decoder-Centric (TDC) algorithm for latent hole identification, with a focal point on the text domain. In contrast to past studies, our approach pays attention to the decoder network, as a decoder has a direct impact on the model's output quality. Furthermore, we provide, for the first time, in-depth empirical analysis of the latent hole phenomenon, investigating several important aspects such as how the holes impact VAE algorithms' performance on text generation, and how the holes are distributed in the latent space.