mlsanityData to 7 Oct 2026

On the Latent Holes of VAEs for Text Generation

arXiv 2110.03318 ↗First posted 7 Oct 2021cs.LG cs.AI cs.CLcited by 4

Not on the map. Why?

Abstract (arXiv)

In this paper, we provide the first focused study on the discontinuities (aka. holes) in the latent space of Variational Auto-Encoders (VAEs), a phenomenon which has been shown to have a detrimental effect on model capacity. When investigating latent holes, existing works are exclusively centred around the encoder network and they merely explore the existence of holes. We tackle these limitations by proposing a highly efficient Tree-based Decoder-Centric (TDC) algorithm for latent hole identification, with a focal point on the text domain. In contrast to past studies, our approach pays attention to the decoder network, as a decoder has a direct impact on the model's output quality. Furthermore, we provide, for the first time, in-depth empirical analysis of the latent hole phenomenon, investigating several important aspects such as how the holes impact VAE algorithms' performance on text generation, and how the holes are distributed in the latent space.

Key facts. Cited by 4 papers in the map (data through 7 Oct 2026). Builds on 23 of its 51 references that are in the map. First posted on arXiv 7 Oct 2021.

Builds on

23 of 51 references.

References · 23

  1. 2015Character-level Convolutional Networks for Text Classificationcited by 2,293
  2. 2013Sinkhorn Distances: Lightspeed Computation of Optimal Transportation Distancescited by 2,277
  3. 2015A large annotated corpus for learning natural language inferencecited by 1,837
  4. 2019Generating Diverse High-Fidelity Images with VQ-VAE-2cited by 1,376
  5. 2019Plug and Play Language Models: A Simple Approach to Controlled Text Generationcited by 793
  6. 2015Generating Sentences from a Continuous Spacecited by 607
  7. 2019Symmetric Cross Entropy for Robust Learning with Noisy Labelscited by 499
  8. 2017Wasserstein Auto-Encoderscited by 371
  9. 2019Cyclical Annealing Schedule: A Simple Approach to Mitigating KL Vanishingcited by 191
  10. 2018Hyperspherical Variational Auto-Encoderscited by 188
  11. 2017DARLA: Improving Zero-Shot Transfer in Reinforcement Learningcited by 147
  12. 2018A Hierarchical Latent Vector Model for Learning Long-Term Structure in Musiccited by 131
11 more
  1. 2019Lagging Inference Networks and Posterior Collapse in Variational Autoencoderscited by 124
  2. 2018IntroVAE: Introspective Variational Autoencoders for Photographic Image Synthesiscited by 85
  3. 2017Improved Variational Autoencoders for Text Modeling using Dilated Convolutionscited by 81
  4. 2018Explorations in Homeomorphic Variational Auto-Encodingcited by 65
  5. 2019Sliced Wasserstein Generative Modelscited by 61
  6. 2020Variational Autoencoders with Riemannian Brownian Motion Priorscited by 34
  7. 2019A Surprisingly Effective Fix for Deep Latent Variable Modeling of Textcited by 32
  8. 2019Implicit Deep Latent Variable Models for Text Generationcited by 26
  9. 2020Fast and Smooth Interpolation on Wasserstein Spacecited by 21
  10. 2020A Batch Normalized Inference Network Keeps the KL Vanishing Awaycited by 20
  11. 2018Topological Constraints on Homeomorphic Auto-Encodingcited by 2

Built on by

Cited by year

Citing papers per year: 2022: 1; 2023: 2; 2024: 0; 2025: 1.

4 papers.

  1. 2022Improving Variational Autoencoders with Density Gap-based Regularizationcited by 3
  2. 2023Learning from Multiple Sources for Data-to-Text and Text-to-Datacited by 1
  3. 2023Vacant Holes for Unsupervised Detection of the Outliers in Compact Latent Representationcited by 1
  4. 2025Disentangled representations via score-based variational autoencoderscited by 0