# On Posterior Collapse and Encoder Feature Dispersion in Sequence VAEs

arXiv: https://arxiv.org/abs/1911.03976 · HTML: https://mlsanity.com/paper/1911.03976/

## Key facts

- Cited by 5 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 10 Nov 2019.

## Built on by (most cited first, 5 of 5)

- [Finetuning Pretrained Transformers into Variational Autoencoders](https://mlsanity.com/paper/2108.02446/) (2021)
- [Autoencoders as Tools for Program Synthesis](https://mlsanity.com/paper/2108.07129/) (2021)
- [AS-IntroVAE: Adversarial Similarity Distance Makes Robust IntroVAE](https://mlsanity.com/paper/2206.13903/) (2022)
- [Transformers as Neural Augmentors: Class Conditional Sentence Generation via Variational Bayes](https://mlsanity.com/paper/2205.09391/) (2022)
- [On the Mechanisms of Collaborative Learning in VAE Recommenders](https://mlsanity.com/paper/2511.06781/) (2025)

## Abstract (arXiv)

Variational autoencoders (VAEs) hold great potential for modelling text, as they could in theory separate high-level semantic and syntactic properties from local regularities of natural language. Practically, however, VAEs with autoregressive decoders often suffer from posterior collapse, a phenomenon where the model learns to ignore the latent variables, causing the sequence VAE to degenerate into a language model. In this paper, we argue that posterior collapse is in part caused by the lack of dispersion in encoder features. We provide empirical evidence to verify this hypothesis, and propose a straightforward fix using pooling. This simple technique effectively prevents posterior collapse, allowing model to achieve significantly better data log-likelihood than standard sequence VAEs. Comparing to existing work, our proposed method is able to achieve comparable or superior performances while being more computationally efficient.