# Improved Variational Autoencoders for Text Modeling using Dilated Convolutions

arXiv: https://arxiv.org/abs/1702.08139 · HTML: https://mlsanity.com/paper/1702.08139/

## Key facts

- Cited by 81 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 27 Feb 2017.

## Built on by (most cited first, 30 of 81)

- [GLM: General Language Model Pretraining with Autoregressive Blank Infilling](https://mlsanity.com/paper/2103.10360/) (2021)
- [Versatile Diffusion: Text, Images and Variations All in One Diffusion Model](https://mlsanity.com/paper/2211.08332/) (2022)
- [An Empirical Survey of Data Augmentation for Limited Data Learning in NLP](https://mlsanity.com/paper/2106.07499/) (2021)
- [Posterior Collapse and Latent Variable Non-identifiability](https://mlsanity.com/paper/2301.00537/) (2023)
- [Transformer-based Conditional Variational Autoencoder for Controllable Story Generation](https://mlsanity.com/paper/2101.00828/) (2021)
- [Latent Feature-Guided Diffusion Models for Shadow Removal](https://mlsanity.com/paper/2312.02156/) (2023)
- [Roll the dice & look before you leap: Going beyond the creative limits of next-token prediction](https://mlsanity.com/paper/2504.15266/) (2025)
- [The Rich Get Richer: Disparate Impact of Semi-Supervised Learning](https://mlsanity.com/paper/2110.06282/) (2021)
- [Fuse It More Deeply! A Variational Transformer with Layer-Wise Latent Variable Inference for Text Generation](https://mlsanity.com/paper/2207.06130/) (2022)
- [Efficient Architecture Search for Diverse Tasks](https://mlsanity.com/paper/2204.07554/) (2022)
- [Weakly-Supervised Hierarchical Models for Predicting Persuasive Strategies in Good-faith Textual Requests](https://mlsanity.com/paper/2101.06351/) (2021)
- [Amortized Variational Inference: A Systematic Review](https://mlsanity.com/paper/2209.10888/) (2022)
- [TextBox: A Unified, Modularized, and Extensible Framework for Text Generation](https://mlsanity.com/paper/2101.02046/) (2021)
- [Action2video: Generating Videos of Human 3D Actions](https://mlsanity.com/paper/2111.06925/) (2021)
- [EEG2Vec: Learning Affective EEG Representations via Variational Autoencoders](https://mlsanity.com/paper/2207.08002/) (2022)
- [NWT: Towards natural audio-to-video generation with representation learning](https://mlsanity.com/paper/2106.04283/) (2021)
- [Unsupervised Abstractive Opinion Summarization by Generating Sentences with Tree-Structured Topic Guidance](https://mlsanity.com/paper/2106.08007/) (2021)
- [Finetuning Pretrained Transformers into Variational Autoencoders](https://mlsanity.com/paper/2108.02446/) (2021)
- [Hierarchical Latent Structure for Multi-Modal Vehicle Trajectory Forecasting](https://mlsanity.com/paper/2207.04624/) (2022)
- [Low-Resource Dense Retrieval for Open-Domain Question Answering: A Comprehensive Survey](https://mlsanity.com/paper/2208.03197/) (2022)
- [Variational Autoencoding Discrete Diffusion with Enhanced Dimensional Correlations Modeling](https://mlsanity.com/paper/2505.17384/) (2025)
- [An Introduction to Variational Inference](https://mlsanity.com/paper/2108.13083/) (2021)
- [RegaVAE: A Retrieval-Augmented Gaussian Mixture Variational Auto-Encoder for Language Modeling](https://mlsanity.com/paper/2310.10567/) (2023)
- [Learning Dynamics in Linear VAE: Posterior Collapse Threshold, Superfluous Latent Space Pitfalls, and Speedup with KL Annealing](https://mlsanity.com/paper/2310.15440/) (2023)
- [Variational Selective Autoencoder: Learning from Partially-Observed Heterogeneous Data](https://mlsanity.com/paper/2102.12679/) (2021)
- [ME-GCN: Multi-dimensional Edge-Embedded Graph Convolutional Networks for Semi-supervised Text Classification](https://mlsanity.com/paper/2204.04618/) (2022)
- [Unified Generation, Reconstruction, and Representation: Generalized Diffusion with Adaptive Latent Encoding-Decoding](https://mlsanity.com/paper/2402.19009/) (2024)
- [A Differentiable Language Model Adversarial Attack on Text Classifiers](https://mlsanity.com/paper/2107.11275/) (2021)
- [Embedding-Aligned Language Models](https://mlsanity.com/paper/2406.00024/) (2024)
- [Training Discrete Deep Generative Models via Gapped Straight-Through Estimator](https://mlsanity.com/paper/2206.07235/) (2022)

## Abstract (arXiv)

Recent work on generative modeling of text has found that variational auto-encoders (VAE) incorporating LSTM decoders perform worse than simpler LSTM language models (Bowman et al., 2015). This negative result is so far poorly understood, but has been attributed to the propensity of LSTM decoders to ignore conditioning information from the encoder. In this paper, we experiment with a new type of decoder for VAE: a dilated CNN. By changing the decoder's dilation architecture, we control the effective context from previously generated words. In experiments, we find that there is a trade off between the contextual capacity of the decoder and the amount of encoding information used. We show that with the right decoder, VAE can outperform LSTM language models. We demonstrate perplexity gains on two datasets, representing the first positive experimental result on the use VAE for generative modeling of text. Further, we conduct an in-depth investigation of the use of VAE (with our new decoding architecture) for semi-supervised and unsupervised labeling tasks, demonstrating gains over several strong baselines.