# Diverse Text Generation via Variational Encoder-Decoder Models with Gaussian Process Priors

arXiv: https://arxiv.org/abs/2204.01227 · HTML: https://mlsanity.com/paper/2204.01227/

## Key facts

- Cited by 13 papers in the map (data through 14 Oct 2026).
- Builds on 34 of its 48 references that are in the map.
- First posted on arXiv 4 Apr 2022.

## Builds on (most cited first, 30 of 34)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer](https://mlsanity.com/paper/1910.10683/) (2019)
- [Neural Machine Translation by Jointly Learning to Align and Translate](https://mlsanity.com/paper/1409.0473/) (2014)
- [Learning Phrase Representations using RNN Encoder-Decoder for Statistical Machine Translation](https://mlsanity.com/paper/1406.1078/) (2014)
- [Auto-Encoding Variational Bayes](https://mlsanity.com/paper/1312.6114/) (2013)
- [Sequence to Sequence Learning with Neural Networks](https://mlsanity.com/paper/1409.3215/) (2014)
- [The Curious Case of Neural Text Degeneration](https://mlsanity.com/paper/1904.09751/) (2019)
- [Get To The Point: Summarization with Pointer-Generator Networks](https://mlsanity.com/paper/1704.04368/) (2017)
- [Hierarchical Neural Story Generation](https://mlsanity.com/paper/1805.04833/) (2018)
- [Effective Approaches to Attention-based Neural Machine Translation](https://mlsanity.com/paper/1508.04025/) (2015)
- [Generating Sentences from a Continuous Space](https://mlsanity.com/paper/1511.06349/) (2015)
- [Texygen: A Benchmarking Platform for Text Generation Models](https://mlsanity.com/paper/1802.01886/) (2018)
- [Diverse Beam Search: Decoding Diverse Solutions from Neural Sequence Models](https://mlsanity.com/paper/1610.02424/) (2016)
- [Incorporating Copying Mechanism in Sequence-to-Sequence Learning](https://mlsanity.com/paper/1603.06393/) (2016)
- [Dear Sir or Madam, May I introduce the GYAFC Dataset: Corpus, Benchmarks and Metrics for Formality Style Transfer](https://mlsanity.com/paper/1803.06535/) (2018)
- [Building End-To-End Dialogue Systems Using Generative Hierarchical Neural Network Models](https://mlsanity.com/paper/1507.04808/) (2015)
- [Semi-Amortized Variational Autoencoders](https://mlsanity.com/paper/1802.02550/) (2018)
- [Mixture Models for Diverse Machine Translation: Tricks of the Trade](https://mlsanity.com/paper/1902.07816/) (2019)
- [A Continuously Growing Dataset of Sentential Paraphrases](https://mlsanity.com/paper/1708.00391/) (2017)
- [A Hierarchical Recurrent Encoder-Decoder For Generative Context-Aware Query Suggestion](https://mlsanity.com/paper/1507.02221/) (2015)
- [Paraphrase Generation with Deep Reinforcement Learning](https://mlsanity.com/paper/1711.00279/) (2017)
- [Shakespearizing Modern Language Using Copy-Enriched Sequence-to-Sequence Models](https://mlsanity.com/paper/1707.01161/) (2017)
- [Fast, Diverse and Accurate Image Captioning Guided By Part-of-Speech](https://mlsanity.com/paper/1805.12589/) (2018)
- [Neural Paraphrase Generation with Stacked Residual LSTM Networks](https://mlsanity.com/paper/1610.03098/) (2016)
- [Avoiding Latent Variable Collapse With Generative Skip Models](https://mlsanity.com/paper/1807.04863/) (2018)
- [Latent Alignment and Variational Attention](https://mlsanity.com/paper/1807.03756/) (2018)
- [Variational Attention for Sequence-to-Sequence Models](https://mlsanity.com/paper/1712.08207/) (2017)
- [Mixture Content Selection for Diverse Sequence Generation](https://mlsanity.com/paper/1909.01953/) (2019)
- [Pre-train and Plug-in: Flexible Conditional Text Generation with Variational Auto-Encoders](https://mlsanity.com/paper/1911.03882/) (2019)
- [Generating Relevant and Coherent Dialogue Responses using Self-separated Conditional Variational AutoEncoders](https://mlsanity.com/paper/2106.03410/) (2021)

## Built on by (most cited first, 13 of 13)

- [DiffuSeq: Sequence to Sequence Text Generation with Diffusion Models](https://mlsanity.com/paper/2210.08933/) (2022)
- [TESS: Text-to-Text Self-Conditioned Simplex Diffusion](https://mlsanity.com/paper/2305.08379/) (2023)
- [Empowering Diffusion Models on the Embedding Space for Text Generation](https://mlsanity.com/paper/2212.09412/) (2022)
- [A Review of Deep Learning for Video Captioning](https://mlsanity.com/paper/2304.11431/) (2023)
- [Smoothie: Smoothing Diffusion on Token Embeddings for Text Generation](https://mlsanity.com/paper/2505.18853/) (2025)
- [InfoDiffusion: Information Entropy Aware Diffusion Process for Non-Autoregressive Text Generation](https://mlsanity.com/paper/2310.11976/) (2023)
- [Integrating Randomness in Large Language Models: A Linear Congruential Generator Approach for Generating Clinically Relevant Content](https://mlsanity.com/paper/2407.03582/) (2024)
- [Improving Linguistic Diversity of Large Language Models with Possibility Exploration Fine-Tuning](https://mlsanity.com/paper/2412.03343/) (2024)
- [Mitigating Negative Style Transfer in Hybrid Dialogue System](https://mlsanity.com/paper/2212.07183/) (2022)
- [Text Diffusion with Reinforced Conditioning](https://mlsanity.com/paper/2402.14843/) (2024)
- [GlyphDiffusion: Text Generation as Image Generation](https://mlsanity.com/paper/2304.12519/) (2023)
- [DiffuSIA: A Spiral Interaction Architecture for Encoder-Decoder Text Diffusion](https://mlsanity.com/paper/2305.11517/) (2023)
- [Boosting Summarization with Normalizing Flows and Aggressive Training](https://mlsanity.com/paper/2311.00588/) (2023)

## Abstract (arXiv)

Generating high quality texts with high diversity is important for many NLG applications, but current methods mostly focus on building deterministic models to generate higher quality texts and do not provide many options for promoting diversity. In this work, we present a novel latent structured variable model to generate high quality texts by enriching contextual representation learning of encoder-decoder models. Specifically, we introduce a stochastic function to map deterministic encoder hidden states into random context variables. The proposed stochastic function is sampled from a Gaussian process prior to (1) provide infinite number of joint Gaussian distributions of random context variables (diversity-promoting) and (2) explicitly model dependency between context variables (accurate-encoding). To address the learning challenge of Gaussian processes, we propose an efficient variational inference approach to approximate the posterior distribution of random context variables. We evaluate our method in two typical text generation tasks: paraphrase generation and text style transfer. Experimental results on benchmark datasets demonstrate that our method improves the generation quality and diversity compared with other baselines.