# DialogVED: A Pre-trained Latent Variable Encoder-Decoder Model for Dialog Response Generation

arXiv: https://arxiv.org/abs/2204.13031 · HTML: https://mlsanity.com/paper/2204.13031/

## Key facts

- Cited by 35 papers in the map (data through 14 Oct 2026).
- Builds on 34 of its 54 references that are in the map.
- First posted on arXiv 27 Apr 2022.

## Builds on (most cited first, 30 of 34)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer](https://mlsanity.com/paper/1910.10683/) (2019)
- [BART: Denoising Sequence-to-Sequence Pre-training for Natural Language Generation, Translation, and Comprehension](https://mlsanity.com/paper/1910.13461/) (2019)
- [Neural Discrete Representation Learning](https://mlsanity.com/paper/1711.00937/) (2017)
- [fairseq: A Fast, Extensible Toolkit for Sequence Modeling](https://mlsanity.com/paper/1904.01038/) (2019)
- [Self-Attention with Relative Position Representations](https://mlsanity.com/paper/1803.02155/) (2018)
- [A Diversity-Promoting Objective Function for Neural Conversation Models](https://mlsanity.com/paper/1510.03055/) (2015)
- [Effective Approaches to Attention-based Neural Machine Translation](https://mlsanity.com/paper/1508.04025/) (2015)
- [DialoGPT: Large-Scale Generative Pre-training for Conversational Response Generation](https://mlsanity.com/paper/1911.00536/) (2019)
- [SpanBERT: Improving Pre-training by Representing and Predicting Spans](https://mlsanity.com/paper/1907.10529/) (2019)
- [Personalizing Dialogue Agents: I have a dog, do you have pets too?](https://mlsanity.com/paper/1801.07243/) (2018)
- [DailyDialog: A Manually Labelled Multi-turn Dialogue Dataset](https://mlsanity.com/paper/1710.03957/) (2017)
- [Improving Variational Inference with Inverse Autoregressive Flow](https://mlsanity.com/paper/1606.04934/) (2016)
- [Generating Sentences from a Continuous Space](https://mlsanity.com/paper/1511.06349/) (2015)
- [Learning Discourse-level Diversity for Neural Dialog Models using Conditional Variational Autoencoders](https://mlsanity.com/paper/1703.10960/) (2017)
- [ProphetNet: Predicting Future N-gram for Sequence-to-Sequence Pre-training](https://mlsanity.com/paper/2001.04063/) (2020)
- [A Neural Conversational Model](https://mlsanity.com/paper/1506.05869/) (2015)
- [A Hierarchical Latent Variable Encoder-Decoder Model for Generating Dialogues](https://mlsanity.com/paper/1605.06069/) (2016)
- [PLATO: Pre-trained Dialogue Generation Model with Discrete Latent Variable](https://mlsanity.com/paper/1910.07931/) (2019)
- [Exploring Versatile Generative Language Model Via Parameter-Efficient Transfer Learning](https://mlsanity.com/paper/2004.03829/) (2020)
- [Audio-Visual Scene-Aware Dialog](https://mlsanity.com/paper/1901.09107/) (2019)
- [Optimus: Organizing Sentences via Pre-trained Modeling of a Latent Space](https://mlsanity.com/paper/2004.04092/) (2020)
- [A Simple, Fast Diverse Decoding Algorithm for Neural Generation](https://mlsanity.com/paper/1611.08562/) (2016)
- [ERNIE-GEN: An Enhanced Multi-Flow Pre-training and Fine-tuning Framework for Natural Language Generation](https://mlsanity.com/paper/2001.11314/) (2020)
- [Unsupervised Discrete Sentence Representation Learning for Interpretable Neural Dialog Generation](https://mlsanity.com/paper/1804.08069/) (2018)
- [DVAE++: Discrete Variational Autoencoders with Overlapping Transformations](https://mlsanity.com/paper/1802.04920/) (2018)
- [Variational Attention for Sequence-to-Sequence Models](https://mlsanity.com/paper/1712.08207/) (2017)
- [Implicit Deep Latent Variable Models for Text Generation](https://mlsanity.com/paper/1908.11527/) (2019)

## Built on by (most cited first, 30 of 35)

- [Lift Yourself Up: Retrieval-augmented Text Generation with Self Memory](https://mlsanity.com/paper/2305.02437/) (2023)
- [DISC-FinLLM: A Chinese Financial Large Language Model based on Multiple Experts Fine-tuning](https://mlsanity.com/paper/2310.15205/) (2023)
- [Recent Trends in Personalized Dialogue Generation: A Review of Datasets, Methodologies, and Evaluations](https://mlsanity.com/paper/2405.17974/) (2024)
- [Learning to Memorize Entailment and Discourse Relations for Persona-Consistent Dialogues](https://mlsanity.com/paper/2301.04871/) (2023)
- [MVP: Multi-task Supervised Pre-training for Natural Language Generation](https://mlsanity.com/paper/2206.12131/) (2022)
- [An Equal-Size Hard EM Algorithm for Diverse Dialogue Generation](https://mlsanity.com/paper/2209.14627/) (2022)
- [Response-act Guided Reinforced Dialogue Generation for Mental Health Counseling](https://mlsanity.com/paper/2301.12729/) (2023)
- [EM Pre-training for Multi-party Dialogue Response Generation](https://mlsanity.com/paper/2305.12412/) (2023)
- [Multi-Stage Pre-training Enhanced by ChatGPT for Multi-Scenario Multi-Domain Dialogue Summarization](https://mlsanity.com/paper/2310.10285/) (2023)
- [PVGRU: Generating Diverse and Relevant Dialogue Responses via Pseudo-Variational Mechanism](https://mlsanity.com/paper/2212.09086/) (2022)
- [UniMC: A Unified Framework for Long-Term Memory Conversation via Relevance Representation Learning](https://mlsanity.com/paper/2306.10543/) (2023)
- [DiactTOD: Learning Generalizable Latent Dialogue Acts for Controllable Task-Oriented Dialogue Systems](https://mlsanity.com/paper/2308.00878/) (2023)
- [DiffusionDialog: A Diffusion Model for Diverse Dialog Generation with Latent Space](https://mlsanity.com/paper/2404.06760/) (2024)
- [PoKE: Prior Knowledge Enhanced Emotional Support Conversation with Latent Variable](https://mlsanity.com/paper/2210.12640/) (2022)
- [Towards Diverse, Relevant and Coherent Open-Domain Dialogue Generation via Hybrid Latent Variables](https://mlsanity.com/paper/2212.01145/) (2022)
- [KNSE: A Knowledge-aware Natural Language Inference Framework for Dialogue Symptom Status Recognition](https://mlsanity.com/paper/2305.16833/) (2023)
- [Dynamic Demonstration Retrieval and Cognitive Understanding for Emotional Support Conversation](https://mlsanity.com/paper/2404.02505/) (2024)
- [Joint Generator-Ranker Learning for Natural Language Generation](https://mlsanity.com/paper/2206.13974/) (2022)
- [Pre-training Multi-party Dialogue Models with Latent Discourse Inference](https://mlsanity.com/paper/2305.15175/) (2023)
- [CauESC: A Causal Aware Model for Emotional Support Conversation](https://mlsanity.com/paper/2401.17755/) (2024)
- [Disentangling Preference Representation and Text Generation for Efficient Individual Preference Alignment](https://mlsanity.com/paper/2412.20834/) (2024)
- [Mitigating Negative Style Transfer in Hybrid Dialogue System](https://mlsanity.com/paper/2212.07183/) (2022)
- [Span-based Named Entity Recognition by Generating and Compressing Information](https://mlsanity.com/paper/2302.05392/) (2023)
- [Context-dependent Instruction Tuning for Dialogue Response Generation](https://mlsanity.com/paper/2311.07006/) (2023)
- [K-ESConv: Knowledge Injection for Emotional Support Dialogue Systems via Prompt Learning](https://mlsanity.com/paper/2312.10371/) (2023)
- [DialoGen: Generalized Long-Range Context Representation for Dialogue Systems](https://mlsanity.com/paper/2210.06282/) (2022)
- [Multi-Step Dialogue Workflow Action Prediction](https://mlsanity.com/paper/2311.09593/) (2023)
- [Token Trails: Navigating Contextual Depths in Conversational AI with ChatLLM](https://mlsanity.com/paper/2404.02402/) (2024)
- [ChatZero:Zero-shot Cross-Lingual Dialogue Generation via Pseudo-Target Language](https://mlsanity.com/paper/2408.08724/) (2024)
- [BoK: Introducing Bag-of-Keywords Loss for Interpretable Dialogue Response Generation](https://mlsanity.com/paper/2501.10328/) (2025)

## Abstract (arXiv)

Dialog response generation in open domain is an important research topic where the main challenge is to generate relevant and diverse responses. In this paper, we propose a new dialog pre-training framework called DialogVED, which introduces continuous latent variables into the enhanced encoder-decoder pre-training framework to increase the relevance and diversity of responses. With the help of a large dialog corpus (Reddit), we pre-train the model using the following 4 tasks adopted in language models (LMs) and variational autoencoders (VAEs): 1) masked language model; 2) response generation; 3) bag-of-words prediction; and 4) KL divergence reduction. We also add additional parameters to model the turn structure in dialogs to improve the performance of the pre-trained model. We conduct experiments on PersonaChat, DailyDialog, and DSTC7-AVSD benchmarks for response generation. Experimental results show that our model achieves the new state-of-the-art results on all these datasets.