# Dialogue Distillation: Open-Domain Dialogue Augmentation Using Unpaired Data

arXiv: https://arxiv.org/abs/2009.09427 · HTML: https://mlsanity.com/paper/2009.09427/

## Key facts

- Cited by 20 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 20 Sep 2020.

## Built on by (most cited first, 20 of 20)

- [CAMEL: Communicative Agents for "Mind" Exploration of Large Language Model Society](https://mlsanity.com/paper/2303.17760/) (2023)
- [Data Augmentation Approaches in Natural Language Processing: A Survey](https://mlsanity.com/paper/2110.01852/) (2021)
- [PLACES: Prompting Language Models for Social Conversation Synthesis](https://mlsanity.com/paper/2302.03269/) (2023)
- [Duplex Conversation: Towards Human-like Interaction in Spoken Dialogue Systems](https://mlsanity.com/paper/2205.15060/) (2022)
- [Rethinking and Refining the Distinct Metric](https://mlsanity.com/paper/2202.13587/) (2022)
- [Diversifying Dialog Generation via Adaptive Label Smoothing](https://mlsanity.com/paper/2105.14556/) (2021)
- [A Model-Agnostic Data Manipulation Method for Persona-based Dialogue Generation](https://mlsanity.com/paper/2204.09867/) (2022)
- [NormDial: A Comparable Bilingual Synthetic Dialog Dataset for Modeling Social Norm Adherence and Violation](https://mlsanity.com/paper/2310.14563/) (2023)
- [PCC: Paraphrasing with Bottom-k Sampling and Cyclic Learning for Curriculum Data Augmentation](https://mlsanity.com/paper/2208.08110/) (2022)
- [Estimating Soft Labels for Out-of-Domain Intent Detection](https://mlsanity.com/paper/2211.05561/) (2022)
- [Prompt Conditioned VAE: Enhancing Generative Replay for Lifelong Learning in Task-Oriented Dialogue](https://mlsanity.com/paper/2210.07783/) (2022)
- [DialoGPS: Dialogue Path Sampling in Continuous Semantic Space for Data Augmentation in Multi-Turn Conversations](https://mlsanity.com/paper/2306.16770/) (2023)
- [Counterfactual Data Augmentation via Perspective Transition for Open-Domain Dialogues](https://mlsanity.com/paper/2210.16838/) (2022)
- [Data Augmentation for Text Generation Without Any Augmented Data](https://mlsanity.com/paper/2105.13650/) (2021)
- [Distilling the Knowledge of Large-scale Generative Models into Retrieval Models for Efficient Open-domain Conversation](https://mlsanity.com/paper/2108.12582/) (2021)
- [Controllable and Diverse Data Augmentation with Large Language Model for Low-Resource Open-Domain Dialogue Generation](https://mlsanity.com/paper/2404.00361/) (2024)
- [Data Augmentation for Conversational AI](https://mlsanity.com/paper/2309.04739/) (2023)
- [Score Before You Speak: Improving Persona Consistency in Dialogue Generation using Response Quality Scores](https://mlsanity.com/paper/2508.06886/) (2025)
- [Accuracy on In-Domain Samples Matters When Building Out-of-Domain detectors: A Reply to Marek et al. (2021)](https://mlsanity.com/paper/2205.11887/) (2022)
- [Hexa: Self-Improving for Knowledge-Grounded Dialogue System](https://mlsanity.com/paper/2310.06404/) (2023)

## Abstract (arXiv)

Recent advances in open-domain dialogue systems rely on the success of neural models that are trained on large-scale data. However, collecting large-scale dialogue data is usually time-consuming and labor-intensive. To address this data dilemma, we propose a novel data augmentation method for training open-domain dialogue models by utilizing unpaired data. Specifically, a data-level distillation process is first proposed to construct augmented dialogues where both post and response are retrieved from the unpaired data. A ranking module is employed to filter out low-quality dialogues. Further, a model-level distillation process is employed to distill a teacher model trained on high-quality paired data to augmented dialogue pairs, thereby preventing dialogue models from being affected by the noise in the augmented data. Automatic and manual evaluation indicates that our method can produce high-quality dialogue pairs with diverse contents, and the proposed data-level and model-level dialogue distillation can improve the performance of competitive baselines.