# Towards Efficient Dialogue Pre-training with Transferable and Interpretable Latent Structure

arXiv: https://arxiv.org/abs/2210.12461 · HTML: https://mlsanity.com/paper/2210.12461/

## Key facts

- Cited by 4 papers in the map (data through 14 Oct 2026).
- Builds on 31 of its 46 references that are in the map.
- First posted on arXiv 22 Oct 2022.

## Builds on (most cited first, 30 of 31)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [Sequence to Sequence Learning with Neural Networks](https://mlsanity.com/paper/1409.3215/) (2014)
- [Categorical Reparameterization with Gumbel-Softmax](https://mlsanity.com/paper/1611.01144/) (2016)
- [A Diversity-Promoting Objective Function for Neural Conversation Models](https://mlsanity.com/paper/1510.03055/) (2015)
- [DialoGPT: Large-Scale Generative Pre-training for Conversational Response Generation](https://mlsanity.com/paper/1911.00536/) (2019)
- [DailyDialog: A Manually Labelled Multi-turn Dialogue Dataset](https://mlsanity.com/paper/1710.03957/) (2017)
- [Convolutional Sequence to Sequence Learning](https://mlsanity.com/paper/1705.03122/) (2017)
- [Wizard of Wikipedia: Knowledge-Powered Conversational agents](https://mlsanity.com/paper/1811.01241/) (2018)
- [Towards a Human-like Open-Domain Chatbot](https://mlsanity.com/paper/2001.09977/) (2020)
- [Learning Discourse-level Diversity for Neural Dialog Models using Conditional Variational Autoencoders](https://mlsanity.com/paper/1703.10960/) (2017)
- [Building End-To-End Dialogue Systems Using Generative Hierarchical Neural Network Models](https://mlsanity.com/paper/1507.04808/) (2015)
- [A Neural Conversational Model](https://mlsanity.com/paper/1506.05869/) (2015)
- [The Second Conversational Intelligence Challenge (ConvAI2)](https://mlsanity.com/paper/1902.00098/) (2019)
- [A Hierarchical Latent Variable Encoder-Decoder Model for Generating Dialogues](https://mlsanity.com/paper/1605.06069/) (2016)
- [Emotional Chatting Machine: Emotional Conversation Generation with Internal and External Memory](https://mlsanity.com/paper/1704.01074/) (2017)
- [Isolating Sources of Disentanglement in Variational Autoencoders](https://mlsanity.com/paper/1802.04942/) (2018)
- [PLATO: Pre-trained Dialogue Generation Model with Discrete Latent Variable](https://mlsanity.com/paper/1910.07931/) (2019)
- [What makes a good conversation? How controllable attributes affect human judgments](https://mlsanity.com/paper/1902.08654/) (2019)
- [Neural Responding Machine for Short-Text Conversation](https://mlsanity.com/paper/1503.02364/) (2015)
- [Knowledge-Grounded Dialogue Generation with Pre-trained Language Models](https://mlsanity.com/paper/2010.08824/) (2020)
- [PLATO-2: Towards Building an Open-Domain Chatbot via Curriculum Learning](https://mlsanity.com/paper/2006.16779/) (2020)
- [S3VAE: Self-Supervised Sequential VAE for Representation Disentanglement and Data Generation](https://mlsanity.com/paper/2005.11437/) (2020)
- [Low-Resource Knowledge-Grounded Dialogue Generation](https://mlsanity.com/paper/2002.10348/) (2020)
- [Conversations Are Not Flat: Modeling the Dynamic Information Flow across Dialogue Utterances](https://mlsanity.com/paper/2106.02227/) (2021)
- [DialogBERT: Discourse-Aware Response Generation via Learning to Recover and Rank Utterances](https://mlsanity.com/paper/2012.01775/) (2020)
- [ReCoSa: Detecting the Relevant Contexts with Self-Attention for Multi-turn Dialogue Generation](https://mlsanity.com/paper/1907.05339/) (2019)
- [Learning to Ask Questions in Open-domain Conversational Systems with Typed Decoders](https://mlsanity.com/paper/1805.04843/) (2018)
- [Structured Attention for Unsupervised Dialogue Structure Induction](https://mlsanity.com/paper/2009.08552/) (2020)
- [Unsupervised Dialog Structure Learning](https://mlsanity.com/paper/1904.03736/) (2019)

## Built on by (most cited first, 4 of 4)

- [Lift Yourself Up: Retrieval-augmented Text Generation with Self Memory](https://mlsanity.com/paper/2305.02437/) (2023)
- [Empathy Level Alignment via Reinforcement Learning for Empathetic Response Generation](https://mlsanity.com/paper/2408.02976/) (2024)
- [Matching-based Term Semantics Pre-training for Spoken Patient Query Understanding](https://mlsanity.com/paper/2303.01341/) (2023)
- [Reasoning over Video: Evaluating How MLLMs Extract, Integrate, and Reconstruct Spatiotemporal Evidence](https://mlsanity.com/paper/2603.13091/) (2026)

## Abstract (arXiv)

With the availability of massive general-domain dialogue data, pre-trained dialogue generation appears to be super appealing to transfer knowledge from the general domain to downstream applications. In most existing work, such transferable ability is mainly obtained by fitting a large model with hundreds of millions of parameters on massive data in an exhaustive way, leading to inefficient running and poor interpretability. This paper proposes a novel dialogue generation model with a latent structure that is easily transferable from the general domain to downstream tasks in a lightweight and transparent way. Experiments on two benchmarks validate the effectiveness of the proposed model. Thanks to the transferable latent structure, our model is able to yield better dialogue responses than four strong baselines in terms of both automatic and human evaluations, and our model with about 22% parameters particularly delivers a 5x speedup in running time compared with the strongest baseline. Moreover, the proposed model is explainable by interpreting the discrete latent variables.