# Pre-training Multi-party Dialogue Models with Latent Discourse Inference

arXiv: https://arxiv.org/abs/2305.15175 · HTML: https://mlsanity.com/paper/2305.15175/

## Key facts

- Cited by 4 papers in the map (data through 14 Oct 2026).
- Builds on 30 of its 39 references that are in the map.
- First posted on arXiv 24 May 2023.

## Builds on (most cited first, 30 of 30)

- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [RoBERTa: A Robustly Optimized BERT Pretraining Approach](https://mlsanity.com/paper/1907.11692/) (2019)
- [Auto-Encoding Variational Bayes](https://mlsanity.com/paper/1312.6114/) (2013)
- [Categorical Reparameterization with Gumbel-Softmax](https://mlsanity.com/paper/1611.01144/) (2016)
- [The Concrete Distribution: A Continuous Relaxation of Discrete Random Variables](https://mlsanity.com/paper/1611.00712/) (2016)
- [DialoGPT: Large-Scale Generative Pre-training for Conversational Response Generation](https://mlsanity.com/paper/1911.00536/) (2019)
- [CoQA: A Conversational Question Answering Challenge](https://mlsanity.com/paper/1808.07042/) (2018)
- [The Ubuntu Dialogue Corpus: A Large Dataset for Research in Unstructured Multi-Turn Dialogue Systems](https://mlsanity.com/paper/1506.08909/) (2015)
- [PLATO: Pre-trained Dialogue Generation Model with Discrete Latent Variable](https://mlsanity.com/paper/1910.07931/) (2019)
- [Controllable Natural Language Generation with Contrastive Prefixes](https://mlsanity.com/paper/2202.13257/) (2022)
- [Speaker-Aware BERT for Multi-Turn Response Selection in Retrieval-Based Chatbots](https://mlsanity.com/paper/2004.03588/) (2020)
- [Molweni: A Challenge Multiparty Dialogues-based Machine Reading Comprehension Dataset with Discourse Structure](https://mlsanity.com/paper/2004.05080/) (2020)
- [MPC-BERT: A Pre-Trained Language Model for Multi-Party Conversation Understanding](https://mlsanity.com/paper/2106.01541/) (2021)
- [Pretraining Methods for Dialog Context Representation Learning](https://mlsanity.com/paper/1906.00414/) (2019)
- [DialogVED: A Pre-trained Latent Variable Encoder-Decoder Model for Dialog Response Generation](https://mlsanity.com/paper/2204.13031/) (2022)
- [Dialogue Response Selection with Hierarchical Curriculum Learning](https://mlsanity.com/paper/2012.14756/) (2020)
- [GSN: A Graph-Structured Network for Multi-Party Dialogues](https://mlsanity.com/paper/1905.13637/) (2019)
- [Multi-turn Response Selection using Dialogue Dependency Relations](https://mlsanity.com/paper/2010.01502/) (2020)
- [Structural Pre-training for Dialogue Comprehension](https://mlsanity.com/paper/2105.10956/) (2021)
- [LOREN: Logic-Regularized Reasoning for Interpretable Fact Verification](https://mlsanity.com/paper/2012.13577/) (2020)
- [DADgraph: A Discourse-aware Dialogue Graph Neural Network for Multiparty Dialogue Machine Reading Comprehension](https://mlsanity.com/paper/2104.12377/) (2021)
- [HeterMPC: A Heterogeneous Graph Neural Network for Response Generation in Multi-Party Conversations](https://mlsanity.com/paper/2203.08500/) (2022)
- [Self- and Pseudo-self-supervised Prediction of Speaker and Key-utterance for Multi-party Dialogue Reading Comprehension](https://mlsanity.com/paper/2109.03772/) (2021)
- [DialogueBERT: A Self-Supervised Learning based Dialogue Pre-training Encoder](https://mlsanity.com/paper/2109.10480/) (2021)
- [Dialogue-oriented Pre-training](https://mlsanity.com/paper/2106.00420/) (2021)
- [Structural Characterization for Dialogue Disentanglement](https://mlsanity.com/paper/2110.08018/) (2021)
- [Back to the Future: Bidirectional Information Decoupling Network for Multi-turn Dialogue Modeling](https://mlsanity.com/paper/2204.08152/) (2022)
- [EM Pre-training for Multi-party Dialogue Response Generation](https://mlsanity.com/paper/2305.12412/) (2023)
- [Multi-tasking Dialogue Comprehension with Discourse Parsing](https://mlsanity.com/paper/2110.03269/) (2021)
- [Successful New-entry Prediction for Multi-Party Online Conversations via Latent Topics and Discourse Modeling](https://mlsanity.com/paper/2108.06975/) (2021)

## Built on by (most cited first, 4 of 4)

- [Multi-Party Conversational Agents: A Survey](https://mlsanity.com/paper/2505.18845/) (2025)
- [UAlign: Pushing the Limit of Template-free Retrosynthesis Prediction with Unsupervised SMILES Alignment](https://mlsanity.com/paper/2404.00044/) (2024)
- [Multi-turn Dialogue Comprehension from a Topic-aware Perspective](https://mlsanity.com/paper/2309.09666/) (2023)
- [DEPTH: Discourse Education through Pre-Training Hierarchically](https://mlsanity.com/paper/2405.07788/) (2024)

## Abstract (arXiv)

Multi-party dialogues are more difficult for models to understand than one-to-one two-party dialogues, since they involve multiple interlocutors, resulting in interweaving reply-to relations and information flows. To step over these obstacles, an effective way is to pre-train a model that understands the discourse structure of multi-party dialogues, namely, to whom each utterance is replying. However, due to the lack of explicitly annotated discourse labels in multi-party dialogue corpora, previous works fail to scale up the pre-training process by putting aside the unlabeled multi-party conversational data for nothing. To fully utilize the unlabeled data, we propose to treat the discourse structures as latent variables, then jointly infer them and pre-train the discourse-aware model by unsupervised latent variable inference methods. Experiments on multiple downstream tasks show that our pre-trained model outperforms strong baselines by large margins and achieves state-of-the-art (SOTA) results, justifying the effectiveness of our method. The official implementation of this paper is available at https://github.com/EricLee8/MPD_EMVI.