# MPC-BERT: A Pre-Trained Language Model for Multi-Party Conversation Understanding

arXiv: https://arxiv.org/abs/2106.01541 · HTML: https://mlsanity.com/paper/2106.01541/

## Key facts

- Cited by 36 papers in the map (data through 14 Oct 2026).
- Builds on 17 of its 26 references that are in the map.
- First posted on arXiv 3 Jun 2021.

## Builds on (most cited first, 17 of 17)

- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [Don't Stop Pretraining: Adapt Language Models to Domains and Tasks](https://mlsanity.com/paper/2004.10964/) (2020)
- [DialoGPT: Large-Scale Generative Pre-training for Conversational Response Generation](https://mlsanity.com/paper/1911.00536/) (2019)
- [Building End-To-End Dialogue Systems Using Generative Hierarchical Neural Network Models](https://mlsanity.com/paper/1507.04808/) (2015)
- [A Hierarchical Latent Variable Encoder-Decoder Model for Generating Dialogues](https://mlsanity.com/paper/1605.06069/) (2016)
- [The Ubuntu Dialogue Corpus: A Large Dataset for Research in Unstructured Multi-Turn Dialogue Systems](https://mlsanity.com/paper/1506.08909/) (2015)
- [Neural Responding Machine for Short-Text Conversation](https://mlsanity.com/paper/1503.02364/) (2015)
- [Generating Informative and Diverse Conversational Responses via Adversarial Information Maximization](https://mlsanity.com/paper/1809.05972/) (2018)
- [Sequential Matching Network: A New Architecture for Multi-turn Response Selection in Retrieval-based Chatbots](https://mlsanity.com/paper/1612.01627/) (2016)
- [Speaker-Aware BERT for Multi-Turn Response Selection in Retrieval-Based Chatbots](https://mlsanity.com/paper/2004.03588/) (2020)
- [GSN: A Graph-Structured Network for Multi-Party Dialogues](https://mlsanity.com/paper/1905.13637/) (2019)
- [Response Selection for Multi-Party Conversations with Dynamic Topic Tracking](https://mlsanity.com/paper/2010.07785/) (2020)
- [Dually Interactive Matching Network for Personalized Response Selection in Retrieval-Based Chatbots](https://mlsanity.com/paper/1908.05859/) (2019)
- [Addressee and Response Selection in Multi-Party Conversations with Speaker Interaction RNNs](https://mlsanity.com/paper/1709.04005/) (2017)
- [Interactive Matching Network for Multi-Turn Response Selection in Retrieval-Based Chatbots](https://mlsanity.com/paper/1901.01824/) (2019)
- [Towards Neural Speaker Modeling in Multi-Party Conversation: The Task, Dataset, and Models](https://mlsanity.com/paper/1708.03152/) (2017)

## Built on by (most cited first, 30 of 36)

- [DialogLM: Pre-trained Model for Long Dialogue Understanding and Summarization](https://mlsanity.com/paper/2109.02492/) (2021)
- [Werewolf Arena: A Case Study in LLM Evaluation via Social Deduction](https://mlsanity.com/paper/2407.13943/) (2024)
- [LiveChat: A Large-Scale Personalized Dialogue Dataset Automatically Constructed from Live Streaming](https://mlsanity.com/paper/2306.08401/) (2023)
- [HeterMPC: A Heterogeneous Graph Neural Network for Response Generation in Multi-Party Conversations](https://mlsanity.com/paper/2203.08500/) (2022)
- [Multi-party Goal Tracking with LLMs: Comparing Pre-training, Fine-tuning, and Prompt Engineering](https://mlsanity.com/paper/2308.15231/) (2023)
- [Multi-User Chat Assistant (MUCA): a Framework Using LLMs to Facilitate Group Conversations](https://mlsanity.com/paper/2401.04883/) (2024)
- [Enhanced Speaker-aware Multi-party Multi-turn Dialogue Comprehension](https://mlsanity.com/paper/2109.04066/) (2021)
- [Is ChatGPT a Good Multi-Party Conversation Solver?](https://mlsanity.com/paper/2310.16301/) (2023)
- [GIFT: Graph-Induced Fine-Tuning for Multi-Party Conversation Understanding](https://mlsanity.com/paper/2305.09360/) (2023)
- [Do LLMs suffer from Multi-Party Hangover? A Diagnostic Approach to Addressee Recognition and Response Selection in Conversations](https://mlsanity.com/paper/2409.18602/) (2024)
- [Friends-MMC: A Dataset for Multi-modal Multi-party Conversation Understanding](https://mlsanity.com/paper/2412.17295/) (2024)
- [Mixed-Session Conversation with Egocentric Memory](https://mlsanity.com/paper/2410.02503/) (2024)
- [Multi-Stage Pre-training Enhanced by ChatGPT for Multi-Scenario Multi-Domain Dialogue Summarization](https://mlsanity.com/paper/2310.10285/) (2023)
- [Multi-Party Conversational Agents: A Survey](https://mlsanity.com/paper/2505.18845/) (2025)
- [Smoothing Dialogue States for Open Conversational Machine Reading](https://mlsanity.com/paper/2108.12599/) (2021)
- [Multi-Party Supervised Fine-tuning of Language Models for Multi-Party Dialogue Generation](https://mlsanity.com/paper/2412.05342/) (2024)
- [MADNet: Maximizing Addressee Deduction Expectation for Multi-Party Conversation Generation](https://mlsanity.com/paper/2305.12733/) (2023)
- [Pre-training Multi-party Dialogue Models with Latent Discourse Inference](https://mlsanity.com/paper/2305.15175/) (2023)
- [OmniDialog: An Omnipotent Pre-training Model for Task-Oriented Dialogue System](https://mlsanity.com/paper/2312.16864/) (2023)
- [Multimodal Conversation Structure Understanding](https://mlsanity.com/paper/2505.17536/) (2025)
- [Semantic-based Pre-training for Dialogue Understanding](https://mlsanity.com/paper/2209.09146/) (2022)
- [CipherDM: Secure Three-Party Inference for Diffusion Model Sampling](https://mlsanity.com/paper/2409.05414/) (2024)
- [WHoW: A Cross-domain Approach for Analysing Conversation Moderation](https://mlsanity.com/paper/2410.15551/) (2024)
- [Towards End-to-End Open Conversational Machine Reading](https://mlsanity.com/paper/2210.07113/) (2022)
- [Dramatic Conversation Disentanglement](https://mlsanity.com/paper/2305.16648/) (2023)
- [Integrating Audio, Visual, and Semantic Information for Enhanced Multimodal Speaker Diarization](https://mlsanity.com/paper/2408.12102/) (2024)
- [Detecting Agreement in Multi-party Conversational AI](https://mlsanity.com/paper/2311.03026/) (2023)
- [SS-MPC: A Sequence-Structured Multi-Party Conversation System](https://mlsanity.com/paper/2502.16920/) (2025)
- [GroupGPT: A Token-efficient and Privacy-preserving Agentic Framework for Multi-User Chat Assistant](https://mlsanity.com/paper/2603.01059/) (2026)
- [MultiTalk: Scaling Full-Duplex Speech Models to Long, Multi-Party, Bilingual Conversation](https://mlsanity.com/paper/2609.36903/) (2026)

## Abstract (arXiv)

Recently, various neural models for multi-party conversation (MPC) have achieved impressive improvements on a variety of tasks such as addressee recognition, speaker identification and response prediction. However, these existing methods on MPC usually represent interlocutors and utterances individually and ignore the inherent complicated structure in MPC which may provide crucial interlocutor and utterance semantics and would enhance the conversation understanding process. To this end, we present MPC-BERT, a pre-trained model for MPC understanding that considers learning who says what to whom in a unified model with several elaborated self-supervised tasks. Particularly, these tasks can be generally categorized into (1) interlocutor structure modeling including reply-to utterance recognition, identical speaker searching and pointer consistency distinction, and (2) utterance semantics modeling including masked shared utterance restoration and shared node detection. We evaluate MPC-BERT on three downstream tasks including addressee recognition, speaker identification and response selection. Experimental results show that MPC-BERT outperforms previous methods by large margins and achieves new state-of-the-art performance on all three downstream tasks at two benchmarks.