# MPCEval: A Benchmark for Multi-Party Conversation Generation

arXiv: https://arxiv.org/abs/2603.04969 · HTML: https://mlsanity.com/paper/2603.04969/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 25 of its 54 references that are in the map.
- First posted on arXiv 5 Mar 2026.

## Builds on (most cited first, 25 of 25)

- [The Llama 3 Herd of Models](https://mlsanity.com/paper/2407.21783/) (2024)
- [Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena](https://mlsanity.com/paper/2306.05685/) (2023)
- [BERTScore: Evaluating Text Generation with BERT](https://mlsanity.com/paper/1904.09675/) (2019)
- [G-Eval: NLG Evaluation using GPT-4 with Better Human Alignment](https://mlsanity.com/paper/2303.16634/) (2023)
- [A Survey on LLM-as-a-Judge](https://mlsanity.com/paper/2411.15594/) (2024)
- [A Diversity-Promoting Objective Function for Neural Conversation Models](https://mlsanity.com/paper/1510.03055/) (2015)
- [BARTScore: Evaluating Generated Text as Text Generation](https://mlsanity.com/paper/2106.11520/) (2021)
- [Transformer in Transformer](https://mlsanity.com/paper/2103.00112/) (2021)
- [Texygen: A Benchmarking Platform for Text Generation Models](https://mlsanity.com/paper/1802.01886/) (2018)
- [Towards a Unified Multi-Dimensional Evaluator for Text Generation](https://mlsanity.com/paper/2210.07197/) (2022)
- [USR: An Unsupervised and Reference Free Evaluation Metric for Dialog Generation](https://mlsanity.com/paper/2005.00456/) (2020)
- [Survey on Evaluation Methods for Dialogue Systems](https://mlsanity.com/paper/1905.04071/) (2019)
- [Human Evaluation of Conversations is an Open Problem: comparing the sensitivity of various methods for evaluating dialogue agents](https://mlsanity.com/paper/2201.04723/) (2022)
- [The Llama 4 Herd: Architecture, Training, Evaluation, and Deployment Notes](https://mlsanity.com/paper/2601.11659/) (2026)
- [DeliData: A dataset for deliberation in multi-party problem solving](https://mlsanity.com/paper/2108.05271/) (2021)
- [A Survey on Recent Advances in Conversational Data Generation](https://mlsanity.com/paper/2405.13003/) (2024)
- [Addressee and Response Selection in Multi-Party Conversations with Speaker Interaction RNNs](https://mlsanity.com/paper/1709.04005/) (2017)
- [Interconnected Question Generation with Coreference Alignment and Conversation Flow Modeling](https://mlsanity.com/paper/1906.06893/) (2019)
- [Generating Information-Seeking Conversations from Unlabeled Documents](https://mlsanity.com/paper/2205.12609/) (2022)
- [NeuralWOZ: Learning to Collect Task-Oriented Dialogue via Model-Based Simulation](https://mlsanity.com/paper/2105.14454/) (2021)
- [HeterMPC: A Heterogeneous Graph Neural Network for Response Generation in Multi-Party Conversations](https://mlsanity.com/paper/2203.08500/) (2022)
- [Is ChatGPT a Good Multi-Party Conversation Solver?](https://mlsanity.com/paper/2310.16301/) (2023)
- [DG2: Data Augmentation Through Document Grounded Dialogue Generation](https://mlsanity.com/paper/2112.08342/) (2021)
- [Generalizability of Large Language Model-Based Agents: A Comprehensive Survey](https://mlsanity.com/paper/2509.16330/) (2025)
- [RADE: Reference-Assisted Dialogue Evaluation for Open-Domain Dialogue](https://mlsanity.com/paper/2309.08156/) (2023)

## Abstract (arXiv)

Multi-party conversation generation, such as smart reply and collaborative assistants, is an increasingly important capability of generative AI, yet its evaluation remains a critical bottleneck. Compared to two-party dialogue, multi-party settings introduce distinct challenges, including complex turn-taking, role-dependent speaker behavior, long-range conversational structure, and multiple equally valid continuations. Accordingly, we introduce MPCEval, a task-aware evaluation and benchmarking suite for multi-party conversation generation. MPCEval decomposes generation quality into speaker modeling, content quality, and speaker--content consistency, and explicitly distinguishes local next-turn prediction from global full-conversation generation. It provides novel, quantitative, reference-free, and reproducible metrics that scale across datasets and models. We apply MPCEval to diverse public and real-world datasets and evaluate modern generation methods alongside human-authored conversations. The results reveal systematic, dimension-specific model characteristics in participation balance, content progression and novelty, and speaker--content consistency, demonstrating that evaluation objectives critically shape model assessment and that single-score evaluation obscures fundamental differences in multi-party conversational behavior. The implementation of MPCEval and the associated evaluation code are publicly available at https://github.com/Owen-Yang-18/MPCEval.