# ComperDial: Commonsense Persona-grounded Dialogue Dataset and Benchmark

arXiv: https://arxiv.org/abs/2406.11228 · HTML: https://mlsanity.com/paper/2406.11228/

## Key facts

- Cited by 3 papers in the map (data through 14 Oct 2026).
- Builds on 29 of its 41 references that are in the map.
- First posted on arXiv 17 Jun 2024.

## Builds on (most cited first, 29 of 29)

- [Chain-of-Thought Prompting Elicits Reasoning in Large Language Models](https://mlsanity.com/paper/2201.11903/) (2022)
- [Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena](https://mlsanity.com/paper/2306.05685/) (2023)
- [BERTScore: Evaluating Text Generation with BERT](https://mlsanity.com/paper/1904.09675/) (2019)
- [G-Eval: NLG Evaluation using GPT-4 with Better Human Alignment](https://mlsanity.com/paper/2303.16634/) (2023)
- [BLEURT: Learning Robust Metrics for Text Generation](https://mlsanity.com/paper/2004.04696/) (2020)
- [Chatbot Arena: An Open Platform for Evaluating LLMs by Human Preference](https://mlsanity.com/paper/2403.04132/) (2024)
- [Personalizing Dialogue Agents: I have a dog, do you have pets too?](https://mlsanity.com/paper/1801.07243/) (2018)
- [ChatEval: Towards Better LLM-based Evaluators through Multi-Agent Debate](https://mlsanity.com/paper/2308.07201/) (2023)
- [Can Large Language Models Be an Alternative to Human Evaluations?](https://mlsanity.com/paper/2305.01937/) (2023)
- [DailyDialog: A Manually Labelled Multi-turn Dialogue Dataset](https://mlsanity.com/paper/1710.03957/) (2017)
- [LMSYS-Chat-1M: A Large-Scale Real-World LLM Conversation Dataset](https://mlsanity.com/paper/2309.11998/) (2023)
- [Towards a Unified Multi-Dimensional Evaluator for Text Generation](https://mlsanity.com/paper/2210.07197/) (2022)
- [Learning Discourse-level Diversity for Neural Dialog Models using Conditional Variational Autoencoders](https://mlsanity.com/paper/1703.10960/) (2017)
- [USR: An Unsupervised and Reference Free Evaluation Metric for Dialog Generation](https://mlsanity.com/paper/2005.00456/) (2020)
- [What makes a good conversation? How controllable attributes affect human judgments](https://mlsanity.com/paper/1902.08654/) (2019)
- [Unsupervised Evaluation of Interactive Dialog with DialoGPT](https://mlsanity.com/paper/2006.12719/) (2020)
- [Topical-Chat: Towards Knowledge-Grounded Open-Domain Conversations](https://mlsanity.com/paper/2308.11995/) (2023)
- [Human Evaluation of Conversations is an Open Problem: comparing the sensitivity of various methods for evaluating dialogue agents](https://mlsanity.com/paper/2201.04723/) (2022)
- [A Comprehensive Analysis of the Effectiveness of Large Language Models as Automatic Dialogue Evaluators](https://mlsanity.com/paper/2312.15407/) (2023)
- [Approximating Interactive Human Evaluation with Self-Play for Open-Domain Dialog Systems](https://mlsanity.com/paper/1906.09308/) (2019)
- [Designing Precise and Robust Dialogue Response Evaluators](https://mlsanity.com/paper/2004.04908/) (2020)
- [Opportunities and Challenges in Neural Dialog Tutoring](https://mlsanity.com/paper/2301.09919/) (2023)
- [Achieving Reliable Human Assessment of Open-Domain Dialogue Systems](https://mlsanity.com/paper/2203.05899/) (2022)
- [PeaCoK: Persona Commonsense Knowledge for Consistent and Engaging Narratives](https://mlsanity.com/paper/2305.02364/) (2023)
- [Learning to Memorize Entailment and Discourse Relations for Persona-Consistent Dialogues](https://mlsanity.com/paper/2301.04871/) (2023)
- [End-to-end Conversation Modeling Track in DSTC6](https://mlsanity.com/paper/1706.07440/) (2017)
- [Interactive Evaluation of Dialog Track at DSTC9](https://mlsanity.com/paper/2207.14403/) (2022)
- [RADE: Reference-Assisted Dialogue Evaluation for Open-Domain Dialogue](https://mlsanity.com/paper/2309.08156/) (2023)
- [Evaluating Open-Domain Dialogues in Latent Space with Next Sentence Prediction and Mutual Information](https://mlsanity.com/paper/2305.16967/) (2023)

## Built on by (most cited first, 3 of 3)

- [How Far are LLMs from Being Our Digital Twins? A Benchmark for Persona-Based Behavior Chain Simulation](https://mlsanity.com/paper/2502.14642/) (2025)
- [Model Fusion with Multi-LoRA Inference for Tool-Enhanced Game Dialogue Agents](https://mlsanity.com/paper/2509.24229/) (2025)
- [Agentic Persona Control and Task State Tracking for Realistic User Simulation in Interactive Scenarios](https://mlsanity.com/paper/2601.15290/) (2025)

## Abstract (arXiv)

We propose a new benchmark, ComperDial, which facilitates the training and evaluation of evaluation metrics for open-domain dialogue systems. ComperDial consists of human-scored responses for 10,395 dialogue turns in 1,485 conversations collected from 99 dialogue agents submitted to the Commonsense Persona-grounded Dialogue (CPD) challenge. As a result, for any dialogue, our benchmark includes multiple diverse responses with variety of characteristics to ensure more robust evaluation of learned dialogue metrics. In addition to single-turn response scores, ComperDial also contains dialogue-level human-annotated scores, enabling joint assessment of multi-turn model responses throughout a dialogue. Finally, building off ComperDial, we devise a new automatic evaluation metric to measure the general similarity of model-generated dialogues to human conversations. Our experimental results demonstrate that our novel metric, CPDScore is more correlated with human judgments than existing metrics. We release both ComperDial and CPDScore to the community to accelerate development of automatic evaluation metrics for open-domain dialogue systems.