mlsanityData to 7 Oct 2026

PingPong: A Benchmark for Role-Playing Language Models with User Emulation and Multi-Model Evaluation

arXiv 2409.06820 ↗First posted 10 Sep 2024cs.CLcited by 10

Not on the map. Why?

Abstract (arXiv)

We introduce a benchmark for evaluating the role-playing capabilities of language models. Our approach leverages different language models to simulate users in dynamic, multi-turn conversations and assess the resulting dialogues. Our methodology involves three main components: a player model that adopts a specific character role, an interrogator model that simulates user behavior in a specific situation, and a judge model ensemble that evaluates conversation quality with 3 metrics: character consistency, entertainment value, and language fluency. We evaluated more than 40 models in both English and Russian, with each model participating in 64 conversations with 8 characters and 8 situations. We conducted experiments comparing automated evaluations with human annotations to validate our approach, demonstrating strong correlations across multiple criteria. This work provides a foundation for a robust and dynamic evaluation of different model capabilities in interactive scenarios.

Key facts. Cited by 10 papers in the map (data through 7 Oct 2026). Builds on 32 of its 41 references that are in the map. First posted on arXiv 10 Sep 2024.

Builds on

32 of 41 references.

References · 32

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  2. 2023GPT-4 Technical Reportcited by 18,132
  3. 2022Training language models to follow instructions with human feedbackcited by 15,712
  4. 2024The Llama 3 Herd of Modelscited by 14,192
  5. 2023Judging LLM-as-a-Judge with MT-Bench and Chatbot Arenacited by 8,393
  6. 2020Measuring Massive Multitask Language Understandingcited by 7,768
  7. 2024Gemini 1.5: Unlocking multimodal understanding across millions of tokens of contextcited by 3,175
  8. 2019The Curious Case of Neural Text Degenerationcited by 2,881
  9. 2024Gemma 2: Improving Open Language Models at a Practical Sizecited by 1,928
  10. 2019SuperGLUE: A Stickier Benchmark for General-Purpose Language Understanding Systemscited by 1,718
  11. 2024Chatbot Arena: An Open Platform for Evaluating LLMs by Human Preferencecited by 1,151
  12. 2024SimPO: Simple Preference Optimization with a Reference-Free Rewardcited by 951
20 more
  1. 2024Length-Controlled AlpacaEval: A Simple Way to Debias Automatic Evaluatorscited by 865
  2. 2022BERTopic: Neural topic modeling with a class-based TF-IDF procedurecited by 779
  3. 2023ChatEval: Towards Better LLM-based Evaluators through Multi-Agent Debatecited by 724
  4. 2024LLM Evaluators Recognize and Favor Their Own Generationscited by 651
  5. 2023Character-LLM: A Trainable Agent for Role-Playingcited by 335
  6. 2024Replacing Judges with Juries: Evaluating LLM Generations with a Panel of Diverse Modelscited by 301
  7. 2024MT-Bench-101: A Fine-Grained Benchmark for Evaluating Large Language Models in Multi-Turn Dialoguescited by 247
  8. 2023InCharacter: Evaluating Personality Fidelity in Role-Playing Agents through Psychological Interviewscited by 177
  9. 2023Simulating Opinion Dynamics with Networks of LLM-based Agentscited by 153
  10. 2024CharacterEval: A Chinese Benchmark for Role-Playing Conversational Agent Evaluationcited by 139
  11. 2023Investigating Data Contamination in Modern Benchmarks for Large Language Modelscited by 136
  12. 2024Pride and Prejudice: LLM Amplifies Self-Bias in Self-Refinementcited by 111
  13. 2024Large Language Models are Superpositions of All Characters: Attaining Arbitrary Role-play via Self-Alignmentcited by 104
  14. 2024The BiGGen Bench: A Principled Benchmark for Fine-grained Evaluation of Language Models with Language Modelscited by 89
  15. 2024PersonaGym: Evaluating Persona Agents and LLMscited by 89
  16. 2024Roleplay-doh: Enabling Domain-Experts to Create LLM-simulated Patients via Eliciting and Adhering to Principlescited by 72
  17. 2023Rewarding Chatbots for Real-World Engagement with Millions of Userscited by 58
  18. 2024How Well Can LLMs Echo Us? Evaluating AI Chatbots' Role-Play Ability with ECHOcited by 19
  19. 2024Vikhr: The Family of Open-Source Instruction-Tuned Large Language Models for Russiancited by 12
  20. 2023PIPPA: A Partially Synthetic Conversational Datasetcited by 6

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2025: 6; 2026: 4.

10 papers.

  1. 2025Evaluating LLM-based Agents for Multi-Turn Conversations: A Surveycited by 56
  2. 20253MDBench: Medical Multimodal Multi-agent Dialogue Benchmarkcited by 16
  3. 2025PersonaEval: Are LLM Evaluators Human Enough to Judge Role-Play?cited by 9
  4. 2025RefuteBench 2.0 -- Agentic Benchmark for Dynamic Evaluation of LLM Responses to Refutation Instructioncited by 2
  5. 2025Concept Incongruence: An Exploration of Time and Death in Role Playingcited by 2
  6. 2025Human or LLM as Standardized Patients? A Comparative Study for Medical Educationcited by 1
  7. 2026TRACE Bench: Task-driven Roleplay Agentic Checklist Evaluationcited by 1
  8. 2026REAR: Test-time Preference Realignment through Reward Decompositioncited by 0
  9. 2026Beyond Borrowed Histories: Person-Aligned User Simulation for Interactive Role-Playing Evaluationcited by 0
  10. 2026StudentSim: Training LLM-based Student Simulatorscited by 0