mlsanityData to 7 Oct 2026

ChatEval: Towards Better LLM-based Evaluators through Multi-Agent Debate

arXiv 2308.07201 ↗First posted 14 Aug 2023cs.CLcited by 724

On the map: LLM agents and tool use → General LLM agent frameworks and benchmarks

Abstract (arXiv)

Text evaluation has historically posed significant challenges, often demanding substantial labor and time cost. With the emergence of large language models (LLMs), researchers have explored LLMs' potential as alternatives for human evaluation. While these single-agent-based approaches show promise, experimental results suggest that further advancements are needed to bridge the gap between their current effectiveness and human-level evaluation quality. Recognizing that best practices of human evaluation processes often involve multiple human annotators collaborating in the evaluation, we resort to a multi-agent debate framework, moving beyond single-agent prompting strategies. The multi-agent-based approach enables a group of LLMs to synergize with an array of intelligent counterparts, harnessing their distinct capabilities and expertise to enhance efficiency and effectiveness in handling intricate tasks. In this paper, we construct a multi-agent referee team called ChatEval to autonomously discuss and evaluate the quality of generated responses from different models on open-ended questions and traditional natural language generation (NLG) tasks. Our analysis shows that ChatEval transcends mere textual scoring, offering a human-mimicking evaluation process for reliable assessments. Our code is available at https://github.com/chanchimin/ChatEval.

Key facts. Cited by 724 papers in the map (data through 7 Oct 2026). Placed in General LLM agent frameworks and benchmarks, one of 16 directions in LLM agents and tool use. Builds on 29 of its 49 references that are in the map. First posted on arXiv 14 Aug 2023.

Builds on

29 of 49 references.

Closest ancestors · 1

Same direction, newest first

  1. 2023Unleashing the Emergent Cognitive Synergy in Large Language Models: A Task-Solving Agent through Multi-Persona Self-Collaboration11 Jul

Other references · 28

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  2. 2020Language Models are Few-Shot Learnerscited by 32,199
  3. 2022Training language models to follow instructions with human feedbackcited by 15,712
  4. 2023Judging LLM-as-a-Judge with MT-Bench and Chatbot Arenacited by 8,393
  5. 2019BERTScore: Evaluating Text Generation with BERTcited by 5,462
  6. 2021Finetuned Language Models Are Zero-Shot Learnerscited by 3,747
  7. 2023Generative Agents: Interactive Simulacra of Human Behaviorcited by 3,500
  8. 2023G-Eval: NLG Evaluation using GPT-4 with Better Human Alignmentcited by 2,205
  9. 2023Improving Factuality and Reasoning in Language Models through Multiagent Debatecited by 1,739
  10. 2021Multitask Prompted Training Enables Zero-Shot Task Generalizationcited by 1,462
  11. 2020BLEURT: Learning Robust Metrics for Text Generationcited by 1,196
  12. 2023Large Language Models are not Fair Evaluatorscited by 1,050
16 more
  1. 2023Encouraging Divergent Thinking in Large Language Models through Multi-Agent Debatecited by 995
  2. 2023Can Large Language Models Be an Alternative to Human Evaluations?cited by 729
  3. 2023Is ChatGPT a Good NLG Evaluator? A Preliminary Studycited by 420
  4. 2019MoverScore: Text Generation Evaluating with Contextualized Embeddings and Earth Mover Distancecited by 415
  5. 2023GPTScore: Evaluate as You Desirecited by 357
  6. 2023PandaLM: An Automatic Evaluation Benchmark for LLM Instruction Tuning Optimizationcited by 311
  7. 2020Evaluation of Text Generation: A Surveycited by 275
  8. 2022Towards a Unified Multi-Dimensional Evaluator for Text Generationcited by 264
  9. 2017Why We Need New Evaluation Metrics for NLGcited by 232
  10. 2020USR: An Unsupervised and Reference Free Evaluation Metric for Dialog Generationcited by 186
  11. 2023Benchmarking Foundation Models with Language-Model-as-an-Examinercited by 172
  12. 2023RoCo: Dialectic Multi-Robot Collaboration with Large Language Modelscited by 171
  13. 2023Human-like Summarization Evaluation with ChatGPTcited by 126
  14. 2023PRD: Peer Rank and Discussion Improve Large Language Model based Evaluationscited by 114
  15. 2021The Perils of Using Mechanical Turk to Evaluate Open-Ended Text Generationcited by 96
  16. 2023Large Language Models are Diverse Role-Players for Summarization Evaluationcited by 62

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 38; 2024: 242; 2025: 257; 2026: 187.

724 papers.

Sort:

By area:

  1. 2023A Survey on Large Language Model based Autonomous Agentscited by 2,203
  2. 2023The Rise and Potential of Large Language Model Based Agents: A Surveycited by 1,422
  3. 2024Large Language Model based Multi-Agents: A Survey of Progress and Challengescited by 752
  4. 2023AgentVerse: Facilitating Multi-Agent Collaboration and Exploring Emergent Behaviorscited by 580
  5. 2025Why Do Multi-Agent LLM Systems Fail?cited by 559
  6. 2024From Generation to Judgment: Opportunities and Challenges of LLM-as-a-judgecited by 480
  7. 2024LLMs-as-Judges: A Comprehensive Survey on LLM-based Evaluation Methodscited by 451
  8. 2024Prometheus 2: An Open Source Language Model Specialized in Evaluating Other Language Modelscited by 431
  9. 2025Multi-Agent Collaboration Mechanisms: A Survey of LLMscited by 424
  10. 2023Cognitive Architectures for Language Agentscited by 422
  11. 2024Mixture-of-Agents Enhances Large Language Model Capabilitiescited by 420
  12. 2023ReConcile: Round-Table Conference Improves Reasoning via Consensus among Diverse LLMscited by 297
38 more
  1. 2024JudgeBench: A Benchmark for Evaluating LLM-based Judgescited by 294
  2. 2023Evaluating Large Language Models at Evaluating Instruction Followingcited by 290
  3. 2023A Dynamic LLM-Powered Agent Network for Task-Oriented Agent Collaborationcited by 281
  4. 2023Large Language Models Empowered Agent-based Modeling and Simulation: A Survey and Perspectivescited by 253
  5. 2023The Unlocking Spell on Base LLMs: Rethinking Alignment via In-Context Learningcited by 251
  6. 2024Scaling Large Language Model-based Multi-Agent Collaborationcited by 235
  7. 2024When Can LLMs Actually Correct Their Own Mistakes? A Critical Survey of Self-Correction of LLMscited by 230
  8. 2024Agent-as-a-Judge: Evaluate Agents with Agentscited by 223
  9. 2023AutoAgents: A Framework for Automatic Agent Generationcited by 211
  10. 2024Two Tales of Persona in LLMs: A Survey of Role-Playing and Personalizationcited by 203
  11. 2025Large Language Model Agent: A Survey on Methodology, Applications and Challengescited by 194
  12. 2024Mobile-Agent-v2: Mobile Device Operation Assistant with Effective Navigation via Multi-Agent Collaborationcited by 187
  13. 2025Which Agent Causes Task Failures and When? On Automated Failure Attribution of LLM Multi-Agent Systemscited by 170
  14. 2024Rethinking the Bounds of LLM Reasoning: Are Multi-Agent Discussions the Key?cited by 153
  15. 2024Agent Smith: A Single Image Can Jailbreak One Million Multimodal LLM Agents Exponentially Fastcited by 149
  16. 2024UFO: A UI-Focused Agent for Windows OS Interactioncited by 147
  17. 2024Large Language Models are Inconsistent and Biased Evaluatorscited by 146
  18. 2025Evaluation and Benchmarking of LLM Agents: A Surveycited by 145
  19. 2024InfiAgent-DABench: Evaluating Agents on Data Analysis Taskscited by 134
  20. 2024MAGIS: LLM-Based Multi-Agent Framework for GitHub Issue Resolutioncited by 132
  21. 2024Exploring Large Language Model based Intelligent Agents: Definitions, Methods, and Prospectscited by 130
  22. 2024Cut the Crap: An Economical Communication Pipeline for LLM-based Multi-Agent Systemscited by 129
  23. 2025Advances and Challenges in Foundation Agents: From Brain-Inspired Intelligence to Evolutionary, Collaborative, and Safe Systemscited by 125
  24. 2024LLM Multi-Agent Systems: Challenges and Open Problemscited by 119
  25. 2024GTBench: Uncovering the Strategic Reasoning Limitations of LLMs via Game-Theoretic Evaluationscited by 118
  26. 2025Red-Teaming LLM Multi-Agent Systems via Communication Attackscited by 118
  27. 2023AgentCF: Collaborative Learning with Autonomous Language Agents for Recommender Systemscited by 116
  28. 2024FinCon: A Synthesized LLM Multi-Agent System with Conceptual Verbal Reinforcement for Enhanced Financial Decision Makingcited by 112
  29. 2024G-Designer: Architecting Multi-agent Communication Topologies via Graph Neural Networkscited by 110
  30. 2025A Survey of Frontiers in LLM Reasoning: Inference Scaling, Learning to Reason, and Agentic Systemscited by 108
  31. 2025Agentic Large Language Models, a surveycited by 105
  32. 2023Branch-Solve-Merge Improves Large Language Model Evaluation and Generationcited by 104
  33. 2023Should we be going MAD? A Look at Multi-Agent Debate Strategies for LLMscited by 103
  34. 2024Improving Multi-Agent Debate with Sparse Communication Topologycited by 103
  35. 2024Trust or Escalate: LLM Judges with Provable Guarantees for Human Agreementcited by 99
  36. 2024A Review of Large Language Models and Autonomous Agents in Chemistrycited by 96
  37. 2024AIOS: LLM Agent Operating Systemcited by 94
  38. 2024On scalable oversight with weak LLMs judging strong LLMscited by 94

50 most cited of 724 All 724 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2025Multi-Agent-as-Judge: Aligning LLM-Agent-Based Automated Evaluation with Multi-Dimensional Human Evaluation
  2. 2025Multi-Agent LLM Judge: automatic personalized LLM judge design for evaluating natural language generation applications
  3. 2025LLM Agents at the Roundtable: A Multi-Perspective and Dialectical Reasoning Framework for Essay Scoring
  4. 2025MEDAL: A Framework for Benchmarking LLMs as Multilingual Open-Domain Dialogue Evaluators
  5. 2023Exploring the Reliability of Large Language Models as Customized Evaluators for Diverse NLP Tasks
  6. 2025Evolutionary Perspectives on the Evaluation of LLM-Based AI Agents: A Comprehensive Survey
  7. 2026A Tri-Agent Framework for Evaluating and Aligning Question Clarification Capabilities of Large Language Models
  8. 2024HumanRankEval: Automatic Evaluation of LMs as Conversational Assistants